Week 3: A mypy/unicode week

Couple of things I’d like to mention this week.

Since I was stuck at unit testing in cwltool last week, I decided to upgrade the mypy package to 0.511 for cwltool, fix new errors which emerge from doing the upgrade, update typesheds in this process and add new typesheds which are py3 compatible.

When I ran mypy without --py2 flag on the codebase, there were alot of errors which were related to confusion between binary data and textual data being processed in the codebase (since one can use str annotation for both kind of data in py2 but not in py3). Due to the lack of docstrings, I’ve to manually go through the code and figure out if it’s binary or textual data and make changes in the annotations or actual code to fix these errors/potential bugs. Before doing that, I decided to use unicode strings consistantly in the codebase for Python 2 by importing from __future__ import unicode literals in each module in the codebase.

Projects like Django have used this approch in the past to port their codebase. Petr has also expressed interest in consistently using unicode strings in the codebase and there have been guides on the internet like this one speaking for and against this approach. The idea is to create better distinction between binary data and unicode strings, hopefully making the process of porting to Python 3 potentially easier for me. If all the textual data will stand out - I can, in more explicit and better manner, deal with file IO and encoding/decoding of data.

I’ve created a new branch, and have pushed my latest commits at: manu-chroma/unicode_literal_usage. This is rebased on top of my previous work in mypy_updates and modernize branch.

Changes made so far does not break backward compatibility. Thus, unit tests and conformance tests still pass in py27 runtime which latest changes, so using unicode everywhere seems like a welcoming change.

P.S. I was also interested in measuring combined code coverage of the codebase for unit tests and conformace tests. With some hacking of cwltest package and conformace test bash script, I was able to do that. Coverage is low at the moment but we’re working on improving it. Also, this made me realise how handy mypy tool is in this process.

P.P.S. I found a bug in mypy stub files for python2 while debugging errors in cwltool. In shorts, it is related to passing unicode strings to stdlib functions works correctly at runtime, but mypy shows unsupported type. For more details: typeshed/issues/1411 I intend to make a PR for this before next mypy release.

ToDo next week? Make all cwltool unit test pass in Python 3 runtime.