v2 source with conversion to v3

If you have a large v2 codebase, then a good start to the migration task is rejigging your source so it can be converted automatically into v3. The Python core developers were farsighted enough to include automatic translation tools as a part of the first v3 release, and this support continues right up the the present day in the form of the 2to3 utility. 2to3 also provides a lib2to3 library containing fixers, functions that handle specific constructs requiring conversion. Other conversion utilities we discuss later in this chapter also leverage lib2to3.

The first task is to assemble your source code and prepare it for conversion to Python 3. Fortunately, nowadays, this is not difficult, as core and distribution developers have thoughtfully provided conversion routines. Installing a Python 3.5 version of Anaconda, for example, lets you (in a command-line environment where tab completion is available) enter 2to3 and hit the Tab key to reveal:

(s3cmdpy3) airhead:s3cmd sholden$ 2to3
2to3      2to3-2    2to3-2.7  2to3-3.4  2to3-3.5  2to32.6

Note, though, that not all of these come from Anaconda (the system in use had a number of Pythons on the system path), as the following directory listing shows:

airhead:s3cmd sholden$ for f in 2to3 2to3- 2to3-2 2to3-2.7 
                                2to3-3.4 2to3-3.5 2to32.6
do
 ls -l $(which $f)
done
lrwxr-xr-x  1 sholden  staff  8 Mar 14 21:45 /Users/sholden/
Projects/Anaconda/bin/2to3 -> 2to3-3.5
lrwxr-xr-x  1 sholden  admin  36 Oct 24 12:30 /usr/local/bin/
2to3-2 -> ../Cellar/python/2.7.10_2/bin/2to3-2
lrwxr-xr-x  1 sholden  admin  38 Oct 24 12:30 /usr/local/bin/
2to3-2.7 -> ../Cellar/python/2.7.10_2/bin/2to3-2.7
lrwxr-xr-x  1 sholden  admin  38 Sep 12  2015 /usr/local/bin/
2to3-3.4 -> ../Cellar/python3/3.4.3_2/bin/2to3-3.4
-rwxr-xr-x  1 sholden  staff  121 Mar 14 21:45 /Users/sholden/
Projects/Anaconda/bin/2to3-3.5
lrwxr-xr-x  1 root  wheel  73 Aug  6  2015 /usr/bin/
2to32.6 -> ../../System/Library/Frameworks/Python.framework/
Versions/2.6/bin/2to32.6

The Anaconda install supplies 2to3.5, and a symbolic link from 2to3 in the same directory; the remainder are from other installs. As long as the Python executable directory is at the beginning of your system path, the default 2to3 is the one targeting that installation: you’ll be converting to the v3 supported by it.

v3 source with conversion to v2

If you are coding in v3 but want to support v2 too, use 3to2, aka lib3to2, documented online. 3to2 uses the same fixers as lib2to3, reversing them to create a backward-compatible version of your code. 3to2 either produces working v2 code, or provides error messages describing why it can’t. Some features in v3 are unavailable in v2: differences may be subtle—test your code in both versions.

Initial run of 2to3

2to3 has a number of different modes of operation. The most useful way to get a handle on the likely difficulty of a conversion is to simply run it on the project’s root directory. The command

2to3 .

scans all code ands report on suggested changes in the form of difference listings (diffs), summarizing which files likely need changing. Here is that summary:

RefactoringTool: Files that need to be modified:
RefactoringTool: ./setup.py
RefactoringTool: ./tappio/__init__.py
RefactoringTool: ./tappio/lexer.py
RefactoringTool: ./tappio/models.py
RefactoringTool: ./tappio/parser.py
RefactoringTool: ./tappio/writer.py
RefactoringTool: ./tappio/scripts/extract.py
RefactoringTool: ./tappio/scripts/graph.py
RefactoringTool: ./tappio/scripts/indent.py
RefactoringTool: ./tappio/scripts/merge.py
RefactoringTool: ./tappio/scripts/missing_accounts.py
RefactoringTool: ./tappio/scripts/move_entries.py
RefactoringTool: ./tappio/scripts/print_accounts.py
RefactoringTool: ./tappio/scripts/print_earnings.py
RefactoringTool: ./tappio/scripts/renumber.py
RefactoringTool: ./tests/helpers.py
RefactoringTool: ./tests/script_import_test.py
RefactoringTool: ./tests/tappio_test.py
RefactoringTool: ./voitto/__init__.py
RefactoringTool: ./voitto/helpers/io.py
RefactoringTool: ./voitto/helpers/tracer.py
RefactoringTool: Warnings/messages while refactoring:
RefactoringTool: ### In file ./tappio/scripts/missing_accounts.py ###
RefactoringTool: Line 36: You should use a for loop here

The diffs show that approximately 20 source files are being recommended for change. As the old saying goes, why keep a dog and bark yourself? We can have 2to3 actually make the recommended changes by adding the -w flag to the command line. Normally 2to3 creates a backup file by renaming the original file from .py to .py.bak. Operating under a version-control system renders this unnecessary, however, so we can also add the -n flag to inhibit the backups:

(venv3) $ 2to3 -wn .

Here are the changes that 2to3 makes—we don’t mention the filenames. Many of the changes are similar in nature, so we don’t provide an exhaustive commentary on each one. As detailed next, many of the changes might be suboptimal; 2to3 is much more focused on producing working code than in optimizing its performance (remember the “golden rule of programming,” covered in “Optimization”). Optimization is best treated as a separate step, once the code has been successfully converted and is passing its tests. The lines to be removed are preceded by minus signs (-), the replacement code by plus signs (+):

- assert type(ch) in (str, unicode)
+ assert type(ch) in (str, str)

v3, of course, does not define unicode, so 2to3 has converted it to str but failed to recognize that this no longer requires the tuple-membership test. Since the statement should work, however, leave this optimization for later.

- token = self.token_iterator.next()
+ token = next(self.token_iterator)

Several similar changes of this nature were recommended. In all cases the next method from v3, now renamed __next__, could simply transliterate the method call. However, 2to3 has chosen to produce better code, compatible across both versions, by using the next function instead.

- for account_number, cents in balances.iteritems():
+ for account_number, cents in balances.items():

Again several similar changes are recommended. In this case the converted code has a slightly different effect if run under v2, since the original was specifically coded to use an iterator rather than the list that (under v2) the items method produces. While this may use more memory, it shouldn’t affect the result.

- map(all_accounts.update, flat_accounts.itervalues())
+ list(map(all_accounts.update, iter(flat_accounts.values())))

This is an interesting change because the original code was rather dubious—this is the line where the migration tool advised use of a for loop instead. Under v2, map returns a list while the v3 map is an iterator. Not only has the itervalues call been changed to values (because v3’s values method produces an iterator; the iter call is redundant, though innocuous), but a list call has also been placed around the result to ensure that a list is still produced. It might appear that the application of the list function serves no useful purpose, but removing the call would mean that the all_accounts.update function would only be applied to accounts as they were consumed from the iterator.

In fact the original code is somewhat dubious, and misused the map function. It would really have been little more complex, and far more comprehensible, to write

for ac_value in iter(flat_accounts.values()):
    all_accounts.update(ac_value)

as 2to3 advised. We leave this optimization for now, though, since despite its dubious nature the code as produced still works.

- print "{account_num}: ALL  {fmt_filenames}".format(**locals())
+ print("{account_num}: ALL  {fmt_filenames}".format(**locals()))

2to3 ensures that all print statements are converted to function calls, and is good at making such changes.

- from StringIO import StringIO
+ from io import StringIO

The StringIO functionality has been moved to the io library in v3. Note that 2to3 does not necessarily produce code that is compatible across both versions; it pursues its principal purpose, conversion to v3—although in this case, as in previous ones, v2 is also happy with the converted code (since StringIO can also be accessed from io in v2, specifically in 2.7).

- assert_true(callable(main))
+ assert_true(isinstance(main, collections.Callable))

This change is a throwback because earlier v3 implementations removed the callable function. Since this was reinstated since v3.2, the conversion isn’t really necessary, but it’s harmless in both v2 and v3.

- except ParserError, e:
+ except ParserError as e:

This change also provides compatible syntax, good in both v2 and v3.

The 2to3 conversion detailed in this initial run of 2to3 does not, however, result in a working program. Running the tests still gives two errors, one in the lexer_single test and one in writer_single. Both complain about differing token lists. This is the first (we’ve inserted line breaks to avoid too-wide code boxes running off the page):

...
  File "/Users/sholden/Projects/Python/voitto/tests/tappio_test.py",
        line 54, in lexer_single
    assert_equal(expected_tokens, tokens)
AssertionError: Lists differ: [('integer', '101')] != 
                              [<Token: integer '101'>]

First differing element 0:
('integer', '101')
<Token: integer '101'>

- [('integer', '101')]
+ [<Token: integer '101'>]

The second one is similar:

...
  File "/Users/sholden/Projects/Python/voitto/tests/tappio_test.py", 
        line 188, in writer_single
    assert_equal(good_tokens, potentially_bad_tokens)
AssertionError: Lists differ:
  [<Tok[1169 chars]ce_close ''>, <Token: brace_close ''>, 
   <Token: brace_close ''>] != [<Tok[1169 chars]ce_close ''>, 
   <Token: brace_close ''>, <Token: brace_close ''>]

First differing element 0:
<Token: brace_open ''>
<Token: brace_open ''>

Diff is 1384 characters long. Set self.maxDiff to None to see it.

These were both traced to the token list comparison routine itself, which the original code provides in the form of a __cmp__ method for the Token class, reading as follows:

def __cmp__(self, other):
    if isinstance(other, Token):
        return cmp((self.token_type, self.value), 
                  (other.token_type, other.value))
    elif (isinstance(other, list) or 
          isinstance(other, tuple)) and len(other) == 2:
        return cmp((self.token_type, self.value), other)
    else:
        raise TypeError('cannot compare Token to {!r}'.format(
                                                       type(other)))

This method is not used in v3, which relies on so-called rich comparison methods. Since no __eq__ is found, equality checks falls back to those inherited from object, which unfortunately don’t measure up. The fix is to replace the __cmp__ method with an __eq__ method. This suffices to implement the equality checks required in the tests; the new code is:

def __eq__(self, other):
    if isinstance(other, Token):
        return (self.token_type, self.value) == 
               (other.token_type, other.value)
    elif (isinstance(other, list) or 
          isinstance(other, tuple)) and len(other) == 2:
        return (self.token_type, self.value) == tuple(other)
    else:
        raise TypeError('cannot compare Token to {!r}'.format(
                                                       type(other)))

This final change is all that is required to have all tests pass under v3:

(ve35) airhead:voitto sholden$ nosetests
....
----------------------------------------------------------------------
Ran 4 tests in 0.028s

OK

Retaining v2 compatibility

While we now have working v3 code, running the tests under v2 shows that we have broken backward compatibility, and the writer_test now fails:

File "/Users/sholden/Projects/Python/voitto/tappio/writer.py", 
        line 48, in write
    self.stream.write(str(token))
TypeError: unicode argument expected, got 'str'

It would be unfair to complain about the fact the 2to3 conversion breaks compatibility, since maintaining compatibility is no part of its purview. Happily, it is relatively easy to take this v3 code and make it also compatible with v2. The process does, for the first time, require writing some code that introspects to determine the particular version of Python. We discuss version compatibility further in “v2/v3 Support with a Single Source Tree”.

The issue is that the original code does not explicitly flag string literals as Unicode, so under v2 they are treated as being of type str rather than the required unicode type. It would be tedious to make this conversion manually, but there’s a solution in the __future__ module: the unicode_literals import. When this import is present in a v2 program, string literals are implicitly treated as Unicode strings, which is exactly the change we want. We therefore added the line

from __future__ import unicode_literals

as the first executable line in each module.

Note that this is not always the panacea it might appear to be (and it certainly doesn’t offer an immediate fix here, as we’ll shortly see). Although it’s helpful in this case, when programs have mixed binary and textual string data there is no one representation to serve both purposes, and many developers prefer to take things gradually. The changes can introduce regressions in Python 2, which have to be addressed. Not only that, but all docstrings implicitly then become Unicode, which prior to 2.7.7 would have caused syntax errors.

This change alone actually increases the number of failing tests to three, and the error is the same in all cases:

  File "/Users/sholden/Projects/Python/voitto/tappio/lexer.py", 
        line 45, in build_set
    assert type(ch) in (str, str)

You can see that the change that 2to3 made to remove the unicode type has come back to bite us because, while it suffices in v3, under v2 we need to explicitly allow strings of type unicode. We can’t just revert that change, since that type isn’t available under v3: attempts to reference it raise a NameError exception.

What we need is a type that is compatible with both versions. The six module (covered later) does introduce such a type, but, rather than carry the overhead of the entire module, in this case we chose to write a simple compatibility module at tappio/compat.py that creates an appropriate definition of the name unicode for the version it’s running on. Following the advice of the porting guide, we use feature detection rather than querying version information. compat.py reads as follows:

try:
    unicode = unicode
except NameError:
    unicode = str

The alert reader might ask themselves why it is necessary to bind the unicode name rather than simply referencing it. The reason is that in v2 the unicode on the RHS comes from the built-in namespace, and so, without the assignment, it would not be available for import from the module.

The modules tappio/writer.py and tappio/lexer.py were modified to import unicode from tappio.compat. In writer.py, str(token) was changed to unicode(token). In lexer.py, isinstance(ch, str) was reverted to isinstance(ch, unicode). After these changes, all tests pass under both v2 and v3.

six

The six library was the earliest project to provide such compatibility adaptation, but some people feel that it is too bulky, and convenience can also be an issue. It has been, however, successfully used in large projects such as Django to provide version compatibility, so you can trust that the library is capable of doing all you need.

six is particularly valuable when you need to retain compatibility with earlier v2 implementations, since it was produced when 2.6 was current and therefore also takes into account the extra issues that 2.6 compatibility raises. However, we recommend you target only Python 2.7, not earlier versions, in such compatibility quests: the task is hard enough without having to make it even harder.

python-modernize

The python-modernize library appeared when Python 3.3 was current. This library provides good compatibility between 2.6, 2.7, and 3.3 by providing a thin wrapper around 2to3, and is used in much the same way. The author, Armin Ronacher, describes its use, and gives porting advice useful to anyone considering dual-version support, in Porting to Python Redux.

python-modernize uses six as a base and operates much like 2to3, reading your code and writing out a version that’s easier to maintain as a dual-source codebase. Install it in a virtual environment with pip install modernize. Its output tends to move code stylistically toward v3.

python-future

The most recent tool for single-source support is the Python-Future library: install it in a virtual environment with pip install future. If you (as we recommend) support only v2 (Python 2.7) and v3 (Python 3.5 and newer), it stands alone, without external dependencies. With the addition of importlib, unittest2, and argparse, it can extend support to 2.6, should you absolutely require that.

Translation happens in two stages: the first stage uses only “safe” fixers that are extremely unlikely to break your code, and does not attempt to use any compatibility functions from the future package. The output from stage one is still v2 code with calls to v2 standard functions; the purpose of the two-stage approach is to remove uncontroversial changes, so that the second stage can focus on those more likely to cause breakage.

The second-stage changes add dependencies on the future package, and converts the source to v3 code that also runs on v2 by using the dependencies. Existing users of v2 might find that the converted output style is more like the code they are used to.

future also comes with a pasteurize command that claims to convert v3 code into v2 code. We found, however, that its default mode was unable to duplicate the v2 compatibility changes we discussed at the end of “Retaining v2 compatibility”.