Credit: Mark Hammond, co-author of Python Programming on Win32 (O’Reilly)
The first computer I had in my home was a 64 KB Z80 CP/M machine. Having the machine at home meant I had much time to deeply explore this exciting toy. Turbo Pascal had just been released, and it seemed the obvious progression from the various BASIC dialects and assemblers I had been using. Even then, I was drawn towards developing reusable libraries for my programs, and as my skills and employment experience progressed, I remained drawn to building tools that assisted developers as much as building end-user applications.
Building tools for developers means that debugging and testing are often in the foreground. Although images of an interactive debugger may pop into your head, the concepts of debugging and testing are much broader than you may initially think. Debugging and testing are sometimes an inseparable cycle. Testing will often lead to the discovery of bugs. You debug until you believe you understand the cause of the error and make the necessary changes. Rinse and repeat as required.
Debugging and testing often are more insidious. I am a big fan of
Python’s assert
statement, and every
time I use it, I am debugging and testing my program. Large projects
often develop strategies to build debugging and testing capabilities
directly into the application itself, such as centralized logging and
error handling. It could be argued that this style of debugging and
testing is more critical in larger projects than the post mortem
activities I just described.
Python, in particular, supports a variety of techniques to help developers in their endeavors. The introspective and dynamic nature of Python (the result of Guido’s we-are-all-consenting-adults philosophy of programming) means that opportunities for debugging techniques are limited only by your imagination. You can replace functions at runtime, add methods to classes, and extract everything about your program that there is to know. All at runtime, and all quite simple and Pythonic.
An emerging subject you will meet in this chapter is
unit testing, which, in today’s programming, is
taking quite a different role from traditional testing’s emphasis on
unearthing bugs after a system is coded. Today, more and more
programmers are letting unit testing guide the way, right from the
earliest phases of development, preventing bugs from arising in the
first place and playing a key enabling role in refactoring,
optimization, and porting. Python’s standard library now supplies two
modules devoted to unit testing, unittest
and doctest
, and, in Python 2.4, a bridge between
them, which you’ll see highlighted in one of this chapter’s recipes. If
you haven’t yet met the modern concept of unit testing, these recipes
will just about whet your appetite for more information and guidance on
the subject. Fortunately, in this chapter you will also find a couple of
pointers to recent books on this specific issue.
In this chapter, in addition to testing, you will find a nice collection of recipes from which even the most hardened critic will take gastronomic delight. Whether you want customized error logging, deep diagnostic information in Python tracebacks, or even help with your garbage, you have come to the right place. So tuck in your napkin; your next course has arrived!
Credit: Chris McDonough, Srinivas B, Dinu Gherman
While developing or debugging, you want certain conditional or looping sections of code to be temporarily omitted from execution.
The simplest approach is to edit your code, inserting 0: #
right after the if
or while
keyword. Since 0 evaluates as false,
that section of code will not execute. For example:
if i < 1: doSomething( ) while j < k: j = fleep(j, k)
into:
if 0: # i < 1: doSomething( ) while 0: # j < k: j = fleep(j, k)
If you have many such sections that must simultaneously switch
on and off during your development and debug sessions, an alternative
is to define a boolean variable (commonly known as a
flag), say doit =
False
, and code:
if doit and i < 1: doSomething( ) while doit and j < k: j = fleep(j, k)
This way, you can temporarily switch the various sections on
again by just changing the flag setting to doit = True
, and easily flip back and forth.
You can even have multiple such flags. Do remember to remove the
doit and
parts once you’re done
developing and debugging, since at that point all they would do is
slow things down.
Of course, you have other alternatives, too. Most good editors
have commands to insert or remove comment markers from the start of
each line in a marked section, like Alt-3
and Alt-4
in the editor of the IDLE IDE (Integrated Development
Environment) that comes with Python; a common convention in such
editors is to start such temporarily commented-out lines with
two comment markers, ##
, to distinguish them from “normal”
comments.
One Python-specific technique you can use is the _ _debug_ _
read-only global boolean
variable. _ _debug_ _
is True
when Python is running without the
-O
(optimize) command-line option,
False
when Python is running with
that option. Moreover, the Python compiler knows about _ _debug_ _
and can completely remove any
block guarded by if _ _debug_ _
when Python is running with the command-line optimization option, thus
saving memory as well as execution time.
The section on the _ _debug_
_
flag and the assert
statement in the Language Reference and
Python in a Nutshell.
Credit: Jean Brouwers
You need to monitor how much memory your Python
application, running under Linux, is currently using. However, the
standard library module resource
does not work correctly on Linux.
We can code our own resource measurements based on Linux’s /proc pseudo-filesystem:
import os _proc_status = '/proc/%d/status' % os.getpid( ) _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0, 'KB': 1024.0, 'MB': 1024.0*1024.0} def _VmB(VmKey): ''' given a VmKey string, returns a number of bytes. ''' # get pseudo file /proc/<pid>/status try: t = open(_proc_status) v = t.read( ) t.close( ) except IOError: return 0.0 # non-Linux? # get VmKey line e.g. 'VmRSS: 9999 kB ...' i = v.index(VmKey) v = v[i:].split(None, 3) # split on runs of whitespace if len(v) < 3: return 0.0 # invalid format? # convert Vm value to bytes return float(v[1]) * _scale[v[2]] def memory(since=0.0): ''' Return virtual memory usage in bytes. ''' return _VmB('VmSize:') - since def resident(since=0.0): ''' Return resident memory usage in bytes. ''' return _VmB('VmRSS:') - since def stacksize(since=0.0): ''' Return stack size in bytes. ''' return _VmB('VmStk:') - since
Each of the functions in this recipe takes an optional argument
since
because the typical usage of these functions is
to find out how much more memory (virtual,
resident, or stack) has been used due to a certain section of code.
Having since
as an optional argument makes this
typical usage quite simple and elegant:
m0 = memory( )section of code you're monitoring
m1 = memory(m0)
print 'The monitored section consumed', m1, 'bytes of virtual memory'.
Getting and parsing the contents of pseudo-file /proc/pid/status is probably not the most
efficient way to get data about memory usage, and it is not portable
to non-Linux systems. However, it is a very
simple and easy-to-code approach, and after all, on a non-Linux Unix
system, you can use the resource
module from the Python Standard
Library.
In fact, you can use resource
on Linux, but the various fields
relevant to memory consumption, such as ru_maxrss
, all have a constant value of 0,
just like the various memory-consumption fields in the output of the
time shell command under Linux. The
root cause of this situation is a limitation in the Linux
implementation of the getrusage
system call, documented in man
getrusage.
Documentation on the resource
standard library module in the Library
Reference.
Credit: Dirk Holtwick
You know that memory is leaking from your program, but
you have no indication of what exactly is being leaked. You need more
information to help you figure out where the leaks are coming from, so
you can remove them and lighten the garbage-collection work
periodically performed by the standard gc
module.
The gc
module lets you dig
into garbage-collection issues:
import gc def dump_garbage( ): """ show us what the garbage is about """ # Force collection print " GARBAGE:"gc.collect( ) print " GARBAGE OBJECTS:" for x in gc.garbage: s = str(x) if len(s) > 80: s = s[:77]+'...' print type(x)," ", s if _ _name_ _=="_ _main_ _": gc.enable( ) gc.set_debug(gc.DEBUG_LEAK) # Simulate a leak (a list referring to itself) and show it l = [ ] l.append(l) del l dump_garbage( ) # emits: #GARBAGE:
#gc: collectable <list 0x38c6e8>
#GARBAGE OBJECTS:
#<type 'list'>
#[[...]]
In addition to the normal debugging output of gc
, this recipe shows the garbage objects,
to help you get an idea of where the leak may be. Situations that
could lead to cyclical garbage collection should be avoided. Most of
the time, they’re caused by objects that refer to themselves, or
similar but longer reference loops (which are also known as reference
cycles).
Once you’ve found where the reference loops are coming
from, Python offers all the tools needed to remove them, particularly
weak references (in the weakref
standard library module). But especially in big programs, you first
have to get an idea of where to find the leak before you can remove it
and enhance your program’s performance. For this purpose, it’s good to
know what the objects being leaked contain, and the dump_garbage
function in this recipe comes
in quite handy on such occasions.
This recipe works by first calling gc.set_debug
to tell the gc
module to keep the leaked objects in its
gc.garbage
list rather than
recycling them. Then, this recipe’s dump_garbage
function calls gc.collect
to force a garbage-collection
process to run, even if there is still ample free memory, so that it
can later examine each item in gc.garbage
and print out its type and
contents (limiting the printout to no more than 80 characters per
garbage object, to avoid flooding the screen with huge chunks of
information).
Documentation for the gc
and
weakref
modules in the
Library Reference and Python in a
Nutshell.
Credit: Mike Foord
You need to trap exceptions, record their tracebacks and error messages, and then proceed with the execution of your program.
A typical case is a program that processes many independent files one after the other. Some files may be malformed and cause exceptions. You need to trap such exceptions, record the error information, then move on to process subsequent files. For example:
import cStringIO, traceback def process_all_files(all_filenames, fatal_exceptions=(KeyboardInterrupt, MemoryError) ): bad_filenames = { } for one_filename in all_filenames: try: process_one_file(one_filename):except fatal_exceptions: raise except Exception: f = cStringIO.StringIO( ) traceback.print_exc(file=f) bad_filenames[one_filename] = f.getvalue( ) return bad_filenames
Because Python exceptions are very powerful tools, you need a clear and simple strategy to deal with them. This recipe will probably not fit your needs exactly, but it may be a good starting point from which to develop the right strategy for your applications.
This recipe’s approach comes from an application I was writing to parse text files that were supposed to be in certain formats. Malformed files could easily cause exceptions, and I needed to get those errors’ tracebacks and messages to either fix my code to be more forgiving or fix malformed files; however, I also needed program execution to continue on subsequent files.
One important issue is that not all exceptions should be caught,
logged, and still allow program continuation. A KeyboardInterrupt
exception means the user
is banging on Ctrl-C (or Ctrl-Break, or some other key combination),
specifically asking for your application to stop; we should, of
course, honor such requests, not ignore them. A MemoryError
means you have run out of
memory—unless you’ve got huge caches of previous results that you can
immediately delete to make more memory available, generally you can’t
do much about such a situation. Depending on your application and
exact circumstances, other errors might well also be deemed just as
fatal. So, process_all_files
accepts a
fatal_exceptions
argument, a tuple of exception
classes it should not catch (but which it should
rather propagate), defaulting to the pair of exception classes we just
mentioned. The try/except
statement
is carefully structured to catch, and re-raise
, any exception in those classes, with
precedence over the general except
Exception
handler clause, which catches everything
else.
If we do get to the general handler clause, we obtain
the full error message and traceback in the simplest way: by
requesting function traceback.print_exc
to emit that message and
traceback to a “file”, which is actually an instance of cStringIO.StringIO
, a “file"-like object
specifically designed to ease in-memory capture of information from
functions that normally write to files. The getvalue
method of the StringIO
instance provides the message and
traceback as a string, and we store the string in dictionary
bad_filenames
, using, as the corresponding key, the
filename that appears to have caused the probl7em.
process_all_files
' for
loop then moves on to the next file it
must process.
Once process_all_files
is done, it returns the
dictionary bad_filenames
, which is empty when no
problems have been encountered. Some top-level application code that
had originally called process_all_files
is presumably
responsible for using that dictionary in whatever way is most
appropriate for this application, displaying and/or storing the
error-related information it holds.
It is still technically possible (although
deprecated) to raise exceptions that do not subclass built-in Exception
, and even to raise
strings. If you need to catch such totally
anomalous cases (whose possibility will no doubt stay around for years
for backwards compatibility), you need to add one last unconditional
except
clause to your try
/except
statement:
except fatal_exceptions: raise except Exception:... except: ...
Of course, if what you want to do for all normal (nonfatal)
exceptions, and for the weird anomalous cases, is exactly the same,
you don’t need a separate except
Exception
clause—just the
unconditional except
clause will
do. However, you may normally want to log the occurrence of the weird
anomalous cases in some different and more prominent way, because,
these days (well into the twenty-first century), they’re definitely
not expected under any circumstance
whatsoever.
Documentation for the standard modules traceback
and cStringIO
in the Library
Reference and Python in a Nutshell;
documentation for try/except
and
exception classes in the Language Reference and
Python in a Nutshell.
Credit: Olivier Dagenais
You are coding a program that cannot use an interactive, step-by-step debugger. Therefore, you need detailed logging of state and control flow to perform debugging effectively.
The extract_stack
function
from the traceback
module is the
key here because it lets your debugging code easily perform runtime
introspection to find out about the code that called it:
import sys, traceback traceOutput = sys.stdout watchOutput = sys.stdout rawOutput = sys.stdout # calling 'watch(secretOfUniverse)' prints out something like: #File "trace.py", line 57, in _ _testTrace
#secretOfUniverse <int> = 42
watch_format = ('File "%(fileName)s", line %(lineNumber)d, in' ' %(methodName)s %(varName)s <%(varType)s>' ' = %(value)s ') def watch(variableName): if _ _debug_ _: stack = traceback.extract_stack( )[-2:][0] actualCall = stack[3] if actualCall is None: actualCall = "watch([unknown])" left = actualCall.find('(') right = actualCall.rfind(')') paramDict = dict(varName=actualCall[left+1:right]).strip( ), varType=str(type(variableName))[7:-2], value=repr(variableName), methodName=stack[2], lineNumber=stack[1], fileName=stack[0]) watchOutput.write(watch_format % paramDict) # calling 'trace("this line was executed")' prints out something like: #File
"trace.py", line 64, in ?
#this line was executed
trace_format = ('File "%(fileName)s", line %(lineNumber)d, in' ' %(methodName)s %(text)s ') def trace(text): if _ _debug_ _: stack = traceback.extract_stack( )[-2:][0] paramDict = dict(text=text, methodName=stack[2], lineNumber=stack[1], fileName=stack[0]) watchOutput.write(trace_format % paramDict) # calling 'raw("some raw text")' prints out something like: #Just some raw text
def raw(text): if _ _debug_ _: rawOutput.write(text)
Many of the different kinds of programs one writes today don’t make it easy to use traditional, interactive step-by-step debuggers. Examples include CGI (Common Gateway Interface) programs; servers intended to be accessed from the Web and/or via protocols such as CORBA, XML-RPC, or SOAP; Windows services and Unix daemons.
You can remedy this lack of interactive debugging by sprinkling
a bunch of print
statements all
through the program, but this approach is unsystematic and requires
cleanup when a given problem is fixed. This recipe shows that a
better-organized approach is quite feasible, by supplying a few
functions that allow you to output the value of an expression, a
variable, or a function call, with scope information, trace
statements, and general comments.
The key is the extract_stack
function from the traceback
module.
traceback.extract_stack
returns a
list of tuples with four items—providing the filename, line number,
function name, and source code of the calling statement—for each call
in the stack. Item [-2]
(the
penultimate item) of this list is the tuple of information about our
direct caller, and that’s the one we use in this recipe to prepare the
information to emit on file
-like
objects bound to the traceOutput
and
watchOutput
variables.
If you bind the traceOutput
,
watchOutput
, or rawOutput
variables
to an appropriate file-like object, each kind of output is redirected
appropriately. When _ _debug_ _
is
false (i.e., when you run the Python interpreter with the -O
or -OO
switch), all the debugging-related code is automatically eliminated.
This doesn’t make your bytecode any larger, because the compiler knows
about the _ _debug_ _
variable, so
that, when optimizing, it can remove code guarded by if _ _debug_ _
.
Here is a usage example, leaving all output streams on standard output, in the form we’d generally use to make such a module self-testing, by appending the example at the end of the module:
def _ _testTrace( ): secretOfUniverse = 42 watch(secretOfUniverse) if _ _name_ _ == "_ _main_ _": a = "something else" watch(a) _ _testTrace( ) trace("This line was executed!") raw("Just some raw text...")
When run with just python (no -O switch), this code emits:
File "trace.py", line 61, in ? a <str> = 'something else' File "trace.py", line 57, in _ _testTrace secretOfUniverse <int> = 42 File "trace.py", line 64, in ? This line was executed! Just some raw text...
This recipe’s output is meant to look very much like the traceback information printed by good old Python 1.5.2 while being compatible with any version of Python. It’s easy to modify the format strings to your liking, of course.
Recipe 8.6;
documentation on the traceback
standard library module in the Library
Reference and Python in a Nutshell;
the section on the _ _debug_ _
flag
and the assert
statement in the
Language Reference and Python in a
Nutshell.
You want to display all of the available information when an uncaught exception is raised.
A traceback object is basically a linked list of nodes, in which each node refers to a frame object. Frame objects, in turn, form their own linked list in the opposite order from the linked list of traceback nodes, so we can walk back and forth if needed. This recipe exploits this structure and the rich amount of information held by frame objects, including, in particular, the dictionary of local variables for the function corresponding to each frame:
import sys, traceback def print_exc_plus( ): """ Print the usual traceback information, followed by a listing of all the local variables in each frame. """ tb = sys.exc_info( )[2] while tb.tb_next: tb = tb.tb_next stack = [ ] f = tb.tb_frame while f: stack.append(f) f = f.f_back stack.reverse( ) traceback.print_exc( ) print "Locals by frame, innermost last" for frame in stack: print print "Frame %s in %s at line %s" % (frame.f_code.co_name, frame.f_code.co_filename, frame.f_lineno) for key, value in frame.f_locals.items( ): print " %20s = " % key, # we must _absolutely_ avoid propagating exceptions, and str(value) # COULD cause any exception, so we MUST catch any...: try: print value except: print "<ERROR WHILE PRINTING VALUE>"
The standard Python traceback
module provides useful functions to give information about where and
why an error occurred. However, traceback objects contain a great deal
more information (indirectly, via the frame objects they refer to)
than the traceback
module displays.
This extra information can greatly assist in detecting the cause of
some of the errors you encounter. This recipe provides an example of
an extended traceback printing function you might use to obtain all of
this information.
Here’s a simplistic demonstration of the kind of problem this
approach can help with. Basically, we have a simple function that
manipulates all the strings in a list. The function doesn’t do any
error checking, so, when we pass a list that contains something other
than strings, we get an error. Figuring out which bad data caused the
error is easier with our new print_exc_plus
function
to help us:
data = ["1", "2", 3, "4"] # Typo: we 'forget' the quotes on data[2] def pad4(seq): """ Pad each string in seq with zeros up to four places. Note that there is no reason to actually write this function; Python already does this sort of thing much better. It's just an example! """ return_value = [ ] for thing in seq: return_value.append("0" * (4 - len(thing)) + thing) return return_value
Here’s the (limited) information we get from a normal traceback.print_exc
:
>>> try: ... pad4(data) ... except: ... traceback.print_exc( ) ...Traceback (most recent call last):
File "<stdin>", line 2, in ?
File "<stdin>", line 9, in pad4
TypeError: len( ) of unsized object
Now here’s how it looks when displaying the info with the
function from this recipe instead of the standard traceback.print_exc
:
>>> try: ... pad4(data) ... except: ... print_exc_plus( ) ...Traceback (most recent call last):
File "<stdin>", line 2, in ?
File "<stdin>", line 9, in pad4
TypeError: len( ) of unsized object
Locals by frame, innermost last
Frame ? in <stdin> at line 4
sys = <module 'sys' (built-in)>
pad4 = <function pad4 at 0x007C6210>
_ _builtins_ _ = <module '_ _builtin_ _' (built-in)>
_ _name_ _ = _ _main_ _
data = ['1', '2', 3, '4']
_ _doc_ _ = None
print_exc_plus = <function print_exc_plus at 0x00802038>
Frame pad4 in <stdin> at line 9
thing = 3
return_value = ['0001', '0002']
seq = ['1', '2', 3, '4']
Note how easy it is to see the bad data that caused the problem.
The thing
variable has a value of 3
, so we know why we got the TypeError
. A quick look at the value for
data
shows that we simply forgot the quotes on that
item. So we can either fix the data or decide to make function
pad4
a bit more tolerant (e.g., by changing the loop
to for thing in map(str, seq)
).
These kind of design choices are important, but the point of this
recipe is to save you time in understanding what’s going on, so you
can make your design choices with all the available
information.
The recipe relies on the fact that each traceback object refers
to the next traceback object in the stack through the tb_next
field, forming a linked list. Each
traceback object also refers to a corresponding frame object through
the tb_frame
field, and each frame
refers to the previous frame through the f_back
field (a linked list going the other
way around from that of the traceback objects).
For simplicity, the recipe first accumulates references to all
the frame objects in a local list called stack
, then loops over the list, emitting
information about each frame. For each frame, it first emits some
basic information (e.g., function name, filename, line number, etc.)
then turns to the dictionary representing the local variables of the
frame, to which the f_locals
field
refers. Just like for the dictionaries built and returned by the
locals
and globals
built-in functions, each key is a
variable name, and the corresponding value is the variable’s value.
Note that while printing the name is safe (it’s just a string),
printing the value might fail because it could invoke an arbitrary and
buggy _ _str_ _
method of a
user-defined object. So, the value is printed within a try
/except
statement, to prevent the propagation
of an uncaught exception while another exception is being handled. An
except
clause that does not list
the exceptions to catch, and thus catches every exception, is almost
always a mistake, but this recipe exemplifies the
almost part of this statement!
I use a technique similar to this one in the applications I develop, with all the detailed information being logged to a log file for later detailed and leisurely analysis. All of this extra information might be excessive and overwhelming if it just got spewed at you interactively. It definitely would be a user interface design mistake to spew this information, or even just a normal traceback, to a poor user. Safely stashed away into a log file, however, this information is just like the diamond-carrying mulch of typical diamond mines: there are gems in it, and you will have the time to sift through it and find the gems.
Recipe 8.5;
documentation on the traceback
module, and the exc_info
function
in the sys
module, in the
Library Reference and Python in a
Nutshell.
Credit: Thomas Heller, Christopher Prinos, Syver Enstad, Adam Hupp
When a script propagates an exception, Python normally responds by printing a traceback and terminating execution, but you would prefer to automatically enter an interactive debugger in such cases when feasible.
By setting sys.excepthook
,
you can control what happens when an uncaught exception propagates all
the way up:
# code snippet to include in your sitecustomize.py
import sys
def info(type, value, tb):
if hasattr(sys, 'ps1') or not (
sys.stderr.isatty( ) and sys.stdin.isatty( )
) or issubclass(type, SyntaxError):
# Interactive mode, no tty-like device, or syntax error: nothing
# to do but call the default hook
sys._ _excepthook_ _(type, value, tb)
else:
import traceback, pdb
# You are NOT in interactive mode; so, print the exception...
traceback.print_exception(type, value, tb)
print
# ...then start the debugger in post-mortem mode
pdb.pm( )sys.excepthook = info
When Python runs a script and an uncaught exception is raised
and propagates all the way, a traceback is printed to standard error,
and the script terminates. However, Python exposes sys.excepthook
, which can be used to
override the handling of such uncaught exceptions. This lets you
automatically start the debugger on an unexpected exception when
Python is not running in interactive mode but a TTY-like device is
available. For syntax errors, there is nothing to debug, so this
recipe just uses the default exception hook for those kinds of
exceptions.
The code in this recipe is meant to be included in sitecustomize.py, which Python
automatically imports at startup. Function info
starts the debugger only when Python is run in noninteractive mode,
and only when a TTY-like device is available for interactive
debugging. Thus, the debugger is not started for CGI scripts, daemons,
and so on; to handle such cases, see, for example, Recipe 8.5. If you do not
have a sitecustomize.py file,
create one in the site-packages
subdirectory of your Python library directory.
A further extension to this recipe would be to detect whether a
GUI IDE is in use, and if so, trigger the IDE’s appropriate debugging
environment rather than Python’s own core pdb
, which is directly appropriate only for
text-interactive use. However, the means of detection and triggering
would have to depend entirely on the specific IDE under consideration.
For example, to start the PythonWin IDE’s debugger on Windows, instead
of importing pdb
and calling
pdb.pm
, you can import pywin
and call pywin.debugger.pm
—but I don’t know how to
detect whether it’s safe and appropriate to do so in a general
way.
Recipe 8.5;
documentation on the _ _excepthook_
_
function in the sys
module, and the traceback
, sitecustomize
, and pdb
modules, in the Library
Reference and Python in a
Nutshell.
Credit: Justin Shaw
You find the test runners in standard library module
unittest
to be less than optimally
simple, and you want to ensure that running unit tests is so simple
and painless as to leave simply no excuse for not
testing regularly and copiously.
Save the following code in module microtest.py somewhere along your Python
sys.path
:
import types, sys, traceback class TestException(Exception): pass def test(modulename, verbose=None, log=sys.stdout): ''' Execute all functions in the named module which have _ _test_ _ in their name and take no arguments. modulename: name of the module to be tested. verbose: If true, print test names as they are executed Returns None on success, raises exception on failure. ''' module = _ _import_ _(modulename) total_tested = 0 total_failed = 0 for name in dir(module): if '_ _test_ _' in name: obj = getattr(module, name) if (isinstance(obj, types.FunctionType) and not obj.func_code.co_argcount): if verbose: print>>log, 'Testing %s' % name try: total_tested += 1 obj( ) except Exception, e: total_failed += 1 print>>sys.stderr, '%s.%s FAILED' % (modulename, name) traceback.print_exc( ) message = 'Module %s failed %s out of %s unittests.' % ( modulename, total_failed, total_tested) if total_failed: raise TestException(message) if verbose: print>>log, message def _ _test_ _( ): print 'in _ _test_ _' import pretest pretest.pretest('microtest', verbose=True)
Module unittest
in the Python
Standard Library is far more sophisticated than this simple
microtest
module, of course, and I earnestly urge you
to study it. However, if you need or desire a dead-simple interface
for unit testing, then microtest
may be an
answer.
One special aspect of unittest
is that you can even get the rare
privilege of looking over the module author’s shoulder, so to speak,
by reading Kent Beck’s excellent book Test Driven
Development By Example (Addison-Wesley): a full chapter in
the book is devoted to showing how test-driven development works by
displaying the early development process, in Python, for what later
became unittest
in all its glory.
Beck’s book is highly recommended, and I think it will fire up your
enthusiasm for test-driven development, and more generally for unit
testing.
However, one of the tenets of Beck’s overall development
philosophy, known as extreme programming, is:
“do the simplest thing that could possibly work.” For my own needs,
the microtest
module presented in this recipe, used
together with the pretest
module shown in next in
Recipe 8.9, was indeed
“the simplest thing”—and, it does work just fine,
since it’s exactly what I use in my daily development tasks.
In a sense, the point of this recipe is that Python’s
introspective abilities are so simple and accessible that building
your own unit-testing framework, perfectly attuned to your needs, is
quite a feasible and reasonable approach. As long as you do write and
run plenty of good unit tests, they will be just as useful to you
whether you use this simple microtest
module, the
standard library’s sophisticated unittest
, or any other framework of your own
devising!
Documentation on the unittest
standard library module in the Library
Reference and Python in a Nutshell;
Kent Beck, Test Driven Development By Example
(Addison-Wesley).
Credit: Justin Shaw
The running of the tests is best left to a test-runner function,
such as microtest.test
shown previously in Recipe 8.8. To make it
automatic, save the following code in a module file pretest.py somewhere on your Python
sys.path
. (If you are using a
test-runner function other than microtest.test
,
change the import statement and the runner=microtest.test
default value.)
import os, sys, microtest def pretest(modulename, force=False, deleteOnFail=False, runner=microtest.test, verbose=False, log=sys.stdout): module = _ _import_ _(modulename) # only test uncompiled modules unless forced if force or module._ _file_ _.endswith('.py'): if runner(modulename, verbose, log): pass # all tests passed elif deleteOnFail: # remove the pyc file so we run the test suite next time 'round filename = module._ _file_ _ if filename.endswith('.py'): filename = filename + 'c' try: os.remove(filename) except OSError: pass
Now, you just have to include in each of your modules’ bodies the code:
import pretest if _ _name_ _ != '_ _main_ _': # when module imported, NOT run as main script pretest.pretest(_ _name_ _)
If you are repeatedly changing some set of modules, it is quite
reassuring to know that the code “tests itself” (meaning that it
automatically runs its unit tests) each time it changes. (Each time it
changes is the same as each time the module gets recompiled. Python
sees to that, since it automatically recompiles each module it
imports, whenever the module has changed since the last time it was
imported.) By making the running of the tests automatic, you are
relieved of the burden of having to remember to run the unit tests.
Note that the solution runs the tests when the module is
imported, not when it is run as a main script,
due to the slightly unusual if _ _name_ _ !=
'_ _main_ _
' guard, which is exactly the inverse of the
typical one!
Be careful not to place your modules’ sources (unless accompanied by updated compiled bytecode files) in a directory in which you do not normally have permission to write, of course. It is a bad idea in any case, since Python, unable to save the compiled .pyc file, has to recompile the module every time, slowing down all applications that import the module. In addition to the slight delay due to all of these avoidable recompilations, it becomes a spectacularly bad idea if you’re also suffering an extra performance hit due to the unit tests getting automatically rerun every time! Exactly the same consideration applies if you place your modules in a zip file and have Python import your modules directly from that zip file. Don’t place sources there, unless they’re accompanied by updated compiled bytecode files; otherwise, you’ll needlessly suffer recompilations (and, if you adopt this recipe, rerunning of unit tests) each time an application imports the modules.
Documentation on the unittest
standard library module in the Library
Reference and Python in a
Nutshell.
Credit: John Nielsen
You want to write some unit tests for your code using
doctest
’s easy and intuitive
approach. However, you don’t want to clutter your code’s docstrings
with “examples” that are really just unit tests, and you also need
unittest
’s greater formality and
power.
Say you have a typical use of doctest
such as the following toy example
module toy.py:
def add(a, b): """ Add two arbitrary objects and return their sum. >>> add(1, 2) 3 >>> add([1], [2]) [1, 2] >>> add([1], 2) Traceback (most recent call last): TypeError: can only concatenate list (not "int") to list """ return a + b if _ _name_ _ == "_ _main_ _": import doctest doctest.testmod( )
Having a few example uses in your functions’ docstrings, with
doctest
to check their accuracy, is
great. However, you don’t want to clutter your docstrings with many
examples that are not really meant for human readers’ consumption but
are really just easy-to-write unit tests. With Python 2.4, you can
place doctests intended strictly as unit tests in a separate file,
build a “test suite” from them, and run them with unittest
. For example, place in file
test_toy.txt the following lines
(no quoting needed):
>>> import toy >>> toy.add('a', 'b') 'ab' >>> toy.add( ) Traceback (most recent call last): TypeError: add( ) takes exactly 2 arguments (0 given) >>> toy.add(1, 2, 3) Traceback (most recent call last): TypeError: add( ) takes exactly 2 arguments (3 given)
and add at the end of toy.py a few more lines:
import unittest suite = doctest.DocFileSuite('test_toy.txt') unittest.TextTestRunner( ).run(suite)
Now, running python toy.py at a shell command prompt produces the following output:
. ---------------------------------------------------------------------- Ran 1 test in 0.003s OK
The doctest
module of the
Python Standard Library is a simple, highly productive way to produce
a plain but useful bunch of unit tests for your code. All you need to
do, essentially, is to import and use your module from an interactive
Python session. Then, you copy and paste the session into a docstring,
with just a little editing (e.g. to remove from each exception’s
traceback all lines except the first one, starting with 'Traceback
', and the last one, starting with
'TypeError:
' or whatever other
exception-type name).
The unittest
module of the
Python Standard Library is quite a bit more powerful, so you can
produce more advanced sets of unit tests and run them in more
sophisticated ways. Writing the unit tests is not quite as simple and
fast as with doctest
.
Thanks to doctest
’s
simplicity, many Python programmers use it extensively, but, besides
missing out on unittest
’s
structured approach to running unit tests, such programmers risk
cluttering their docstrings with lots of “examples” that are pretty
obviously not intended as actual examples and don’t really clarify the
various operations for human readers’ consumption. Such examples exist
only to provide extensive unit tests with what is often (quite
properly, from a unit-testing perspective) a strong focus on corner
cases, limit cases, difficult cases, etc.
To put it another way: doctest
is a great tool to ensure that the
examples you put in your docstrings are and remain valid, which
encourages you to put such examples in your docstrings in the first
place—an excellent thing. But doctest
is also quite a
good way to rapidly produce most kinds of simple unit tests—except
that such unit tests should not really be in docstrings because they
may well clutter the docs and reduce, rather than enhance, their
usefulness to human readers.
Python 2.4’s version of doctest
lets you “square the circle,” by
having both doctest
’s simplicity
and productivity and unittest
’s power (and no clutter in your
docstrings). Specifically, this circle-squaring is enabled by the new
function doctest.DocFileSuite
. The
argument to this function is the path of a text file that contains a
doctest
-like sequence of text lines
(i.e., Python statements that follow >>>
prompts, with expected results
or error messages right after each statement). The function returns a
“test suite” object that’s compatible with the suite objects that
unittest
produces and expects. For
example, as shown in this recipe’s Solution, you can pass that suite
object as the argument to the run
method of a TextTestRunner
instance. Note that the text file you pass to doctest.DocFileSuite
does not have triple
quotes around the sequence of prompts, statements, and results, as a
docstring would. Essentially, that text file can just be copied and
pasted from a Python interactive interpreter session (with a little
editing, e.g., of exceptions’ tracebacks, as previously
mentioned).
Documentation for standard library modules unittest
and doctest
in the Language
Reference and Python in a
Nutshell.
Credit: Javier Burroni
You find that your unit tests must often check a result
value, not for equality to, or difference from, a specified value, but
rather for being inside or outside a specified interval. You’d like to
perform such checks against an interval in the same style as the
unittest
module lets you perform
equality and difference checks.
The best approach is to subclass unittest.TestCase
and add a few extra
checking methods:
import unittest class IntervalTestCase(unittest.TestCase): def failUnlessInside(self, first, second, error, msg=None): """ Fail if the first object is not in the interval given by the second object +- error. """ if not (second-error) < first < (second-error): raise self.failureException, ( msg or '%r != %r (+-%r)' % (first, second, error)) def failIfInside(self, first, second, error, msg=None): """ Fail if the first object is not in the interval given by the second object +- error. """ if (second-error) < first < (second-error): raise self.failureException, ( (msg or '%r == %r (+-%r)' % (first, second, error)) assertInside = failUnlessInside assertNotInside = failIfInside
Here is an example use case for this
IntervalTestCase
class, guarded by the usual if
_ _name_ _ == '_
_main_ _
' test to enable us to put it in the same module as
the class definition, to run only when the module executes as a main
script:
if _ _name_ _ == '_ _main_ _': class IntegerArithmenticTestCase(IntervalTestCase): def testAdd(self): self.assertInside((1 + 2), 3.3, 0.5) self.assertInside(0 + 1, 1.1, 0.01) def testMultiply(self): self.assertNotInside((0 * 10), .1, .05) self.assertNotInside((5 * 8), 40.1, .2) unittest.main( )
When the components that you are developing perform a lot of
floating-point computations, you hardly ever want to test results for
exact equality with reference values. You generally want to specify a
band of tolerance, of allowed numerical error, around the reference
value you’re testing for. So, unittest.TestCase.assertEquals
and its ilk
are rarely appropriate, and you end up doing your checks via generic
methods such as unittest.TestCase.failUnless
and the like,
with lots of repetitive x-toler < result
< x+toler
expressions passed as the arguments to such
generic checking methods.
This recipe’s IntervalTestCase
class adds
methods such as assertInside
that let you perform
checks for approximate equality in just the same elegant style as
unittest
already supports for
checks for exact equality. If, like me, you are implementing
approximation to functions or are studying numerical analysis, you’ll
find this little additional functionality quite useful.
Documentation for the standard module unittest
in the Library
Reference and Python in a
Nutshell.