Credit: David Beazley, University of Chicago
One of Python’s most powerful features is its ability to be hooked to libraries and programs written in classic compiled languages such as C, C++, and Fortran. A large number of Python’s built-in library modules are written as extension modules in C so that operating system services, networking functions, databases, and other features can be easily accessed from the interpreter. In addition, a number of application programmers write extensions in order to use Python as a framework for controlling large software packages coded in other languages.
The gory details of how Python interfaces with other languages can be found in various Python programming books, as well as online documentation at www.python.org (directory Demo, distributed as part of the Python source distribution, also contains several useful examples). However, the general approach revolves around the creation of special wrapper functions that hook into the interpreter. For example, if you had a C function like this:
int gcd(int x, int y) { int g = y; while (x > 0) { g = x; x = y % x; y = g; } return g; }
and you wanted to access it from Python in a module named
spam
, you’d write some special wrapper code
like this:
#include "Python.h" extern int gcd(int, int); PyObject *wrap_gcd(PyObject *self, PyObject *args) { int x, y, g; if(!PyArg_ParseTuple(args, "ii", &x, &y)) return NULL; g = gcd(x, y); return Py_BuildValue("i", g); } /* List of all functions in the module */ static PyMethodDef spammethods[ ] = { {"gcd", wrap_gcd, METH_VARARGS }, { NULL, NULL } }; /* Module initialization function */ void initspam(void) { Py_InitModule("spam", spammethods); }
Once this code is compiled into an extension module, you can use
the gcd
function just as you would expect. For
example:
>>> import spam >>> spam.gcd(63,56)7
>>> spam.gcd(71,89)1
This short example extends in a natural way to larger programming libraries—each function that you want to access from Python simply gets its own wrapper.
Although writing simple extension functions is fairly straightforward, writing many wrappers quickly becomes tedious and prone to error if you are building anything of reasonable complexity. Therefore, a lot of programmers rely on automatic module building tools to simplify the process. Python is fortunate to have a variety of such tools, many of which are listed below:
bgen is a module-building tool that can be found in the Tools directory of a standard Python distribution. Maintained by Jack Jansen, it is used to generate many of the extension modules available in the Macintosh version of Python, but it is not Mac specific.
pyfort is a tool developed by Paul Dubois that can be used to build extension modules for Fortran code. Details are available at the following web page: http://pyfortran.sourceforge.net.
f2py is a wrapper generator for creating extensions in Fortran 90/95 that has been developed by Pearu Peterson. Details are available at http://cens.ioc.ee/projects/f2py2e/.
SIP is a C++ module builder developed by Phil Thompson that
creates wrappers for C++ classes. The system has most notably been
used to create the PyQt
and
PyKDE
extension modules. More
information can be found at http://www.thekompany.com/projects/pykde.
WrapPy is another C++ module builder that produces extension modules by reading C++ header files. It is developed by Greg Couch and is available at http://www.cgl.ucsf.edu/home/gregc/wrappy/index.html.
Boost Python Library, developed by David Abrahams, provides one of the most powerful and unusual C++ wrapping techniques. Classes are automatically wrapped into Python extensions by simply writing a few additional C++ classes that specify information about the extension module. More information is available at http://www.boost.org/libs/python/doc/.
SWIG (Simplified Wrapper and Interface Generator) is an automatic extension-building tool that reads annotated C and C++ header files and produces extension modules for Python, Tcl, Perl, and a variety of other high-level languages such as Scheme, Ruby, Java, OCAML (Objective Caml), and C#. SWIG is able to wrap a large subset of C++ language features into a Python extension module. However, since I developed SWIG, I may be a little biased :-). In any event, further details are available at http://www.swig.org.
Pyrex is a language for writing Python extension modules, developed by Greg Ewing. The Pyrex language is a large subset of Python, with semantics slightly less fluidly dynamic than Python, and the addition of a few language constructs (particularly optional declarations of types of parameters and variables) that enables the Pyrex compiler to generate fast C code. Further details are available at http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/.
Regardless of the approach used to build Python extension modules,
certain important topics remain somewhat mysterious to many extension
programmers. The recipes in this chapter describe some of the common
problems and extension-building tricks that are rarely covered in the
standard documentation or other Python books. Topics include interacting
with threads, returning NULL
values,
accessing Python sequences and iterables, creating extension types, and
debugging.
One recipe, in particular, highlights an especially important
topic: you don’t necessarily have to use other languages (even one as
close to Python as Pyrex is) to write Python extensions to access
functionality that’s available through dynamically loaded libraries
(.DLLs on Windows, .sos on Linux, .dylib on Mac OS X, etc.). It often may be
sufficient to use existing third-party general-purpose extensions, such
as the classic calldll
or the newer
ctypes
packages, which enable you to
wrap such dynamic libraries and make their functionality available to
your Python programs, by writing just a little pure Python code.
Credit: Alex Martelli
You want to code and build a C extension type for Python with a minimal amount of hard work.
First of all, we need to create a setup.py file to use the distutils
package to build and install our
module:
from distutils.core import setup, Extension setup(name = "elemlist", version = "1.0", maintainer = "Alex Martelli", maintainer_email = "[email protected]", description = "Sample, simple Python extension module", ext_modules = [Extension('elemlist',sources=['elemlist.c'])] )
Then, we need a file elemlist.c with our module’s source code:
#include "Python.h" /* type-definition and utility-macros */ typedef struct { PyObject_HEAD PyObject *car, *cdr; } cons_cell; staticforward PyTypeObject cons_type; /* a type-testing macro (we don't actually use it here) */ #define is_cons(v) ((v)->ob_type == &cons_type) /* utility macros to access car and cdr, as either lvalues or rvalues */ #define carof(v) (((cons_cell*)(v))->car) #define cdrof(v) (((cons_cell*)(v))->cdr) /* ctor ("internal" factory-function) and dtor */ static cons_cell* cons_new(PyObject *car, PyObject *cdr) { cons_cell *cons = PyObject_New(cons_cell, &cons_type); if(cons) { cons->car = car; Py_INCREF(car); /* INCREF when holding a PyObject */ cons->cdr = cdr; Py_INCREF(cdr); /* ditto */ } return cons; } static void cons_dealloc(cons_cell* cons) { /* DECREF when releasing previously-held PyObject*'s */ Py_DECREF(cons->car); Py_DECREF(cons->cdr); PyObject_Del(cons); } /* A minimal Python type-object */ statichere PyTypeObject cons_type = { PyObject_HEAD_INIT(0) /* initialize to 0 to ensure Win32 portability */ 0, /* ob_size */ "cons", /* tp_name */ sizeof(cons_cell), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor)cons_dealloc, /* tp_dealloc */ /* implied by ISO C: all zeros thereafter, i.e., no other method */ }; /* module-functions */ static PyObject* cons(PyObject *self, PyObject *args) /* the exposed factory-function */ { PyObject *car, *cdr; if(!PyArg_ParseTuple(args, "OO", &car, &cdr)) return 0; return (PyObject*)cons_new(car, cdr); } static PyObject* car(PyObject *self, PyObject *args) /* car-accessor */ { PyObject *cons; if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons)) /* type-checked */ return 0; return Py_BuildValue("O", carof(cons)); } static PyObject* cdr(PyObject *self, PyObject *args) /* cdr-accessor */ { PyObject *cons; if(!PyArg_ParseTuple(args, "O!", &cons_type, &cons)) /* type-checked */ return 0; return Py_BuildValue("O", cdrof(cons)); } static PyObject* setcar(PyObject *self, PyObject *args) /* car-setter */ { PyObject *cons; PyObject *value; if(!PyArg_ParseTuple(args, "O!O", &cons_type, &cons, &value)) return 0; Py_INCREF(value); Py_DECREF(carof(cons)); carof(cons) = value; return Py_BuildValue(""); } static PyObject* setcdr(PyObject *self, PyObject *args) /* cdr-setter */ { PyObject *cons; PyObject *value; if(!PyArg_ParseTuple(args, "O!O", &cons_type, &cons, &value)) return 0; Py_INCREF(value); Py_DECREF(cdrof(cons)); cdrof(cons) = value; return Py_BuildValue(""); } static PyMethodDef elemlist_module_functions[ ] = { {"cons", cons, METH_VARARGS}, {"car", car, METH_VARARGS}, {"cdr", cdr, METH_VARARGS}, {"setcar", setcar, METH_VARARGS}, {"setcdr", setcdr, METH_VARARGS}, {0, 0} }; /* module entry-point (module-initialization) function */ void initelemlist(void) { /* Create the module, with its functions */ PyObject *m = Py_InitModule("elemlist", elemlist_module_functions); /* Finish initializing the type-objects */ cons_type.ob_type = &PyType_Type; }
C-coded Python extension types have an undeserved aura of mystery and difficulty. Sure, it’s a lot of work to implement every possible feature, but a minimal yet useful type doesn’t necessarily take all that much effort.
This module is roughly equivalent to the Python-coded module:
def cons(car, cdr): return car, cdr def car(conscell): return conscell[0] def cdr(conscell): return conscell[1] def setcar(conscell, value): conscell[0] = value def setcdr(conscell, value): conscell[1] = value
except that the C source is about 25 times larger, even excluding comments and empty lines (and it is not much faster than the Python-coded version, either).
However, the point of this recipe is to demonstrate a minimal
C-coded extension type. I’m not even supplying object methods (except
the indispensable destructor) but, rather, I am providing module-level
functions to build cons
cells and
to read and write their car
and cdr
fields. This recipe also shows the utter simplicity of building a
C-coded extension module on any platform, thanks to the distutils
package, which does all of the
hard work.
Lisp-savvy readers will have recognized from the names involved
that this little extension offers the core functionality to implement
a Lisp-like linked list type—using some NIL
marker (e.g. None
), by convention, as the cdr
of the last cons
-cell of a list, and otherwise "cons
ing up a list” by having every cdr
be another con
s-cell. You might easily
constrain the cdr
to be either None
or another cons
-cell, giving up on generality for a bit
of extra error checking.
Because this recipe is meant as an introduction to writing extension modules in C for Python, here are the instructions for building this extension module, assuming you have a Windows machine with Python 2.3 and Microsoft Visual C++ 6 (or the free command-line equivalent that you can download from Microsoft’s site as a part of their .NET Framework SDK). You can presumably translate mentally to other platforms such as Linux with gcc, Mac OS X with gcc, and so on. On the other hand, using different C compilers on Windows involves more work, and I’m not going to cover that here (see http://sebsauvage.net/python/mingw.html).
Here are the steps you should follow to build this recipe’s extension:
Make a new directory—for example, C:TempEL.
Open a command-prompt window, and go to the new directory.
In the new directory, create the files setup.py and elemlist.c with the contents of the recipe’s text.
Run the following at the command prompt (assuming you’ve performed a standard Python 2.3 installation, so that your python.exe lives in C:Python23):
<m>C:TempEL> C:Python23python setup.py install</m>
This command will result in lots of output, which you should
examine to check for problems. Presumably, all has gone well, and
the new elemlist
extension module has been built
and installed.
Now try the extension by running the following at the DOS prompt:
<m>C:TempEL> C:Python23python</m>(snipped: various greeting messages from Python)
>>> from elemlist import cons, car, cdr
>>> a = cons(1, cons(2, cons(3, ( ))))
>>> car(cdr(a))
2
>>>
There—your new extension module is installed and ready!
The Extending and Embedding manual is
available as part of the standard Python documentation set at
http://www.python.org/doc/current/ext/ext.html;
the section “Distributing Python Modules” of the standard Python
documentation set is still incomplete, but it’s a reliable source of
information on the distutils
package. Python in a Nutshell covers the
essentials of extending and embedding and of the distutils
package.
Credit: Alex Martelli
The Pyrex language is the simplest and fastest way to build
Python extensions. Once we have Pyrex installed, the next step is to
create a setup.py file to use the
distutils
package to build and
install our module:
from distutils.core import setup, Extension from Pyrex.Distutils import build_ext setup(name = "elemlist", version = "1.0", maintainer = "Alex Martelli", maintainer_email = "[email protected]", description = "Simple Python extension module in Pyrex", ext_modules = [Extension('elemlist',sources=['elemlist.pyx'])], cmdclass = {'build_ext': build_ext}, )
Then, we need a file elemlist.pyx with our module’s source code:
cdef class cons: cdef public object car, cdr def _ _init_ _(self, car, cdr): self.car = car self.cdr = cdr def _ _repr_ _(self): return 'cons(%r, %r)' % (self.car, self.cdr)
Pyrex is a language for writing Python extension modules. It was
developed by Greg Ewing and is freely downloadable and installable.
The Pyrex language is a large subset of Python, with the addition of a
few language constructs to allow easy generation of fast C code. In
this recipe, the only special Pyrex construct we use is the cdef
statement, which lets us express
C-level declarations.
This module is roughly equivalent to the Python-coded module:
class cons(object): _ _slots_ _ = ('car', 'cdr') def _ _init_ _(self, car, cdr): self.car = car self.cdr = cdr def _ _repr_ _(self): return 'cons(%r, %r)' % (self.car, self.cdr)
As you see, Pyrex code is very similar to Python code. Pyrex
code gets compiled into C, which in turn gets compiled to machine
code, while Python code is compiled into bytecode instead. For such a
simple type as cons
, the Pyrex
version is not much faster than the Python version, but a pure C
version, such as the one that I demonstrated previously in Recipe 17.1, despite having
25 times as much code, wouldn’t be any faster either.
Building a compiled extension module is just as simple when it’s
coded in Pyrex as when it’s coded directly in C, thanks to the
distutils
package, which does all
of the hard work. (You need to have Pyrex installed.) When you build a
Pyrex-coded module, you may get warnings from your C compiler about
symbols with names starting with _
_pyx
or _ _pyx
that are
defined but not used, or declared but not defined. Do not let these
warning messages worry you: your C compiler is running with the
highest possible level of warnings, and the little anomalies it’s
warning about are just perfectly normal and innocuous artifacts in the
C sources that Pyrex automatically generates. Pyrex is not quite
finished yet (the Pyrex version at the time of writing is 0.9.3), so
no attention has yet been spent on purely cosmetic warts. (By the time
you read this, a mature version of Pyrex may have been released, with
all i’s dotted and all t’s crossed. Nevertheless, I would recommend
Pyrex even if the latest version still causes numerous
warnings.)
Abundant documentation on Pyrex, as well as examples, can be found in the directory (and particularly in subdirectories Doc and Demos) where you unpacked Pyrex’s .tar.gz file; essentially the same documentation can also be read online, starting from the Pyrex web site at http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/.
Credit: Ralf W. Grosse-Kunstleve, David Abrahams
You want to use a C++ library in Python. For example, you might have a fast rational-numbers library, coded in C++, that you wish to wrap for use from Python.
Boost, http://www.boost.org, is a large free package with more than 50 fast and solid C++ libraries. Among those libraries, we find both Boost.Rational, a rational number library, and Boost.Python, which makes it easy to turn any other C++ library into a Python extension. So, we simply use the latter to wrap the former:
#include <boost/python.hpp> #include <boost/rational.hpp> /* two specific conversion functions: rational to float and to str */ static double as_float(boost::rational<int> const& self) { return double(self.numerator( )) / self.denominator( ); } static boost::python::object as_str(boost::rational<int> const& self) { using boost::python::str; if (self.denominator( ) == 1) return str(self.numerator( )); return str(self.numerator( )) + "/" + str(self.denominator( )); } /* the 'rational' Python extension module, with just one class in it: */ BOOST_PYTHON_MODULE(rational) { boost::python::class_<boost::rational<int> >("int") .def(boost::python::init<int, optional<int> >( )) .def("numerator", &boost::rational<int>::numerator) .def("denominator", &boost::rational<int>::denominator) .def("_ _float_ _", as_float) .def("_ _str_ _", as_str) .def(-self) .def(self + self) .def(self - self) .def(self * self) .def(self / self) .def(self + int( )) .def(self - int( )) .def(self * int( )) .def(self / int( )) .def(int( ) + self) .def(int( ) - self) .def(int( ) * self) .def(int( ) / self) ; }
Once you have built and installed the rational
extension shown in this recipe’s Solution, you can use it to perform
simple, natural tasks, such as:
>>> import rational >>> x = rational.int(1, 3) >>> y = rational.int(-2, 4) >>> print "x =", xx = 1/3
>>> print "y =", yy = -1/2
>>> print "x+y =", x+yx+y = -1/6
>>> print "x*2 =", x * 2x*2 = 2/3
>>> print "3/y =", 3 / y3/y = -6
Compiling and linking Boost.Python extensions is supported by the Boost.Build tool; we do not cover that topic here. Extensive documentation is available online at the Boost site. Such tools as make and SCons are also popular for software compilation and linking tasks, including tasks that involve Boost.
The Solution’s code shows off a few of Boost.Python’s powerful features. Consider the snippet:
BOOST_PYTHON_MODULE(rational) { class_<boost::rational<int> >("int") ...
The BOOST_PYTHON_MODULE
macro
takes a module name as a parameter, and a module body immediately
afterwards within braces, and does all that’s needed to make a module
we can import
from Python.
The class_
template,
instantiated with the boost::rational
type as a parameter and
“called” with the string argument "int
“, does all we need to have as part of
our module a Python-usable class, named rational.int
,
each of whose instances wraps an instance of the boost::rational
class. The type boost::rational
is itself a template, and we
instantiate that template with int
as a parameter, to use int
as the
type of each rational number’s numerator and denominator.
If we stopped here, wrapping a C++ class in the class_
template, and exposing the wrapper
without adding any more to it, we’d have a rather empty type available
on the Python side. It would have no constructor (save for the default
argument-less one), no methods, and no attributes. To remedy this, the
Solution code continues with several .def(...)
calls, which are
chained: each call enriches the object, and also
returns it, so you can just string such calls one after the other. The
methods we add with all those def
calls include a constructor (which uses the init
template), then a couple of ordinary
methods that delegate to the methods of the same names in the wrapped
class (accessors to the numerator and denominator parts of a rational
number), and then a couple of type-conversion special methods for
which we’ve previously coded corresponding functions (just before the
BOOST_PYTHON_MODULE
macro). Note,
in particular, that the implementation of the as_str
function is so concise because it makes full use of Boost.Python’s
object interface—it’s almost like writing Python in C++.
The baker’s dozen of .def(...)
calls that begins with:
.def(-self)
and proceeds all the way to:
.def(int( ) / self)
exposes all the arithmetic special methods for our new
rational.int
class—unary minus (_ _neg_ _
), and the four operations, each in
three versions—between two instances of our class, and between such
instances and int
s on either side
(_ _add_ _
, _ _radd_ _
, etc.). The magic is performed
using expression templates, a technique
originally developed for optimizing high-performance matrix algebra
expressions. Boost.Python’s use of expression templates has a
different purpose, but it certainly comes in handy anyway!
A comprehensive rational number extension would require more
functionality—comparison operators, _ _repr_
_
, _ _hash_ _
, support
for pickling, and so on. A more complete implementation, one that is
actively used in applications, can be found at http://cvs.sourceforge.net/viewcvs.py/cctbx/boost_adaptbx/,
in the file rational_ext.cpp.
Boost’s site is http://www.boost.org; the rational number library Boost.Rational, is at http://www.boost.org/libs/rational; Boost.Python is at http://www.boost.org/libs/python.
Credit: Stefano Spinucci
You want to avoid writing a Python extension in C, by directly calling from Python functions that already exist in a Windows DLL.
The third-party ctypes
extension makes this task pretty easy:
from ctypes import windll, c_int, c_string, byref # load 'Ehllapi.dll' (from current dir), and function 'hllapi' from the DLL Ehllap32 = windll.ehllapi hllapi = Ehllap32.hllapi # prepare the arguments with types and initial values h_func = c_int(1) h_text = c_string('A') h_len = c_int(1) h_ret = c_int(999) # call the function hllapi(byref(h_func), h_text, byref(h_len), byref(h_ret)) # print the resulting values of all arguments after the call print h_func.value, h_text.value, h_len.value, h_ret.value
I needed the code in this recipe specifically to call a C function whose prototype is:
void FAR PASCAL hllapi(int FAR *, char FAR *, int FAR *, int FAR *);
from a DLL named Ehllapi.DLL (an implementation of the IBM
3270 HLLAPI for an Italian 3270 terminal emulator, as it happens).
Thomas Heller’s ctypes
extension,
found at http://sourceforge.net/projects/ctypes, made
the task very easy. In particular, ctypes
makes mincemeat of problems related
to representing function arguments that must belong to a certain C
type and possibly get passed “by reference” (i.e., via a
pointer).
In the past, I used another extension, known as calldll
, which was (and still is) available
from http://www.nightmare.com/software.html. While
once very useful, calldll
cannot
rely on some of the modern techniques that ctypes
uses internally, because these
possibilities were introduced only in relatively recent versions of
Python. calldll
, using a single
membuf
Python type to represent all
possible C types, tends to be much more cumbersome than ctypes
when they are both used to perform
the same tasks.
Judge for yourself: here is a working calldll
version of the same script that I
just showed how to code with ctypes
:
import calldll, struct # some helpful auxiliary functions def myPrintLong(vVar): ''' print a long contained in a membuf ''' print calldll.read_long(vVar.address( )) def myPrintString(vVar): ''' print a string contained in a membuf ''' a = calldll.read_string(vVar.address( )) print a, len(a) def mySetLong(vMemBuf, vValueToSet): ''' set to an unsigned long the value of a membuf with len == 4 ''' vMemBuf.write(struct.pack('L', vValueToSet)) def mySetString(vMemBuf, vValueToSet): ''' set to a string (with terminator) the value of a membuf ''' pack_format = "%ds" % 1+len(vValueToSet) # +1 for the string_packed = struct.pack(pack_format, vValueToSet) # pack( ) adds the vMemBuf.write(string_packed) # load 'Ehllapi.dll' (from current dir), and function 'hllapi' from the DLL dll_handle = calldll.load_library ('.\Ehllapi') function_address = calldll.get_proc_address (dll_handle, 'HLLAPI') # allocate and init three membufs with the size to hold an unsigned long Lsize = struct.calcsize('L') vFunction = calldll.membuf(Lsize) mySetLong(vFunction, 1) vTextLen = calldll.membuf(Lsize) vResult = calldll.membuf(Lsize) mySetLong(vResult, 1) # allocate a membuf as large as the DLL requires; in this case, space # for 24 x 80 characters + 1 for a terminator vText = calldll.membuf(1921) # init the text and text-length variables based on string of interest string_value_to_write = 'A' mySetString(vText, string_value_to_write) mySetLong(vTextLen, len(string_value_to_write)) # call the function, print the results, and clean up calldll.call_foreign_function(function_address, 'llll', 'l', (vFunction.address( ), vText.address( ), vTextLen.address( ), vResult.address( ))) myPrintLong(vResult) myPrintString(vText) calldll.free_library(dll_handle)
To be honest, I can’t quite be sure whether all of these
gyrations are truly indispensable to making this calldll
-based version work. Whenever I try
to simplify this version a bit, something or other always breaks
noisily, so I’ve stopped messing with it. One reason the ctypes
-based version is cleaner and simpler
is that ctypes
has never given me
trouble, so I’ve been encouraged to continue working on that version
to improve it.
ctypes
is at http://sourceforge.net/projects/ctypes;
calldll
is at http://www.nightmare.com/software.html.
Credit: Joe VanAndel, Mark Hammond
You want to use SWIG-generated modules in a multithreaded environment; therefore, the C code in those modules must release the Python global interpreter lock (see the Introduction to Chapter 9 for more information about the global interpreter lock).
Use a typemap
for SWIG,
written by Mark Hammond, that was posted on comp.lang.python
. It maps Win32 API
functions that return BOOL
to
Python functions that return None
and raise exceptions to diagnose errors. The wrapped function must set
the standard Windows global LastError
if it returns FALSE
(indicating that it has detected an
error). The wrapping function also automatically releases the Python
global interpreter lock (GIL) for the duration of the wrapped
function’s execution, to allow free multithreading.
%typedef BOOL BOOLAPI %typemap(python,except) BOOLAPI {Py_BEGIN_ALLOW_THREADS $function Py_END_ALLOW_THREADS if (!$source) { $cleanup return PyWin_SetAPIError("$name"); } }
To use multiple threads effectively, you must release the Python
GIL from your C-coded extension whenever it’s safe to do so. The
simplest way to do this with SWIG is to use an except
directive, as shown in the recipe’s
typemap
. Within the typemap
, you can then use the normal Python
C API’s macros Py_BEGIN_ALLOW_THREADS
and Py_END_ALLOW_THREADS
(around the call to the
wrapped function, indicated by the special SWIG directive $function
) to release the GIL and acquire it
again.
Another interesting effect of this simple typemap
is that it turns the C-oriented
error-return convention (returning FALSE
and setting a global error indicator
code) into a highly Pythonic convention (raising an exception).
SWIG and its typemap
s are
documented at http://www.swig.org; Windows API documentation
on LastError
is available from the
Microsoft MSDN site at http://msdn.microsoft.com; Chapter 9 for general information
on threads and particularly its Introduction for information on the
GIL.
Credit: Luther Blissett
You have an existing C function that takes as an
argument a C array of C-level values (e.g., double
s), and you want to wrap it into a
Python-callable C extension that takes as an argument a Python
sequence or iterator.
The easiest way to accept an arbitrary Python sequence (or any
other iterable object) in the Python C API is with the PySequence_Fast
function. It builds and
returns a tuple when needed but returns only its argument (with the
reference count incremented) when the argument is already a list or
tuple:
#include <Python.h>
/* a preexisting C-level function you want to expose, e.g: */
static double total(double* data, int len)
{
double total = 0.0;
int i;
for(i=0; i<len; ++i)
total += data[i];
return total;
}
/* here is how you expose it to Python code: */
static PyObject *totalDoubles(PyObject *self, PyObject *args)
{
PyObject* seq;
double *dbar;
double result;
int seqlen;
int i;
/* get one argument as a sequence */
if(!PyArg_ParseTuple(args, "O", &seq))
return 0;seq = PySequence_Fast(seq, "argument must be iterable");
if(!seq)
return 0;
/* prepare data as an array of doubles */
seqlen = PySequence_Fast_GET_SIZE(seq);
dbar = malloc(seqlen*sizeof(double));
if(!dbar) {
Py_DECREF(seq);
return PyErr_NoMemory( );
}
for(i=0; i < seqlen; i++) {
PyObject *fitem;
PyObject *item = PySequence_Fast_GET_ITEM(seq, i);
if(!item) {
Py_DECREF(seq);
free(dbar);
return 0;
}
fitem = PyNumber_Float(item);
if(!fitem) {
Py_DECREF(seq);
free(dbar);
PyErr_SetString(PyExc_TypeError, "all items must be numbers");
return 0;
}
dbar[i] = PyFloat_AS_DOUBLE(fitem);
Py_DECREF(fitem);
}
/* clean up, compute, and return result */
Py_DECREF(seq);
result = total(dbar, seqlen);
free(dbar);
return Py_BuildValue("d", result);
}
static PyMethodDef totalMethods[ ] = {
{"total", totalDoubles, METH_VARARGS, "Sum a sequence of numbers."},
{0} /* sentinel */
};
void
inittotal(void)
{
(void) Py_InitModule("total", totalMethods);
}
The two best ways for your C-coded, Python-callable extension
functions to accept generic Python sequences as arguments are PySequence_Fast
and PyObject_GetIter
. The latter, which I cover
in the next recipe, can often save some memory, but it is appropriate
only when it’s OK for the rest of your C code to get the items one at
a time, without knowing beforehand how many items there will be in
total. You often have preexisting C functions from an existing library
that you want to expose to Python code, and such functions may require
C arrays as their input arguments. Thus, this recipe shows how to
build a C array (in this case, an array of double
) from a generic Python sequence (or
other iterable) argument, so that you can pass the array (and the
integer that gives the array’s length) to your existing C function
(represented here, purely as an example, by the total
function at the start of the recipe). (In the real world, you would
use Python’s built-in function sum
for this specific functionality, rather than exposing any existing C
function (but this is meant to be just an
example!)
PySequence_Fast
takes two
arguments: a Python iterable object to be presented as a sequence, and
a string to use as the error message in case the Python object cannot
be presented as a sequence, in which case PySequence_Fast
returns 0
(the C null pointer, NULL
, an error indicator). If the Python
object is already a list or tuple, PySequence_Fast
returns the same object with
the reference count increased by one. If the Python object is any
other kind of sequence (or any iterator, or other iterable), PySequence_Fast
builds and returns a new
tuple with all items already in place. In any case, PySequence_Fast
returns an object on which
you can call PySequence_Fast_GET_SIZE
to obtain the
sequence length (as we do in the recipe, in order to malloc
the appropriate amount of storage for
the C array) and PySequence_Fast_GET_ITEM
to get an item
given a valid index (an int
between
0, included, and the sequence length, excluded).
The recipe requires quite a bit of care (as is typical of all
C-coded Python extensions, and more generally of any C code) to deal
properly with memory issues and error conditions. For C-coded Python
extensions, in particular, it’s imperative that you know which Python
C API functions return new references (which you
must Py_DECREF
when you are done
with them) and which ones return borrowed
references (which you must not Py_DECREF
when you’re done with them; on the
contrary, you must Py_INCREF
such a
reference if you want to keep a copy for a longer time). In this
specific case, you have to know the following (by reading the Python
documentation):
PyArg_ParseTuple
produces
borrowed references.
PySequence_Fast
returns a
new reference.
PySequence_Fast_GET_ITEM
returns a borrowed reference.
PyNumber_Float
returns a
new reference.
There is method to this madness, even though, as you start your career as a coder of C API Python extensions, you’ll no doubt have to double-check each case carefully. Python’s C API strives to return borrowed references (for the sake of the modest performance increase that they afford, by avoiding needless incrementing and decrementing of reference counts), when it knows it can always do so safely (i.e., it knows that the reference it is returning necessarily refers to an already existing object). However, Python’s C API has to return a new reference when it’s possible (or certain) that a new object may have to be created.
For example, in the preceding list, PyNumber_Float
and PySequence_Fast
may be able to return the
same object they were given as an argument, but it’s also quite
possible that they may have to create a new object for this purpose,
to ensure that the returned object has the correct type. Therefore,
these two functions are specified as always returning new references.
PyArg_ParseTuple
and PySequence_Fast_GET_ITEM
, on the other hand,
always return references to objects that already exist elsewhere (as
items in the arguments’ tuple, or as items in the fast-sequence
container, respectively). Therefore, these two functions can afford to
return borrowed references and are thus specified as doing so.
One last note: in this recipe, as soon as we obtain an item from
the fast-sequence container, we immediately try to transform it into a
Python float
object, and thus we
have to deal with the possibility that the transformation will fail
(e.g., if we’re passed a sequence containing a string, a complex
number, etc.). It is most often futile to first attempt a check (with
PyNumber_Check
) because the check
might succeed, and the later transformation attempt might fail anyway
(e.g., with a complex-number item). Therefore, it’s better to attempt
the transformation and deal with the resulting error, if any. This
approach is yet another case of the common situation in which it’s
easier to get forgiveness than permission!
As usual, the best way to build this extension (assuming e.g.,
that you’ve saved the extension’s source code as a file named
total.c) is with the distutils
package. Place a file named
setup.py in the same directory as
the C source:
from distutils.core import setup, Extension setup(name="total", maintainer="Luther Blissett", maintainer_email= "[email protected]", ext_modules=[Extension('total', sources=['total.c'])] )
then build and install by running:
$ python setup.py install
An appealing aspect of this approach is that it works on any platform, assuming that you have access to the same C compiler used to build your version of Python, and permission to write on the site-packages directory where the resulting dynamically loaded library gets installed.
The Extending and Embedding manual is
available as part of the standard Python documentation set at
http://www.python.org/doc/current/ext/ext.html;
documentation on the Python C API is at http://www.python.org/doc/current/api/api.html;
the section “Distributing Python Modules” in the standard Python
documentation set is still incomplete, but it’s a good source of
information on the distutils
package; Python in a Nutshell covers the
essentials of extending and embedding, of the Python C API, and of the
distutils
package.
Credit: Luther Blissett
You want to write a Python-callable C extension that takes as an argument a Python sequence (or other iterable) and accesses it sequentially, one item at a time, requiring no extra storage.
If you can afford to access the sequence item-by-item, without
knowing in advance the number of items it has, you can often save
memory by using PyObject_GetIter
instead of PySequence_Fast
:
#include <Python.h>
static PyObject *totalIter(PyObject *self, PyObject *args)
{
PyObject* seq;
PyObject* item;
double result;
/* get one argument as an iterator */
if(!PyArg_ParseTuple(args, "O", &seq))
return 0;seq = PyObject_GetIter(seq);
if(!seq)
return 0;
/* process data sequentially */
result = 0.0;
while((item=PyIter_Next(seq))) {
PyObject *fitem;
fitem = PyNumber_Float(item);
if(!fitem) {
Py_DECREF(seq);
Py_DECREF(item);
PyErr_SetString(PyExc_TypeError, "all items must be numbers");
return 0;
}
result += PyFloat_AS_DOUBLE(fitem);
Py_DECREF(fitem);
Py_DECREF(item);
}
/* clean up and return result */
Py_DECREF(seq);
return Py_BuildValue("d", result);
}
static PyMethodDef totitMethods[ ] = {
{"totit", totalIter, METH_VARARGS, "Sum a sequence of numbers."},
{0} /* sentinel */
};
void
inittotit(void)
{
(void) Py_InitModule("totit", totitMethods);
}
PyObject_GetIter
is
appropriate only when it’s OK for the rest of your C code to get the
items one at a time, without knowing in advance the number of items in
total. When this condition is met, PyObject_GetIter
gives you roughly the same
performance as PySequence_Fast
(if
the input argument is a list or tuple), but it can save memory
allocation, and therefore can run faster, if the input argument is an
iterator or another kind of sequence or iterable. In this recipe’s
function, since we are just summing the items, it is indeed perfectly
OK to get them one at a time, and we don’t need to know in advance the
total number; therefore, using PyObject_GetIter
is preferable. (In the real
world, you would use Python’s built-in function sum
for this specific functionality, rather
than coding a dedicated C function, but then, this
is meant to be just an example!)
PyObject_GetIter
takes one
argument: a Python object from which an iterator is desired (much like
Python’s iter
built-in function).
It either returns 0
, indicating an
error, or an iterator object, on which you can repeatedly call
PyIter_Next
to get the next item
(or 0
, NULL
, which does not indicate an error, but
rather indicates the end of the iteration). Both PyObject_GetIter
and PyIter_Next
return new references, so we
must use Py_DECREF
when we’re done
with the respective objects.
As usual, the best way to build this extension (assuming that
you’ve saved it as a file named totit.c) is with the distutils
package. Place in the same
directory as the C source a file named setup.py such as:
from distutils.core import setup, Extension setup(name="totit", maintainer="Luther Blissett", maintainer_email= "[email protected]", ext_modules=[Extension('totit', sources=['totit.c'])] )
then build and install by running:
<m>$ python setup.py install</m>
Part of the appeal of this approach is that it works on any platform, assuming that you have access to the same C compiler used to build your version of Python, and permission to write on the site-packages directory where the resulting dynamically loaded library gets installed.
Since Python extensions are often coded in C to maximize performance, it’s interesting to measure performance compared to pure Python code dealing with the same task. A typical measurement setup might be a script such as the following timon.py:
import timeit, operator from total import total from totit import totit def timo(fn, sq, init): T = timeit.Timer('timon.%s(%s)'%(fn,sq), 'import timon '+init) print ' %5.5s: %5.2f' % (fn, T.timeit(40000)) def totpy(x): result = 0.0 for item in x: result += item return result def totre(x): return reduce(operator.add, x, 0.0) def totsu(x): return sum(x, 0.0) if _ _name_ _ == '_ _main_ _': print 'on lists:' for f in 'totre totpy total totit totsu'.split( ): timo(f, 'seq', 'seq=range(2000)') print 'on iters:' for f in 'totre totpy total totit totsu'.split( ): timo(f, 'g( )', 'def g( ): for x in range(2000): yield x')
This script uses the timeit
module of the Python Standard Library to measure accurately 40,000
calls to each function on 2,000-item lists and 2,000-item generators.
The timeit.Timer
constructor takes
two string arguments: first the statement we’re timing, then the setup
statements that run before timing begins. Here, the statement we’re
timing calls functions in this module; therefore, the setup statements
must import this module—which is why we add the import timon
at the beginning of the setup
statement string. I have also taken care to make all these functions
strictly comparable, by having them all sum float
s (not just int
s). This purpose is the reason that I
provide the explicit 0.0
initial
arguments to built-in functions reduce
and sum
.
On my machine, running with the command-line switch -O
so that Python can optimize operations a
little bit, the timing results on Python 2.3 are:
<m>$ python -O timon.py</m>on lists:
totre: 136.04
totpy: 118.18
total: 56.61
totit: 59.66
totsu: 74.11
on iters:
totre: 220.86
totpy: 198.98
total: 199.72
totit: 201.70
totsu: 157.44
As you can see, the most important optimization is to avoid the
“attractive nuisance” of the reduce
built-in function: even a pure Python loop is faster! When we’re
dealing with lists, the special-purpose C-coded extensions presented
in the last two recipes are fastest; but when we’re dealing with
generators, the fastest solution is provided by the built-in function
sum
. In practice, one would always
use sum
for this functionality,
rather than bothering to code or expose special-purpose C
functions.
The Extending and Embedding manual is
available as part of the standard Python documentation set at
http://www.python.org/doc/current/ext/ext.html;
documentation on the Python C API is at http://www.python.org/doc/current/api/api.html;
the section “Distributing Python Modules” in the standard Python
documentation set is still incomplete but is a good source of
information on the distutils
package: Chapter 19 of this
book covers iterators and generators in pure Python terms;
Python in a Nutshell covers the essentials of
extending and embedding, of the Python C API, of the distutils
package, and of iterators;
Python’s Library Reference covers the timeit
module.
Credit: Alex Martelli
Your C-coded, Python-callable function in an extension module
needs to return nothing in particular (i.e., a Python None
), but it must, of course, do so without
messing up reference counts.
Suppose we need an empty C-coded function, equivalent to Python:
def empty1(*args): pass
or, identically:
def empty2(*args): return None
Despite the simplicity of the task, there are right and wrong ways to perform it. The canonical solution is:
static PyObject* empty3(PyObject* self, PyObject* args) { Py_INCREF(Py_None); return Py_None; }
and the simplest, but still correct way, is:
static PyObject*
empty4(PyObject* self, PyObject* args)
{return Py_BuildValue("");
}
A function written in C for Python often needs to return nothing
in particular. In Python terms, it must return None
. Don’t just code return Py_None
; from C: that messes up
reference counts! None
—the Python
object we must explicitly return from a Python-callable, C-coded
function—is a normal Python object, subject to all normal reference
count rules. One of these rules is that each function must Py_INCREF
the Python object it
returns.
A bare return Py_None;
is a
nasty lurking bug—a frequent beginner’s error that messes up reference
counts:
static PyObject* empty5(PyObject* self, PyObject* args) { return Py_None; /* ***WRONG*** */ }
Either explicitly Py_INCREF
the None
object you’re returning,
or (a simpler approach, but one that costs a few machine cycles)
delegate the work to the handy function Py_BuildValue
, which can be used to handle
just about all cases of returning values from C to Python, offering
potential uniformity advantages. To have Py_BuildValue
build a properly incref’d
None
on your behalf, call it with
just one argument, an empty format string.
In Python 2.4, the C API has gained a new macro just for this
purpose. If you’re coding a C extension that supports only Python 2.4,
you can write Py_RETURN_NONE;
instead of the return
statement,
and the macro takes care of everything for you.
The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; documentation on the Python C API is at http://www.python.org/doc/current/api/api.html.
Credit: Joe VanAndel, Michael Aivazis
A dynamically loaded C/C++ Python extension is giving you trouble on a Unix or Unix-like platform, and you want to use the interactive debugger gdb to determine what’s wrong.
One way to determine the cause of core dumps or other serious
trouble with a C Python extension is to compile the extension source
with -g
and then follow these steps. (You may also
want to recompile any other extensions you use, such as Numeric, with
-g
, if you hadn’t built them that way in the first
place.)
<m>% gdb /usr/bin/python2.1</m> (gdb) br _PyImport_LoadDynamicModule (gdb) run # start python (gdb) cont # repeat until your extension is loaded (gdb) # you may need an import statement at python's >>> prompt (gdb) finish # finish loading your extension module (gdb) br wrap_myfunction # break at the entry point in your code (gdb) disable 1 # don't break for any more modules being loaded (gdb) cont # back to Python, run things normally from here
If a dynamically loaded C/C++ extension is causing Python to core dump, or causing some other kind of serious trouble, this recipe can help you determine the root cause, by demonstrating a technique for debugging your extension using gdb (if you use Unix or some Unix-like platform, and gdb is your debugger of choice). The overall concept generalizes to other debuggers with abilities similar to gdb’s.
The main point of this recipe is that you cannot set a break on
your function at the start, because your function lives in a dynamic
library (shared object) that isn’t initially loaded. However, you can
break in the PyImport_LoadDynamicModule
function, and
eventually (when your module is finally being loaded) get control at
the debugger prompt right after your module is in memory. You are then
able, at last, to set the breakpoints you need.
This technique works. However, if you do this kind of thing often, the process of stepping through all the modules, as Python loads them at startup, can easily become tedious. A handier alternative, although more invasive, requires you to modify your Python sources and rebuild Python from them.
The key idea of this handier alternative is to add a do-nothing function somewhere in the body of code that Python loads immediately. Specifically, you can edit the Modules/main.c file, adding one new function:
void Py_DebugTrap(void) { }
In the extension you’re debugging, you can now add a call to
Py_DebugTrap( )
right where you want to break into
the code. The Py_DebugTrap( )
symbol is immediately
available when you start gdb, because the symbol
lives in main.c. So you can
immediately set a breakpoint there, as soon as you are at the
gdb prompt, then continue. This approach even
works in parallel under MPI (message passing interface).
The gdb online documentation (just type
help
at the
gdb interactive prompt), manual pages, and online
manual (http://www.gnu.org/manual/gdb-4.17/gdb.html).
Credit: Will Ware
You’re developing C extensions, and you experience memory problems. You suspect mismanagement of reference counts and want to check whether your C extension code is correctly managing reference counts.
To chase these problems in an optimal way, you need to alter
Python’s sources and rebuild Python. Specifically, add the following
function in Objects/object.c,
immediately before the _Py_PrintReferences
function:
void _Py_CountReferences(FILE *fp) { int nr, no; PyObject *op; for (nr = no = 0, op = refchain._ob_next; op != &refchain; op = op->_ob_next, nr += op->ob_refcnt, no += 1) { } fprintf(fp, "%d refs (%d), %d objs ", nr, _Py_RefTotal, no); }
I place the following macros in my C extensions:
#if defined(Py_DEBUG) || defined(DEBUG) extern void _Py_CountReferences(FILE*); #define CURIOUS(x) { fprintf(stderr, _ _FILE_ _ ":%d ", _ _LINE_ _); x; } #else #define CURIOUS(x) #endif #define MARKER( ) CURIOUS(fprintf(stderr, " ")) #define DESCRIBE(x) CURIOUS(fprintf(stderr, " " #x "=%d ", x)) #define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, " " #x "=%08x ", x)) #define COUNTREFS( ) CURIOUS(_Py_CountReferences(stderr))
To debug, I rebuild Python using make
OPT="-DPy_DEBUG", which causes the code under Py_TRACE_REFS
to be built. My own makefile
for my extensions uses the same trick by including these lines:
debug: make clean; make OPT="-g -DPy_DEBUG" all CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python2.3
When I’m developing C extensions and running into memory
problems, I find that the typical cause is mismanagement of reference
counts, particularly misuse of Py_INCREF
and Py_DECREF
, as well as forgetfulness of the
reference-count effects of functions like Py_BuildValue
, PyArg_ParseTuple
, and PyTuple/List_SetItem/GetItem
. The Python
sources offer help with this problem (search for Py_TRACE_REFS
), and function sys.getrefcounts
in the Python Standard
Library is also helpful. Nevertheless, it’s useful to add this
recipe’s function in Objects/object.c just before _Py_PrintReferences
.
Unlike _Py_PrintReferences
,
this recipe’s _Py_CountReferences
function prints
only the totals of all the refcounts and number of objects in the
system, so it can be sensibly called, even in loops that repeat
millions of times, while _Py_PrintReferences
would print out way too
much stuff to be useful. The information printed by
_Py_CountReferences
can help you identify errantly
wandering Py_INCREF
s and Py_DECREF
s.
_Py_CountReferences
plays it safe by performing its
own counts of objects references, which it prints side by side with
the “official” count of references that Python itself maintains (when
compiled for debugging) as global variable _Py_RefTotal
. Should any discrepancy arise,
you know something deeply wrong is going
on.
When I suspect that one of my C-coded functions is responsible
for memory problems, I liberally sprinkle the suspect function with
calls to the COUNTREFS
macro. Doing so allows me to
keep track of exactly how many references are being created or
destroyed as I go through my function. This information is
particularly useful in tight loops, in which dumb mistakes can cause
reference counts to grow ridiculously fast. Also, reference counts
that shrink too fast (because of overzealous use of Py_DECREF
) can cause core dumps because the
memory for objects that should still exist has been reallocated for
new objects.
The only documentation in this case is Python’s own source code. Use the source, Luke!