Credit: Jeremy Hylton, Google, Inc.
The recipes in this chapter describe simple techniques for using Python in distributed systems. Programming distributed systems is a difficult challenge, and recipes alone won’t even come close to completely solving it. The recipes help you get programs on different computers talking to each other, so that you can start writing applications.
Remote Procedure Call (RPC) is an attractive approach to structuring a distributed system. The details of network communication are exposed through an interface that looks like normal procedure calls. When you call a function on a remote server, the RPC system is responsible for all the details of communication. It encodes the arguments so they can be passed over the network to the server, which might use different internal representations for the data. It invokes the right function on the remote machine and waits for a response.
The recipes in this chapter use three different systems that provide RPC interfaces—Common Object Request Broker Architecture (CORBA), Twisted’s Perspective Broker (PB), and, in most recipes, XML-RPC. These systems are attractive because they make it easy to connect programs that can be running on different computers and might even be written in different languages. CORBA is a rather “heavyweight” protocol, very rich in functionality, with specialized and dedicated marshaling and transport layers (and much more besides). XML-RPC is a lightweight, simple-to-use protocol, which uses XML to marshal the call and its associated data, and HTTP for transport. Being simple and lightweight, XML-RPC is less functionally rich than CORBA. Both CORBA and XML-RPC are well-established standards, with implementations available for a wide variety of languages. In particular, XML-RPC is so simple and widespread that XML-RPC recipes take up half this chapter, a good, if rough, indication of how often Pythonistas are using it in preference to other distributed programming approaches.
PB is also “lightweight”, while offering richer functionality than
XML-RPC. However, PB is not a separate standard but a part of the
Twisted framework, so PB implementations are currently limited to what
Twisted itself provides and are mostly in Python. Perspective Broker is
unusual among RPC systems because it openly exposes at application level
the fact that network transactions are
asynchronous, not
synchronous like procedure calls within a single
process. Therefore, in PB, the launching of a call to the remote
procedure does not necessarily imply an immediate wait for the
procedure’s results; rather, the “result"s arrive “later” through a
callback mechanism (specifically, Twisted’s deferred
objects). This asynchronous approach,
which is the conceptual core of the Twisted framework, offers
substantial advantages in terms of performance and scalability of the
“result"ing network applications, but it may take some getting used to.
Simon Foster’s approach, shown in recipe, Recipe 15.7, is a simple way
to get started exploring Perspective Broker.
XML-RPC is well supported by the Python Standard Library, with
the xmlrpclib
module for writing
XML-RPC clients and the SimpleXMLRPCServer
module for writing XML-RPC
servers. For Twisted, CORBA, and other RPC standards yet (such as the
emerging SOAP—Simple Object Access Protocol—system), you need to install
third-party extensions before you can get started. The recipes in this
chapter include pointers to the software you need. Unfortunately, you
will not find pointers specifically to SOAP resources for Python in the
recipes: for such pointers, I suggest you check out http://pywebsvcs.sourceforge.net/.
The Python Standard Library also provides a set of modules
for doing the lower-level work of network programming—socket
, select
, asyncore
, and asynchat
. The library also includes modules
for marshaling data and sending it across sockets: struct
, pickle
, xdrlib
. Chapter 13, includes recipes in
which some of these modules are directly used, and Chapter 7, contains recipes dealing
with serialization and marshaling. These lower-level modules, in turn,
provide the plumbing for many other higher-level modules that directly
support a variety of standard network protocols. Jeff Bauer offers Recipe 15.9, using the
telnetlib module to send commands to remote machines via the Telnet
protocol. Telnet is not a very secure protocol, and thus, except for use
within well-protected LANs, has been largely supplanted by more secure
protocols such as SSH (Secure Shell). Peter Cogolo and Anna Martelli
Ravenscroft offer similar functionality to Bauer’s, in Recipe 15.10, which uses SSH
(via the third-party package paramiko
) rather than Telnet.
Six of the recipes, just about half of the chapter, focus on
XML-RPC. Rael Dornfest and Jeremy Hylton demonstrate how to write an
XML-RPC client program that retrieves data from O’Reilly’s Meerkat
service. Recipe 15.1 is
only three lines long (including the import
statement): indeed, this extreme
conciseness is the recipe’s main appeal.
Brian Quinlan and Jeff Bauer contribute two different recipes for
constructing XML-RPC servers. Quinlan, in Recipe 15.2, shows how to use
the SimpleXMLRPCServer
module from
the Python Standard Library to handle incoming requests. Bauer’s is
Recipe 15.3. Medusa,
like Twisted, is a framework for writing asynchronous network programs.
In both cases, the libraries do most of the work; other than a few lines
of initialization and registration, the server looks like normal Python
code.
Christop Dietze (with contributions from Brian Quinlan and Jeff Bauer), in Recipe 15.4, elaborates on the XML-RPC server theme by showing how to add the ability that enables remote clients to terminate the server cleanly. Rune Hansen, in Recipe 15.5, shows how to add several minor but useful niceties to your XML-RPC servers.
Peter Arwanitis, in Recipe 15.6, demonstrates how to implement an XML-RPC server with Twisted and, at the same time, give your server a GUI, thanks to the wxPython GUI framework.
A strong alternative to XML-based protocols is CORBA, an object-based RPC mechanism using its own protocol, IIOP (Internet Inter-Orb Protocol). CORBA is a mature technology compared to XML-RPC (or, even more, SOAP, which isn’t used in any of these recipes—apparently, Pythonistas aren’t doing all that much with SOAP yet). CORBA was introduced in 1991. The Python language binding was officially approved more recently, in February 2000, and several ORBs (Object Request Brokers—roughly, CORBA servers) support Python. Duncan Grisby, a researcher at AT&T Laboratories in Cambridge (U.K.), describes the basics of getting a CORBA client and server running in Recipe 15.8, which uses omniORB, a free ORB, and the Python binding he wrote for it.
CORBA has a reputation for complexity, but Grisby’s recipe makes
it look straightforward. More steps are involved in the CORBA client
than in the XML-RPC client example, but they are not difficult. To
connect an XML-RPC client to a server, you just need a URL. To connect a
CORBA client to a server, you need a URL—a special corbaloc
URL—and you also need to know the
server’s interface. Of course, you need to know the interface regardless
of protocol, but CORBA uses it explicitly. In general, CORBA offers more
features than other distributed programming frameworks—interfaces, type
checking, passing references to objects, and more. CORBA also supports
just about every Python data type as argument or result.
Regardless of the protocols or systems you choose, the recipes in
this chapter can help get you started. Inter-program communication is an
important part of building a distributed system, but it’s just one part.
Once you have a client and server working, you’ll find you have to deal
with other interesting, challenging problems—error detection,
concurrency, and security, to name a few. The recipes here won’t solve
those problems, but they will prevent you from getting caught up in
unimportant details of the communication protocols. Rob Riggs in Recipe 15.11 presents a
simple way to use HTTPS (as supported by the Python Standard Library
module httplib
) to authenticate SSL
clients; Simon Foster’s previously mentioned Perspective Broker recipe
provides a way to implement one specific but frequent strategy for error
detection and handling, namely periodically trying to reconnect to a
server after a timeout or explicitly discovered network error.
Credit: Rael Dornfest, Jeremy Hylton
You need to make a method call to an XML-RPC server.
The xmlrpclib
module makes
writing XML-RPC clients very easy. For example, we can use XML-RPC to
access O’Reilly’s Meerkat server and get the five most recent items
about Python:
from xmlrpclib import Server
server = Server("http://www.oreillynet.com/meerkat/xml-rpc/server.php")
print server.meerkat.getItems(
{'search': '[Pp]ython', 'num_items': 5, 'descriptions': 0}
)
XML-RPC is a simple, lightweight approach to distributed
processing. xmlrpclib
, which makes
it easy to write XML-RPC clients in Python, is part of the Python
Standard Library.
To use xmlrpclib
, you
first instantiate a proxy to the server, calling the ServerProxy
class (also known by the name
Server
) and passing in the URL to
which you want to connect. Then, on that proxy instance, you can
access and call whatever methods the remote XML-RPC server supplies.
In this case, you know that Meerkat supplies a getItems
method, so you call the method of
the same name on the server proxy instance. The proxy relays the call
to the server, waits for the server to respond, and finally returns
the call’s results.
This recipe uses O’Reilly’s Meerkat service, intended for syndication of contents such as news and product announcements. Specifically, the recipe queries Meerkat for the five most recent items mentioning either “Python” or “python”. If you try this recipe, be warned that response times from Meerkat are variable, depending on the quality of your Internet connection, the time of day, and the level of traffic on the Internet. If the script takes a long time to answer, it doesn’t mean you did something wrong—it just means you have to be patient!
Using xmlrpclib
by passing
raw dictionaries, as in this recipe’s code, is quite workable but
somewhat unPythonic. Here’s an easy alternative that looks
nicer:
from xmlrpclib import Server server = Server("http://www.oreillynet.com/meerkat/xml-rpc/server.php") class MeerkatQuery(object): def _ _init_ _(self, search, num_items=5, descriptions=0): self.search = search self.num_items = num_items self.descriptions = descriptions q = MeerkatQuery("[Pp]ython") print server.meerkat.getItems(q)
You can package the instance attributes and their default values
in several different ways, but the main point of this variant is that,
as the argument to the getItems
method, an instance object with the right attributes works just as
well as a dictionary object with the same information packaged as
dictionary items.
The xmlrpclib
module is part
of the Python Standard Library and is well documented in its chapter
of the Library Reference portion of Python’s
online documentation. Meerkat is at http://www.oreillynet.com/meerkat/.
Credit: Brian Quinlan
Module SimpleXMLRPCServer
,
which is part of the Python Standard Library, makes writing XML-RPC
servers reasonably easy. Here’s how you can write an XML-RPC
server:
# Server code sxr_server.py import SimpleXMLRPCServer class StringFunctions(object): def _ _init_ _(self): # Make all the functions in Python's standard string module # available as 'python_string.func_name' for each func_name import string self.python_string = string def _privateFunction(self): # This function cannot be called directly through XML-RPC because # it starts with an underscore character '_', i.e., it's "private" return "you'll never get this result on the client" def chop_in_half(self, astr): return astr[:len(astr)/2] def repeat(self, astr, times): return astr * times if _ _name_ _=='_ _main_ _': server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8000)) server.register_instance(StringFunctions( )) server.register_function(lambda astr: '_' + astr, '_string') server.serve_forever( )
And here is a client script that accesses the server you just wrote:
# Client code sxr_client.py import xmlrpclib server = xmlrpclib.Server('http://localhost:8000') print server.chop_in_half('I am a confident guy') # emits:I an a con
print server.repeat('Repetition is the key to learning! ', 5) # emits 5 lines, allRepetition is the key to learning!
print server._string('<= underscore') # emits_<= underscore
print server.python_string.join(['I', 'like it!'], " don't ") # emitsI don't like it!
print server._privateFunction( ) # this will throw an exception # terminates client script with traceback for xmlrpclib.Fault
This recipe demonstrates the creation of a simple XML-RPC server
using the SimpleXMLRPCServer
module
of the standard Python library. The module contains a class of the
same name that listens for HTTP requests on a specified port and
dispatches any XML-RPC calls to registered instances or registered
functions. This recipe demonstrates both usages. To create a server,
we instantiate the SimpleXMLRPCServer
class, supplying the
hostname and port for the server. Then, on that instance, we call
register_instance
as many times as
needed to make other instances available as services. In addition, or
as an alternative, we call register_function
to make functions
similarly available as services. Once we have registered all the
instances and/or all the functions we want to expose, we call the
serve_forever
method of the server
instance, and our XML-RPC server is active. Yes, it is really that
simple. The only output on the shell prompt window on which you run
the server is one line of log information each time a client accesses
the server; the only way to terminate the server is to send it an
interrupt, for example with a Ctrl-C keystroke.
Registering functions (as opposed to an instance) is necessary
when a function name begins with an underscore (_
) or contains characters not allowed in
Python identifiers (e.g., accented letters, punctuation marks, etc.).
Dotted names (e.g., python_string.join
) are correctly
resolved for registered instances.
The SimpleXMLRPCServer
module
is part of the Python Standard Library and is documented in its
chapter of the Library Reference portion of
Python’s online documentation.
Credit: Jeff Bauer
You need to establish a lightweight, highly scalable, distributed processing system and want to use the XML-RPC protocol.
Package medusa
lets
you implement lightweight, highly scalable, asynchronous
(event-driven) network servers. An XML-RPC handler is included in the
Medusa distribution. Here is how you can code an XML-RPC server with
Medusa:
# xmlrpc_server.py from socket import gethostname from medusa.xmlrpc_handler import xmlrpc_handler from medusa.http_server import http_server from medusa import asyncore class xmlrpc_server(xmlrpc_handler): # initialize and run the server def _ _init_ _(self, host=None, port=8182): if host is None: host = gethostname( ) hs = http_server(host, port) hs.install_handler(self) asyncore.loop( ) # an example of a method to be exposed via the XML-RPC protocol def add(self, op1, op2): return op1 + op2 # the infrastructure ("plumbing") to expose methods def call(self, method, params): print "calling method: %s, params: %s" % (method, params) if method == 'add': return self.add(*params) return "method not found: %s" % method if _ _name_ _ == '_ _main_ _': server = xmlrpc_server( )
And here is a client script that accesses the server you just wrote:
# xmlrpc_client.py from socket import gethostname from xmlrpclib import Transport, dumps class xmlrpc_connection(object): def _ _init_ _(self, host=None, port=8182): if host is None: host = gethostname( ) self.host = "%s:%s" % (host, port) self.transport = Transport( ) def remote(self, method, params=( )): """ Invoke the server with the given method name and parameters. The return value is always a tuple. """ return self.transport.request(self.host, '/RPC2', dumps(params, method)) if _ _name_ _ == '_ _main_ _': connection = xmlrpc_connection( ) answer, = connection.remote("add", (40, 2)) print "The answer is:", answer
This recipe demonstrates remote method calls between two machines (or two processes, even on the same machine) using the XML-RPC protocol and provides a complete example of working client/server code.
XML-RPC is one of the easiest ways to handle distributed processing tasks. There’s no messing around with the low-level socket details, nor is it necessary to write an interface definition. The protocol is platform and language neutral. The XML-RPC specification can be found at http://www.xml-rpc.com and is well worth studying. It’s nowhere as functionally rich as heavyweight stuff like CORBA, but, to compensate, it is much simpler!
To run this recipe’s Solution, you must download the Medusa
library from http://www.nightmare.com (the Python Standard
Library includes the asyncore
and
asynchat
modules, originally from
Medusa, but not the other parts of Medusa required for this recipe).
With Medusa, you implement an XML-RPC server by subclassing the
xmlrpc_handler
class and passing an
instance of your class to the install_handler
method of an instance of
http_server
. HTTP is the
transport-level protocol used by the XML-RPC standard, and http_server
handles all transport-level
issues on your behalf. You need to provide only the handler part, by
customizing xmlrpc_handler
through
subclassing and method overriding. Specifically, you must override the
call
method, which the Medusa
framework calls on your instance with the name of the XML-RPC method
being called, along with its parameters, as arguments. This is exactly
what we do in this recipe, in which we expose a single XML-RPC method
named add
which accepts two numeric parameters and
returns their sum as the method’s result.
This recipe’s XML-RPC client uses xmlrpclib
in a more sophisticated way than
Recipe 15.1, by
accessing the Transport
class
explicitly. In theory, this approach allows finer-grained control.
However, this recipe does not exert that kind of control, and it’s
rarely required in XML-RPC clients that you actually deploy,
anyway.
The xmlrpclib
module is part
of the Python Standard Library and is documented in a chapter of the
Library Reference portion of Python’s online
documentation. Medusa is at http://www.nightmare.com.
Credit: Christoph Dietze, Brian Quinlan, Jeff Bauer
You are coding an XML-RPC server, using the Python
Standard Library’s SimpleXMLRPCServer
module, and you want to
make it possible for a remote client to cause the XML-RPC server to
exit cleanly.
You have to use your own request-handling loop (instead of the
serve_forever
method of SimpleXMLRPCServer
) so that you can stop
looping when appropriate. For example:
import SimpleXMLRPCServer running = True def finis( ): global running running = False return 1 server = SimpleXMLRPCServer.SimpleXMLRPCServer(('127.0.0.1', 8000)) server.register_function(finis) while running: server.handle_request( )
SimpleXMLRPCServer
’s serve_forever
method, as its name implies, attempts to keep serving “forever”—that
is, it keeps serving until the whole server process is killed.
Sometimes, it’s useful to allow remote clients to request a clean
termination of a service by remotely calling a server-exposed
function, and this recipe demonstrates the simplest way to allow this
functionality.
The finis
function (which gets exposed to
remote clients via the register_function
call) sets the global
variable running
to False
(and then returns something that is
not None
because the XML-RPC
protocol cannot deal with the None
object). Using the while running
loop, instead of a serve_forever
call, then ensures that the server stops serving and terminates when
the variable running
becomes false.
If you prefer to subclass SimpleXMLRPCServer
, you can obtain a similar
effect by overriding the serve_forever
method: that is, instead of
placing the simple while
running: server.handle_request
loop inline,
you can code, for example (with the same function
finis
as in the recipe’s Solution):
class MyServer(SimpleXMLRPCServer.SimpleXMLRPCServer): def serve_forever(self): while running: self.handle_request( ) server = MyServer(('127.0.0.1', 8000)) server.register_function(finis) server.serve_forever( )
However, this alternative approach offers no special advantage
(unless you have a fetish for being object oriented for no particular
purpose), and, since this alternative approach is telling a little
white lie (by using the name serve_forever
for a method that does
not keep serving “forever”!), the simpler
approach in the recipe’s Solution can definitely be
recommended.
The SimpleXMLRPCServer
module
is part of the Python Standard Library and is documented in a chapter
of the Library Reference portion of Python’s
online documentation.
Credit: Rune Hansen
You are coding XML-RPC servers with the Python Standard
Library SimpleXMLRPCServer
class
and want to ensure you’re using the simple but useful idioms that can
ease your coding, or give your servers more flexibility at no
substantial cost to you.
Here are a few tweaks I generally use, to enhance my servers’
usability, when I’m developing servers based on SimpleXMLRPCServer
:
# give the base class a short, readable nickname from SimpleXMLRPCServer import SimpleXMLRPCServer as BaseServer class Server(BaseServer): def _ _init_ _(self, host, port): # accept separate hostname and portnumber and group them BaseServer._ _init_ _(self, (host, port)) def server_bind(self): # allow fast restart of the server after it's killed import socket self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) BaseServer.server_bind(self) allowedClientHosts = '127.0.0.1', '192.168.0.15', def verify_request(self, request, client_address): # forbid requests except from specific client hosts return client_address[0] in self.allowedClientHosts
The recipe begins with a statement of the form from
module
import name
as
nickname
, a Python idiom that is often
handy for importing something under a short and usable nickname. It’s
certainly miles better than having to repeatedly write SimpleXMLRPCServer.SimpleXMLRPCServer
after
a simple import
statement, or using
the ill-advised construct from
module
import *, which mixes up all the
namespaces and can often cause subtle and hard-to-find bugs.
The sole purpose of the _ _init_
_
statement of class Server
is to accept
host
and port
as separate parameters
and group them into the required tuple. I find myself often writing
such statements with the many Python functions and classes that
require this address tuple grouping (your
tastes, of course, may be different).
By default, a server socket belonging to a process that dies is
kept busy for quite a while. Particularly during development, it is
handy to kill such a process, edit the script, and restart
immediately. For such an immediate restart to work, you must ensure
the code of your server sets the SO_REUSEADDR
option on the relevant socket,
as the recipe’s code does in its overridden method server_bind
.
Last but not least, the recipe overrides verify_request
in order to apply a simple
check that refuses service except to requests coming from client hosts
on a predefined list. This approach doesn’t provide rock-solid
security, but nevertheless, it is potentially useful. Again, it’s
particularly useful during development, to help avoid those cases
where some other developer on the same LAN accidentally connects his
client to the server I’m just developing, and we both experience
puzzling problems until we figure out what’s happened!
The SimpleXMLRPCServer
module
is part of the Python Standard Library and is documented in a chapter
of the Library Reference portion of Python’s
online documentation.
Credit: Peter Arwanitis, Alex Martelli
You are writing an XML-RPC server and want to add a GUI to it, or you’re writing a GUI application that you want to be able to interact as an XML-RPC server too.
As long as you use Twisted for the network interaction, and
wxPython for the GUI, this task is reasonably easy, since these
packages can cooperate through the twisted.internet.wxreactor
module. You do
need to have specific incantations at the start of your program, as
follows:
# To use wxPython and Twisted together, do the following, in exact order: import wx from twisted.internet import wxreactor wxreactor.install( ) from twisted.internet import reactor # Then, have whatever other imports as may be necessary to your program from twisted.web import xmlrpc, server class MyFrame(wx.Frame): ''' Main window for this wx application. ''' def _ _init_ _(self, parent, ID, title, pos=wx.DefaultPosition, size=(200, 100), style=wx.DEFAULT_FRAME_STYLE): wx.Frame._ _init_ _(self, parent, ID, title, pos, size, style) wx.EVT_CLOSE(self, self.OnCloseWindow) def OnCloseWindow(self, event): self.Destroy( ) reactor.stop( ) class MyXMLRPCApp(wx.App, xmlrpc.XMLRPC): ''' We're a wx Application _AND_ an XML-RPC server too. ''' def OnInit(self): ''' wx-related startup code: builds the GUI. ''' self.frame = MyFrame(None, -1, 'Hello') self.frame.Show(True) self.SetTopWindow(self.frame) return True # methods exposed to XML-RPC clients: def xmlrpc_stop(self): """ Closes the wx application. """ self.frame.Close( ) return 'Shutdown initiated' def xmlrpc_title(self, x): """ Change the wx application's window's caption. """ self.frame.SetTitle(x) return 'Title set to %r' % x def xmlrpc_add(self, x, y): """ Provide some computational services to clients. """ return x + y if _ _name_ _ == '_ _main_ _': # pass False to emit stdout/stderr to shell, not an additional wx window app = MyXMLRPCApp(False) # Make the wx application twisted-aware reactor.registerWxApp(app) # Make a XML-RPC Server listening to port 7080 reactor.listenTCP(7080, server.Site(app)) # Start both reactor parts (wx MainLoop and XML-RPC server) reactor.run( )
It is often useful to give an XML-RPC server a GUI, for example, to display the current status to an operator or administrator. Conversely, it is often useful to give a GUI application the ability to accept remote requests from other programs, and making the application an XML-RPC server is an excellent, simple way to accomplish that purpose.
Either way, if you use Twisted for the networking part, you’re
off to a good start, because Twisted offers specialized reactor
implementations to ease cooperation
with several GUI toolkits. In particular, this recipe shows how a
Twisted-based XML-RPC server can sport a wxPython GUI thanks to the
twisted.internet.wxreactor
module.
To try this recipe, save the code from the “Solution” as a
Python script and start it from a shell. If you run some kind of
“personal firewall” that’s normally set to impede TCP/IP communication
between programs running on your machine, ensure it’s set to let such
communication happen on TCP port 7080
. Then, from any interactive Python
interpreter session on the same machine, do:
>>> import xmlrpclib >>> s = xmlrpclib.ServerProxy('http://localhost:7080') >>> s.add(23, 42)65
>>> s.title('Changed Title')Title set to 'Changed Title'
Observe that the title of the wx
application’s window has changed. Now, you can close the application,
either by whatever GUI means you normally use on your platform (it is
a totally cross-platform application, after all), or by calling
s.stop( )
from the same Python
interpreter interactive session that we just showed. You can also run
such a client on any other machine, as long as it has open TCP/IP
connectivity on port 7080
with the
machine running the server. (In particular, make sure you open port
7080
on any firewall that would
normally block that port, whether the firewall is on either of the
machines, or on any other network apparatus that may lie between
them.)
Both Twisted and wxPython, while already rich and solid
frameworks, are still growing and changing, so it may be important to
ensure you have the right releases installed properly on your machine.
This recipe should run on any platform that is equipped with Python
2.3 or better, wxPython 2.4.2.4 or better, and Twisted 1.3.0 or
better. Of course, we don’t have access to every platform in the
world, nor to all future releases of these tools, so we tested the
recipe only under Windows/XP, Mac OS X 10.3.6, and Linux, with Python
2.3 and 2.4, wxPython 2.4.2.4, and some
2.5.x.y
releases, and Twisted 1.3.0
specifically.
Since the recipe relies only on published, supported aspects of the various tools, one can hope that the recipe will also work elsewhere, and will work with future releases of the tools. However, if this recipe’s approach does not prove satisfactory for your purposes, you may want to try a different approach based on threads, shown at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286201.
Twisted’s home page is http://www.twistedmatrix.com; documentation on Twisted XML-RPC support is at http://www.twistedmatrix.com/documents/current/howto/xmlrpc; wxPython’s home page is http://www.wxpython.org.
Credit: Simon Foster
You want to implement Python clients and servers for some distributed processing task, without repetitious “boilerplate” code, and with excellent performance and scalability characteristics.
Use the Perspective Broker (PB) subsystem
of the Twisted framework. A PB server just
subclasses the PB’s Root
class and
adds remotely callable methods. Here is an example of a server script
which adds just one remotely callable method, named
Pong
:
from twisted.spread import pb from twisted.internet import reactor PORT = 8992 class Ponger(pb.Root): def remote_Pong(self, ball): print 'CATCH', ball, ball += 1 print 'THROW', ball return ball reactor.listenTCP(PORT, pb.BrokerFactory(Ponger( ))) reactor.run( )
We could write an equally trivial script for the client side of the interaction, but let’s instead have a rather feature-rich PB client, which deals with important issues often ignored in introductory examples of distributed programming, such as error handling:
from twisted.spread import pb from twisted.internet import reactor import sys PORT = 8992 DELAY = 1 DOG_DELAY = 2 RESTART_DELAY = 5 class Pinger(object): def _ _init_ _(self, host): self.ping = None self.host = host self.ball = 0 self._start( ) def _start(self): print 'Waiting for Server', self.host dfr = pb.getObjectAt(self.host, PORT, 30) dfr.addCallbacks(self._gotRemote, self._remoteFail) def _gotRemote(self, remote): remote.notifyOnDisconnect(self._remoteFail) self.remote = remote self._ping( ) def _remoteFail(self, _ _): if self.ping: print 'ping failed, canceling and restarting' self.ping.cancel( ) self.ping = None self.restart = reactor.callLater(RESTART_DELAY, self._start) def _watchdog(self): print 'ping timed out, canceling and restarting' self._start( ) def _ping(self): self.dog = reactor.callLater(DOG_DELAY, self._watchdog) self.ball += 1 print 'THROW', self.ball, dfr = self.remote.callRemote('Pong', self.ball) dfr.addCallbacks(self._pong, self._remoteFail) def _pong(self, ball): self.dog.cancel( ) print 'CATCH', ball self.ball = ball self.ping = reactor.callLater(DELAY, self._ping) if _ _name_ _ == '_ _main_ _': if len(sys.argv) != 2: print 'Usage: %s serverhost' % sys.argv[0] sys.exit(1) host = sys.argv[1] print 'Ping-pong client to host', host Pinger(host) reactor.run( )
Twisted is a framework for asynchronous
(also known as event-driven) programming of
network clients, servers, proxies, and so on. The asynchronous
programming model (which Twisted implements through the Reactor Design
Pattern embodied in the twisted.internet.reactor
module) provides
excellent performance and scalability characteristics for
Twisted-based programs.
Twisted also includes many subsystems that offer your programs
ready-to-go networking functionality. One of these subsystems,
Perspective Broker (PB), is implemented in the twisted.spread.pb
module. PB lets you code
distributed-programming clients and servers, with an ease of use
that’s most clearly displayed in the server program at the start of
this recipe’s Solution. In just a few lines of code, the server class
is able to expose remotely callable methods: all it takes is
subclassing the Root
class of the
pb
module and naming each remotely
callable method with a prefix of remote_
.
Most of the client code in this recipe is concerned with diagnosing and handling possible problems and errors with the connection to the server. Specifically, if the connection fails for any reason, including a timeout diagnosed by the watchdog timer that the client sets up each time it pings, the client attempts to reconnect to the server. If you kill the server, the client keeps trying to reconnect, periodically, until you restart the server.
Error-handling apart, the client is essentially as simple as the
server. In the method _start
, the client calls
function getObjectAt
of module
twisted.spread.pb
, which takes as
its arguments the server’s host, a port number, and a “time-out” delay
in seconds. As usual in Python networking, the host can be either a
network name, such as localhost
, or
a string representing an IP address, such as 127.0.0.1
.
If no problems arise, getObjectAt
returns an object that proxies
for the remote PB server. The proxy object, in turn, has a callRemote
method, which takes as its
arguments the method name as a string, followed by any arguments you
are passing to the remote method. callRemote
returns a Twisted deferred
object, the lynchpin of Twisted’s
style of asynchronous (event-driven) programming. Learning to use
deferred
s effectively is the
fundamental step in learning to program with Twisted.
A deferred
object represents
an event that may occur in the future (the
success-case) or may end in failure. Given a
deferred
, you can add callbacks to
it for both success and failure cases. (You can also
chain callbacks, a possibility that this recipe
does not exploit.) When the deferred
’s event occurs,
Twisted calls your “success-case” callback,
passing as its argument the “result” of the deferred
. Alternatively, if the deferred
ends in failure,
Twisted calls your
failure-case callback, passing as its argument
a failure object that wraps a Python exception object.
As you see in this recipe, despite deferred
s’ potentially rich and vast
functionality, their use is really quite simple in most cases. For
example, in the failure cases, the client in this recipe wants to
retry connecting: therefore, method _remoteFail
accepts the failure-object argument with an argument name of “two
underscores” (_ _
), a common Python convention that
indicates the argument will be ignored.
The Twisted web site, at http://www.twistedmatrix.com, has abundant
documentation about all of Twisted’s elements and subsystems,
including Perspective Broker and deferred
objects.
Credit: Duncan Grisby
You need to implement a CORBA server and client to distribute a processing task, such as the all-important network-centralized fortune-cookie distribution.
CORBA is a solid, rich, mature object-oriented RPC protocol, and several CORBA ORBs offer excellent Python support. This recipe requires multiple files. Here is the interface definition file, fortune.idl, coded in CORBA’s own IDL (Interface Definition Language):
module Fortune { interface CookieServer { string get_cookie( ); }; };
This code is quite readable even if you’ve never seen CORBA’s
IDL before: it defines a module named Fortune
, whose
only contents is an interface named CookieServer
,
whose only contents is a function (method) named
get_cookie
, which takes no arguments and returns a
string. This code says nothing at all about the
implementation: IDL is a language for defining
interfaces.
The server script is a simple Python program:
import sys, os import CORBA, Fortune, Fortune_ _POA FORTUNE_PATH = "/usr/games/fortune" class CookieServer_i(Fortune_ _POA.CookieServer): def get_cookie(self): pipe = os.popen(FORTUNE_PATH) cookie = pipe.read( ) if pipe.close( ): # An error occurred with the pipe cookie = "Oh dear, couldn't get a fortune " return cookie orb = CORBA.ORB_init(sys.argv) poa = orb.resolve_initial_references("RootPOA") servant = CookieServer_i( ) poa.activate_object(servant) print orb.object_to_string(servant._this( )) # see the Discussion session about what this print statement emits poa._get_the_POAManager( ).activate( ) orb.run( )
And here’s a demonstration of client code for this server, using a Python interactive command shell:
>>> import CORBA, Fortune >>> orb = CORBA.ORB_init( ) >>> o = orb.string_to_object( ... "corbaloc::host.example.com/fortune") >>> o = o._narrow(Fortune.CookieServer) >>> print o.get_cookie( )
CORBA has a reputation for being hard to use, but it is really very easy, especially with Python. This example shows the complete CORBA implementation of a fortune-cookie server and its client. To run this example, you need a Python-compatible CORBA implementation (i.e., an ORB)—or, if you wish, two such ORBs, since you can use two different CORBA implementations, one for the client and one for the server, and let them interoperate with the CORBA IIOP inter-ORB protocol. Several free CORBA implementations, which fully support Python, are available for you to download and install. The Python language support is part of the CORBA standards, so, if a certain ORB supports Python at all, you can code your Python source for it in just the same way as you can code it for any other compliant ORB, be it free or commercial. In this recipe, we use the free ORB known as omniORB. With omniORB, you can use omniORBpy, which lets you develop CORBA applications from Python.
With most ORBs, you must convert the interface definition coded in IDL into Python declarations with an IDL compiler. For example, with omniORBpy:
omniidl -bpython fortune.idl
This creates Python modules named Fortune
and Fortune_ _POA
, in files Fortune.py and Fortune_POA.py, to be used by clients and
servers, respectively.
In the server, we implement the CookieServer
CORBA interface by importing Fortune_ _POA
and
subclassing the CookieServer
class that the module
exposes. Specifically, in our own subclass, we need to override the
get_cookie
method (i.e., implement the methods that
the interface asserts we’re implementing). Then, we start CORBA to get
an orb
instance, ask the ORB for a
POA (Portable Object Adaptor), instantiate our own
interface-implementing object, and pass it to the POA instance’s
activate_object
method. Finally, we
call the activate
method on the POA
manager and the run
method on the
ORB to start our service.
When you run the server, it prints out a long hex string, such as:
IOR:010000001d00000049444c3a466f7274756e652f436f6f6b69655365727665723 a312e300000000001000000000000005c000000010102000d0000003135382e313234 2e36342e330000f90a07000000666f7274756e6500020000000000000008000000010 0000000545441010000001c0000000100000001000100010000000100010509010100 0100000009010100
Printing this string is the purpose of the object_to_string
call that our recipe’s
server performs just before it activates and runs.
You have to pass this string value as the argument of the
client’s orb.string_to_object( )
call to contact your server. Such long hex strings may not be
convenient to communicate to clients. To remedy this, it’s easy to
make your server support a simple corbaloc
URL string, like the one used in
the client example, but doing so involves omniORB-specific code that
is not necessarily portable to other ORBs. (See the omniORBpy manual
for details of corbaloc
URL
support.)
You can download omniORBpy, including its documentation, from http://www.omniorb.org/omniORBpy/.
Credit: Jeff Bauer
You need to send commands to one or more logins that can be on a local machine, or a remote machine, and the Telnet protocol is acceptable.
Telnet is one of the oldest protocols in the TCP/IP stack, but
it may still be serviceable (at least within an intranet that is well
protected against sniffing and spoofing attacks). In any case,
Python’s standard module telnetlib
supports Telnet quite well:
# auto_telnet.py - remote control via telnet import os, sys, telnetlib from getpass import getpass class AutoTelnet(object): def _ _init_ _(self, user_list, cmd_list, **kw): # optional parameters are host, timeout in seconds, command # prompt to expect from the host on successful logins: self.host = kw.get('host', 'localhost') self.timeout = kw.get('timeout', 600) self.command_prompt = kw.get('command_prompt', "$ ") # collect passwords for each user, interactively self.passwd = { } for user in user_list: self.passwd[user] = getpass("Enter user '%s' password: " % user) # instantiate Telnet proxy self.telnet = telnetlib.Telnet( ) for user in user_list: # login with given host and user, and act appropriately self.telnet.open(self.host) ok = self.action(user, cmd_list) if not ok: print "Unable to process:", user self.telnet.close( ) def action(self, user, cmd_list): # wait for a login prompt t = self.telnet t.write(" ") login_prompt = "login: " response = t.read_until(login_prompt, 5) if login_prompt in response: print response else: return 0 # supply user and password for login t.write("%s " % user) password_prompt = "Password:" response = t.read_until(password_prompt, 3) if password_prompt in response: print response else: return 0 t.write("%s " % self.passwd[user]) # wait for command prompt to indicate successful login response = t.read_until(self.command_prompt, 5) if self.command_prompt not in response: return 0 # send each command and wait for command prompt after each for cmd in cmd_list: t.write("%s " % cmd) response = t.read_until(self.command_prompt, self.timeout) if self.command_prompt not in response: return 0 print response return 1 if _ _name_ _ == '_ _main_ _': # code which runs as a main script, only basename = os.path.splitext(os.path.basename(sys.argv[0]))[0] logname = os.environ.get("LOGNAME", os.environ.get("USERNAME")) host = 'localhost' import getopt optlist, user_list = getopt.getopt(sys.argv[1:], 'c:f:h:') usage = """ usage: %s [-h host] [-f cmdfile] [-c "command"] user1 user2 ... -c command -f command file -h host (default: '%s') Example: %s -c "echo $HOME" %s """ % (basename, host, basename, logname) if len(sys.argv) < 2: print usage sys.exit(1) cmd_list = [ ] for opt, optarg in optlist: if opt == '-f': for r in open(optarg): if r.rstrip( ): cmd_list.append(r) elif opt == '-c': command = optarg if command[0] == '"' and command[-1] == '"': command = command[1:-1] cmd_list.append(command) elif opt == '-h': host = optarg autoTelnet = AutoTelnet(user_list, cmd_list, host=host)
Python’s telnetlib
lets you
easily automate access to Telnet servers, even from non-Unix machines.
As a flexible alternative to the popen
functions, which only run commands
locally as the user that’s running the script, telnetlib
, which can work across an intranet
and can login and run commands as different users, is a handy
technique to have in your system administration toolbox.
Production code generally has to be made more robust, but this
recipe should be enough to get you started in the right direction. The
recipe’s AutoTelnet
class instantiates a single
telnetlib.Telnet
object and uses
that single object in a loop over a list of users. For each user, the
recipe calls the open
method of the
Telnet
instance to open the
connection to the specified host, runs a series of commands in
AutoTelnet
’s action
method, and
finally calls the close
method of
the Telnet
instance to terminate
the connection.
AutoTelnet
’s action
method is
where the action is. All operations depend on two methods of the
Telnet
instance. The write
method takes a single string argument
and writes it to the connection. The read_until
method takes two arguments, a
string to wait for and a timeout in seconds, and returns a string with
all the characters received from the connection until the timeout
elapsed or the waited-for string occurred. action
’s
code uses these two methods to wait for a login prompt and send the
username; wait for a password prompt and send the password; and then,
repeatedly, wait for a command prompt (typically from a Unix shell at
the other end of the connection) and send the commands in the list
sequentially (waiting for a command prompt again after sending each
one).
One warning (which applies to any use of Telnet and some other old protocols): except when transmitting completely public data, not protected by passwords that might be of interest to intruders of ill will, do not run Telnet (or non-anonymous FTP, for that matter) on networks on which you are not completely sure that nobody is packet-sniffing, since these protocols date from an older, more trusting age. These protocols let passwords and everything else travel in the clear, open to any snooper. This issue is not Python specific; it applies to any implementation of these protocols, since it depends on the definition of the protocols themselves. Whether or not you use Python, be advised: if there is any risk that someone might be packet-sniffing, use SSH instead, as shown next in Recipe 15.10 so that no password ever travels on the network in the clear, and so that the connection stream itself gets encrypted.
Documentation on the standard library module telnetlib
in the Library
Reference; Recipe 15.10.
Credit: Peter Cogolo, Anna Martelli Ravenscroft
You need to send commands, using the SSH protocol, to one or more logins that can be on a local machine or a remote machine.
SSH is a secure replacement for the old Telnet protocol. One way
to use SSH from a Python program is with the third-party paramiko
package:
# auto_ssh.py - remote control via ssh import os, sys, paramiko from getpass import getpass paramiko.util.log_to_file('auto_ssh.log', 0) def parse_user(user, default_host, default_port): ''' given name[@host[:port]], returns name, host, int(port), applying defaults for hose and/or port if necessary ''' if '@' not in user: return user, default_host, default_port user, host = user.split('@', 1) if ':' in host: host, port = host.split(':', 1) else: port = default_port return user, host, int(port) def autoSsh(users, cmds, host='localhost', port=22, timeout=5.0, maxsize=2000, passwords=None): ''' run commands for given users, w/default host, port, and timeout, emitting to standard output all given commands and their responses (no more than 'maxsize' characters of each response). ''' if passwords is None: passwords = { } for user in users: if user not in passwords: passwords[user] = getpass("Enter user '%s' password: " % user) for user in users: user, host, port = parse_user(user, default_host, default_port) try: transport = paramiko.Transport((host, port)) transport.connect(username=user, password=passwords[user]) channel = transport.open_session( ) if timeout: channel.settimeout(timeout) for cmd in cmd_list: channel.exec_command(cmd) response = channel.recv(max_size) print 'CMD %r(%r) -> %s' % (cmd, user, response) except Exception, err: print "ERR: unable to process %r: %s" % (user, err) if _ _name_ _ == '_ _main_ _': logname = os.environ.get("LOGNAME", os.environ.get("USERNAME")) host = 'localhost' port = 22 usage = """ usage: %s [-h host] [-p port] [-f cmdfile] [-c "command"] user1 user2 ... -c command -f command file -h default host (default: localhost) -p default host (default: 22) Example: %s -c "echo $HOME" %s same as: %s -c "echo $HOME" %s@localhost:22 """ % (sys.argv[0], sys.argv[0], logname, sys.argv[0], logname) import getopt optlist, user_list = getopt.getopt(sys.argv[1:], 'c:f:h:p:') if not user_list: print usage sys.exit(1) cmd_list = [ ] for opt, optarg in optlist: if opt == '-f': for r in open(optarg, 'rU'): if r.rstrip( ): cmd_list.append(r) elif opt == '-c': command = optarg if command[0] == '"' and command[-1] == '"': command = command[1:-1] cmd_list.append(command) elif opt == '-h': host = optarg elif opt == '-p': port = optarg else: print 'unknown option %r' % opt print usage sys.exit(1) autoSsh(user_list, cmd_list, host=host, port=port)
The third-party extension paramiko
package lets you easily automate
access to all sorts of SSH services, even from non-Unix machines.
paramiko
even lets you write your
own SSH servers in Python. In this recipe, however, we use paramiko
on the client side, as a more
secure alternative to the similar use of telnetlib
shown previously in Recipe 15.9.
Production code generally has to be made more robust, but this
recipe should be enough to get you started in the right direction. The
recipe’s autoSsh
function first ensures it knows
passwords for all the users (asking interactively for the passwords of
users it doesn’t know about). Then, it loops over all the users,
parsing strings such as foo@bar:2222
to mean user foo
at host bar
, port 2222
, and defaulting the host and port
values, if necessary.
The loop body relies on two types of objects supplied by
paramiko
, Transport
and Channel
. The transport is constructed by
giving it the (
host
, port
)
pair and then a connection is
made with a username and password. (Alternatively, depending on the
SSH server, one might connect using a private key, but this recipe
uses just a password.) The channel is obtained from the transport, and
the recipe then sets a timeout (by default, 6 seconds) to ensure that
no long-term hanging occurs in case of problems
with an SSH server or the network path to it. Finally, an inner loop
over all commands sends each command, receives a response (up to a
maximum length in bytes, 2000 by default), and prints the command and
response.
paramiko
’s home page at
http://www.lag.net/~robey/paramiko/; paramiko
requires another third-party
extension to Python, the Python Cryptography Toolkit, whose home page
is at http://www.amk.ca/python/code/crypto; docs on
SSH at http://www.openssh.com/, http://www.ucolick.org/~sla/ssh/, http://kimmo.suominen.com/docs/ssh/; Richard
Silverman and Daniel J. Barrett, SSH: The Secure Shell, The
Definitive Guide (O’Reilly); Recipe 15.9.
Credit: Rob Riggs
You want your Python application to check SSL client
authentication, by delegating, over HTTPS, to an Apache server that is
running mod_ssl
.
The Apache web server has good support for SSL, and we can write a Python script to exploit that support to authenticate a client. For example:
import httplib CERT_FILE = '/home/robr/mycert' PKEY_FILE = '/home/robr/mycert' HOSTNAME = 'localhost' conn = httplib.HTTPSConnection(HOSTNAME, key_file = PKEY_FILE, cert_file = CERT_FILE) conn.putrequest('GET', '/ssltest/') conn.endheaders( ) response = conn.getresponse( ) print response.read( )
The Solution code assumes that mycert is a certificate file formatted by
PEM (Privacy-enhanced Electronic Mail), which includes both the public
certificate and the private key. You can keep the public and private
keys in separate files: you need to pass the names of the files in
question as the values for the key_file
and cert_file
arguments of HTTPSConnection
.
To safely perform SSL authentication, you will generally set up your own certification authority (CA). You do not want to enable a third-party organization to hand out all the “keys” to the locks that you put up to protect your security.
The Apache server installation that you use for this authentication needs to be configured to require SSL client authentication with the appropriate CA. My httpd.conf file contains the stanza:
SSLCACertificatePath /etc/httpd/conf/ssl.crt SSLCACertificateFile /etc/httpd/conf/ssl.crt/myCA.crt SSLVerifyClient require SSLVerifyDepth 2 SSLRequireSSL
The configuration of an Apache server cannot refer to more than
one SSLCACertificateFile
. You can
put more than one CA certificate in that file, but doing so grants
authentication to any client who has a certificate from any
one of the certificate authorities you accept,
which is unlikely to be what you want. Therefore, this recipe is fully
applicable only when you can reasonably set up an Apache server to
accept your own CA as the sole recognized one. In exchange for this
modest inconvenience, however, you do get a handy and robust approach
to client authentication between web-enabled applications,
particularly good for SOAP or XML-RPC implementations, or custom
applications that communicate via HTTP/HTTPS.
Descriptions of SSL and its use with Apache can be found at
http://httpd.apache.org/docs-2.0/ssl/ssl_howto.html
and http://www.pseudonym.org/ssl/ssl_cook.html.
The httplib
module is part of the
Python Standard Library and is documented in a chapter of the
Library Reference portion of Python’s online
documentation.