Credit: Brent Burley
You need to run an external process in a Unix-like environment and capture both the output and error streams from the external process.
The popen2
module lets you capture both streams,
but you also need help from
fcntl
to make the streams nonblocking and thus
avoid deadlocks:
import os, popen2, fcntl, FCNTL, select def makeNonBlocking(fd): fl = fcntl.fcntl(fd, FCNTL.F_GETFL) try: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.O_NDELAY) except AttributeError: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.FNDELAY) def getCommandOutput(command): child = popen2.Popen3(command, 1) # Capture stdout and stderr from command child.tochild.close( ) # don't need to write to child's stdin outfile = child.fromchild outfd = outfile.fileno( ) errfile = child.childerr errfd = errfile.fileno( ) makeNonBlocking(outfd) # Don't deadlock! Make fd's nonblocking. makeNonBlocking(errfd) outdata = errdata = '' outeof = erreof = 0 while 1: ready = select.select([outfd,errfd],[],[]) # Wait for input if outfd in ready[0]: outchunk = outfile.read( ) if outchunk == '': outeof = 1 outdata = outdata + outchunk if errfd in ready[0]: errchunk = errfile.read( ) if errchunk == '': erreof = 1 errdata = errdata + errchunk if outeof and erreof: break select.select([],[],[],.1) # Allow a little time for buffers to fill err = child.wait( ) if err != 0: raise RuntimeError, '%s failed with exit code %d %s' % ( command, err, errdata) return outdata def getCommandOutput2(command): child = os.popen(command) data = child.read( ) err = child.close( ) if err: raise RuntimeError, '%s failed with exit code %d' % (command, err) return data
This recipe shows how to execute a Unix shell command and capture the
output and error streams in Python. By contrast,
os.system
sends both streams directly to the
terminal. The presented
getCommandOutput(command)
function executes a command and returns the
command’s output. If the command fails, an exception
is raised, using the text captured from the
command’s stderr
as part of the
exception’s arguments.
Most of complexity of this code is due
to the difficulty of capturing both the output and error streams of
the child process at the same time. Normal (blocking) read calls may
deadlock if the child is trying to write to one stream, and the
parent is waiting for data on the other stream, so the streams must
be set to nonblocking, and select
must be used to
wait for data on the streams.
Note that the second select
call adds a 0.1-second
sleep after each read. Counterintuitively, this allows the code to
run much faster, since it gives the child time to put more data in
the buffer. Without this, the parent may try to read only a few bytes
at a time, which can be very expensive.
If you want to capture only the output, and don’t
mind the error stream going to the terminal, you can use the much
simpler code presented in getCommandOutput2
. If
you want to suppress the error stream altogether,
that’s easy, too. You can append
2>/dev/null
to the command. For example:
ls -1 2>/dev/null
Since Version 2.0, Python includes the os.popen4
function, which combines the output and error streams of the child
process. However, the streams are combined in a potentially messy
way, depending on how they are buffered in the child process, so this
recipe can still help.