Pylint is another open source static analyzer originally created by Logilab. Pylint is more complex than Pyflakes; it allows more customization. However, it is slower than Pyflakes. For more information check out http://www.logilab.org/card/pylint_manual.
In this recipe, we will again download the NumPy code from the Git repository—this step is omitted for brevity.
You can install Pylint from the source distribution. However, there are many dependencies, so you are better off installing with either easy_install
, or pip
. The installation commands are as follows:
easy_install pylint sudo pip install pylint
We will again analyze from the top directory of the NumPy codebase. Please notice that we are getting much more output. In fact, Pylint prints so much text that most of it had to be cut out here:
pylint *.py $ pylint *.py No config file found, using default configuration ************* Module pavement C: 60: Line too long (81/80) C:139: Line too long (81/80) ... W: 50: TODO W:168: XXX: find out which env variable is necessary to avoid the pb with python W: 71: Reimport 'md5' (imported line 143) F: 73: Unable to import 'paver' F: 74: Unable to import 'paver.easy' C: 79: Invalid name "setup_py" (should match (([A-Z_][A-Z0-9_]*)|(__.*__))$) F: 86: Unable to import 'numpy.version' E: 86: No name 'version' in module 'numpy' C:149: Operator not followed by a space if sys.platform =="darwin": ^^ C:202:prepare_nsis_script: Missing docstring W:228:bdist_superpack: Redefining name 'options' from outer scope (line 74) C:231:bdist_superpack.copy_bdist: Missing docstring W:275:bdist_wininst_nosse: Redefining name 'options' from outer scope (line 74) ...
Pylint outputs raw text, by default; but we could have requested HTML output, if desired. The messages have the following format:
MESSAGE_TYPE: LINE_NUM:[OBJECT:] MESSAGE
Pychecker is an old, static analysis tool, which is not very actively developed, but it's good enough to be mentioned here. The last version at the time of writing was 0.8.19, and was last updated in 2011. Pychecker tries to import each module and process it. The code is then analyzed to find issues such as passing incorrect number of parameters, incorrect format strings using non-existing methods, and other problems. In this recipe, we will again analyze code, but this time with Pychecker.
Download the tar.gz
from Sourceforge (http://pychecker.sourceforge.net/). Unpack the tarball and run the following command:
python setup.py install
pip
.We can, alternatively, install Pychecker using pip
:
sudo pip install http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
Let's analyze the code, just like in the previous recipes. The command we need is:
pychecker *.py ... Warnings... ... setup.py:21: Imported module (re) not used setup.py:27: Module (builtins) re-imported ...
Docstrings are strings embedded in Python code that resemble interactive sessions. These strings can be used to test certain assumptions, or just provide examples. We need to use the doctest
module to run these tests.
Let's write a simple example that is supposed to calculate the factorial, but doesn't cover all the possible boundary conditions. In other words, some tests will fail.
Write the docstring with a test that will pass, and a test that will fail. This should look like what you would normally see in a Python shell:
""" Test for the factorial of 3 that should pass. >>> factorial(3) 6 Test for the factorial of 0 that should fail. >>> factorial(0) 1 """
Write the following NumPy code:
return numpy.arange(1, n+1).cumprod()[-1]
We want this code to fail on purpose, sometimes. It will create an array of sequential numbers, calculate the cumulative product of the array, and return the last element.
As previously stated, we need to use the doctest
module to run the tests:
doctest.testmod()
The following is the complete factorial and docstring test example code:
import numpy import doctest def factorial(n): """ Test for the factorial of 3 that should pass. >>> factorial(3) 6 Test for the factorial of 0 that should fail. >>> factorial(0) 1 """ return numpy.arange(1, n+1).cumprod()[-1] doctest.testmod()
We can get verbose output with the -v
option, as shown here:
python docstringtest.py -v Trying: factorial(3) Expecting: 6 ok Trying: factorial(0) Expecting: 1 ********************************************************************** File "docstringtest.py", line 11, in __main__.factorial Failed example: factorial(0) Exception raised: Traceback (most recent call last): File ".../doctest.py", line 1253, in __run compileflags, 1) in test.globs File "<doctest __main__.factorial[1]>", line 1, in <module> factorial(0) File "docstringtest.py", line 14, in factorial return numpy.arange(1, n+1).cumprod()[-1] IndexError: index out of bounds 1 items had no tests: __main__ ********************************************************************** 1 items had failures: 1 of 2 in __main__.factorial 2 tests in 2 items. 1 passed and 1 failed. ***Test Failed*** 1 failures.
Test-driven development (TDD) is the best thing that happened to software development this century. One of the most important aspects of TDD is the almost manic focus on unit testing.
Unit tests are automated tests that test a small piece of code, usually a function or method. Python has the PyUnit API for unit testing. As NumPy users, we can make use of the convenience functions in the numpy.testing
module, as well. This module, as its name suggests, is dedicated to testing.
Let's write some code to be tested.
We start by writing the following factorial function:
def factorial(n): if n == 0: return 1 if n < 0: raise ValueError, "Don't be so negative" return numpy.arange(1, n+1).cumprod()
The code is the same as in the previous recipe, but we added a few checks for boundary conditions.
Now we will write the unit test. Maybe you have noticed that we don't have that many classes in this book; somehow, it didn't seem that necessary.
Let's write a class for a change. This class will contain the unit tests. It extends the
TestCase
class from the unittest
module, which is a part of standard Python. We test for calling the factorial function with:
class FactorialTest(unittest.TestCase): def test_factorial(self): #Test for the factorial of 3 that should pass. self.assertEqual(6, factorial(3)[-1]) numpy.testing.assert_equal(numpy.array([1, 2, 6]), factorial(3)) def test_zero(self): #Test for the factorial of 0 that should pass. self.assertEqual(1, factorial(0)) def test_negative(self): #Test for the factorial of negative numbers that should fail. # It should throw a ValueError, but we expect IndexError self.assertRaises(IndexError, factorial(-10))
The code for the factorial and the unit test in its entirety is as follows:
import numpy import unittest def factorial(n): if n == 0: return 1 if n < 0: raise ValueError, "Don't be so negative" return numpy.arange(1, n+1).cumprod() class FactorialTest(unittest.TestCase): def test_factorial(self): #Test for the factorial of 3 that should pass. self.assertEqual(6, factorial(3)[-1]) numpy.testing.assert_equal(numpy.array([1, 2, 6]), factorial(3)) def test_zero(self): #Test for the factorial of 0 that should pass. self.assertEqual(1, factorial(0)) def test_negative(self): #Test for the factorial of negative numbers that should fail. # It should throw a ValueError, but we expect IndexError self.assertRaises(IndexError, factorial(-10)) if __name__ == '__main__': unittest.main()
The negative numbers test failed, as you can see in the following output:
.E. ====================================================================== ERROR: test_negative (__main__.FactorialTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "unit_test.py", line 26, in test_negative self.assertRaises(IndexError, factorial(-10)) File "unit_test.py", line 9, in factorial raise ValueError, "Don't be so negative" ValueError: Don't be so negative ---------------------------------------------------------------------- Ran 3 tests in 0.001s FAILED (errors=1)
We saw how to implement simple unit tests using the standard unittest
Python module. We wrote a test class, which extends the TestCase
class from the unittest
module. The following functions were used to perform various tests:
Function |
Description |
---|---|
Tests whether two NumPy arrays are equal | |
Tests whether two values are equal | |
Tests whether an exception is thrown |
The NumPy testing
package has a number of test functions that we should know about:
Function |
Description |
---|---|
Raises an exception if two numbers are not equal up to a specified precision. | |
Raises an exception if two numbers are not equal up to a certain significance. | |
Raises an exception if two arrays are not equal up to a specified precision. | |
Raises an exception if two arrays are not equal. | |
Raises an exception if two arrays do not have the same shape, and the elements of the first array are strictly less than the elements of the second array. | |
Fails if a specified exception is not raised by a callable invoked with defined arguments. | |
Fails if a specified warning is not thrown. | |
Asserts that two strings are equal. |
Mocks are objects created as substitutes for real objects, with the purpose of testing a part of the behavior of the real objects. If you have seen the Body Snatchers movie, you already have an understanding of the basic idea. Generally speaking, mocking is only useful when the real objects under test are expensive to create, such as a database connection, or when testing could have undesired side effects; for instance, we might not want to write to the file system or database.
In this recipe, we will test a nuclear reactor—not a real one, of course. This nuclear reactor class performs a factorial calculation that could, in theory, cause a chain reaction with a nuclear disaster as consequence. We will mock the factorial computation with a mock, using the mock
package.
First, we will install the mock package; after which, we will create a mock and test a piece of code.
In order to install the mock
package, execute the following command:
sudo easy_install mock
The nuclear reactor class has a
do_work
method, which calls a dangerous factorial
method, which we want to mock. Create a mock as follows:
reactor.factorial = MagicMock(return_value=6)
This ensures that the mock returns a value of 6
.
We can check the behavior of a mock and from that, the behavior of the real object under test, in several ways. For instance, we can assert that the potentially explosive factorial
method is being called with the correct arguments as follows:
reactor.factorial.assert_called_with(3, "mocked")
The complete test code with mocks is as follows:
from mock import MagicMock import numpy import unittest class NuclearReactor(): def __init__(self, n): self.n = n def do_work(self, msg): print "Working" return self.factorial(self.n, msg) def factorial(self, n, msg): print msg if n == 0: return 1 if n < 0: raise ValueError, "Core meltdown" return numpy.arange(1, n+1).cumprod() class NuclearReactorTest(unittest.TestCase): def test_called(self): reactor = NuclearReactor(3) reactor.factorial = MagicMock(return_value=6) result = reactor.do_work("mocked") self.assertEqual(6, result) reactor.factorial.assert_called_with(3, "mocked") def test_unmocked(self): reactor = NuclearReactor(3) reactor.factorial(3, "unmocked") numpy.testing.assert_raises(ValueError) if __name__ == '__main__': unittest.main()
We pass a string to the
factorial
method to show that the code with mock does not exercise the real code. The unit test works in the same way as the unit test in the previous recipe. The second test here does not test anything.
The purpose of the second test is just to demonstrate what happens if we exercise the real code without mocks.
The output of the tests is as follows:
Working .unmocked . ---------------------------------------------------------------------- Ran 2 tests in 0.000s OK
Mocks do not have any behavior. They are like alien clones pretending to be real people, only dumber than aliens. An alien clone wouldn't be able to tell you the birthday of the real person it is replacing. We need to set them up to respond in an appropriate manner. For instance, the mock returned 6
in this example. We can record what is happening to the mock—how many times it is being called and with which arguments.
BDD (Behavior Driven Development) is another hot acronym that you might have come across. In BDD, we start by defining (in English) the expected behavior of the system under test, according to certain conventions and rules. In this recipe, we will see an example of those conventions.
The idea behind this approach is that we can have people who may not be able to program, write a major part of the tests. A feature written by these people takes the form of a sentence consisting of several steps. Each step is more or less a unit test that we can write, for instance, using NumPy. There are many Python BDD frameworks. In this recipe, we will be using Lettuce to test the factorial function.
In this section, we will see how to install Lettuce, set up the tests, and write the specifications for the test.
In order to install Lettuce, run either of the following commands:
pip install lettuce sudo easy_install lettuce
Lettuce requires a special directory structure for the tests. In the tests
directory, we will have a directory named features
containing the factorial.feature
file, along with the functional descriptions and test code in the steps.py
file:
./tests: features ./tests/features: factorial.feature steps.py
Coming up with the business requirements is a hard job. Writing it all down in such a way that it is easy to test is even harder. Luckily, the requirements for these recipes are pretty trivial—we will just write down different input values and the expected outputs.
We have different scenarios with Given
, When
, and Then
sections, which correspond to different test steps. We will define the following three scenarios for the factorial feature:
Feature: Compute factorial Scenario: Factorial of 0 Given I have the number 0 When I compute its factorial Then I see the number 1 Scenario: Factorial of 1 Given I have the number 1 When I compute its factorial Then I see the number 1 Scenario: Factorial of 3 Given I have the number 3 When I compute its factorial Then I see the number 1, 2, 6
We will define methods that correspond to the steps of our scenario. We should pay extra attention to the text used to annotate the methods. It matches the text in the business scenarios file, and we are using regular expressions to get input parameters.
In the first two scenarios, we are matching numbers, and in the last we match any text. The NumPy fromstring
function is used to create a string from a NumPy array, with an integer data type and comma separator in the string. The following code tests our scenarios:
from lettuce import * import numpy @step('I have the number (d+)') def have_the_number(step, number): world.number = int(number) @step('I compute its factorial') def compute_its_factorial(step): world.number = factorial(world.number) @step('I see the number (.*)') def check_number(step, expected): expected = numpy.fromstring(expected, dtype=int, sep=',') numpy.testing.assert_equal(world.number, expected, "Got %s" % world.number) def factorial(n): if n == 0: return 1 if n < 0: raise ValueError, "Core meltdown" return numpy.arange(1, n+1).cumprod()
In order to run the tests, go to the tests
directory and type the following command:
$ lettuce Feature: Compute factorial # features/factorial.feature:1 Scenario: Factorial of 0 # features/factorial.feature:3 Given I have the number 0 # features/steps.py:5 When I compute its factorial # features/steps.py:9 Then I see the number 1 # features/steps.py:13 Scenario: Factorial of 1 # features/factorial.feature:8 Given I have the number 1 # features/steps.py:5 When I compute its factorial # features/steps.py:9 Then I see the number 1 # features/steps.py:13 Scenario: Factorial of 3 # features/factorial.feature:13 Given I have the number 3 # features/steps.py:5 When I compute its factorial # features/steps.py:9 Then I see the number 1, 2, 6 # features/steps.py:13 1 feature (1 passed) 3 scenarios (3 passed) 9 steps (9 passed)