Chapter 12

Testing and Debugging

There are always times when your code won't do what it should be doing. You see the inputs and work through the code, but somehow, it spits out an output that just shouldn't be possible. This can be one of the most infuriating parts of programming.

Investigating Bugs by Printing Out the Values

There are loads of ways to find out exactly what's happening, but one of the simplest is judicious use of print() statements. By using these to print out the value of every variable, you can usually get to the bottom of what's going on.

Take a look at the following code for a simple menu system. It doesn't produce any errors, but whatever input you give it, it always says "Unknown choice". (It's on the website as chapter12-debug.py):

choices = {1:"Start", 2:"Edit", 3:"Quit"}
for key, value in choices.items():

    print("Press ", key, " to ", value)



user_input = input("Enter choice: ")



if user_input in choices.values():

    print("You chose", choices[user_input])

else:

    print("Unknown choice")

Perhaps you've seen the problem already, but if you haven't, what's the best way to find it? The problem is that the if statement isn't correctly identifying when the user_input is valid, so add some print() statements to see what's happening:

choices = {1:"Start", 2:"Edit", 3:"Quit"}

for key, value in choices.items():

    print("Press ", key, " to ", value)



user_input = input("Enter choice: ")



print("user_input: ", user_input)

print("choices: ", choices)

print("choices.values(): ", choices.values())



if user_input in choices.values():

    print("You chose", choices[user_input])

else:
    print("Unknown choice")

Straight away you should see the problem: choices.values() should be choices.keys(). The code was checking the wrong part of the choices dictionary. Make the change in the code and try running it again. With that bug fixed, everything should be fine.

Oh no, it still doesn't work! There must be another bug. Have another look at the output from the print statements:

user_input: 1

choices: {1: 'Start', 2: 'Edit', 3: 'Quit'}

choices.values(): dict_values(['Start', 'Edit', 'Quit'])

Can you see why it's failing? After the value of variable, the second most important thing is the data type of that value, so expand the print statements to include more details about what's going on there:

print("user_input: ", user_input)

print("choices: ", choices)

print("choices.values(): ", choices.values())

print("type(user_input): ", type(user_input))

for key in choices.keys():

    print("type(key): ", type(key), "key: ", key)

If you run this, it should output:

Press 1 to Start

Press 2 to Edit

Press 3 to Quit

Enter choice: 1

user_input: 1

choices: {1: 'Start', 2: 'Edit', 3: 'Quit'}

choices.values(): dict_values(['Start', 'Edit', 'Quit'])

type(user_input): <class 'str'>

type(key): <class 'int'> key: 1

type(key): <class 'int'> key: 2

type(key): <class 'int'> key: 3y

Unknown choice

Now you can see that the cause of the problem is that user_input is a string, but the keys of choices are integers. The easiest way to solve this is to change the invocation of choices to make the keys strings:

choices = {"1":"Start", "2":"Edit", "3":"Quit"}

Now the basic logic of the program is working as expected; however, it still spits out loads of extra text that the user doesn't want to see. Obviously you could just delete the print() statements, but they may be useful again in the future. You can keep them in the code, but have a flag that can be set to turn them off and on like as follows:

debug = True



choices = {"1":"Start", "2":"Edit", "3":"Quit"}

for key, value in choices.items():

    print("Press ", key, " to ", value)



user_input = input("Enter choice: ")



if debug:

    print("DEBUG user_input: ", user_input)

    print("DEBUG choices: ", choices)

    print("DEBUG choices.values(): ", choices.values())

    print("DEBUG type(user_input): ", type(user_input))

    for key in choices.keys():

        print("DEBUG type(key): ", type(key), "key: ", key)


if user_input in choices.keys():

    print("You chose", choices[user_input])

else:

    print("Unknown choice")

If you encounter any problems in the future, all you need to do is change the debug variable to True. Prefixing all the lines with DEBUG also makes it easy to see which output is normal, and which is debugging.

Finding Bugs by Testing

Debugging is the process of getting rid of problems in your programs. It can be quite challenging, but it can be even more difficult to find the problems in the first place. This might sound silly, but it's true. As a program gets larger, the number of different ways it can be used increases, and the more different ways something can be used, the more places there are to check for bugs.

Imagine, for example, a word processor that has options for style, page layouts, file formats, layout managers, and so on. Bugs could lurk in any of these areas, so it's important for the developers to check to make sure everything's running as it should. They could even hide in combinations; for example, a problem may occur only if a particular font is used with a particular layout.

Checking Bits of Code with Unit Tests

The most basic form of testing programs is the unit test. This is where you take one small piece of code and make sure it's behaving as it should. Typically, these are used to check that individual methods and functions are working properly.

Essentially, all a unit test does is run a piece of code with a particular set of inputs and check that their outputs are correct.

Take, for example, a function that takes a string of characters and returns a string with the same letters, but converted to uppercase. This could be implemented and tested as follows:

def capitalise(input_string):

    output_string = ""

    for character in input_string:

       if character.isupper():

           output_string = output_string + character

       else:

           output_string = output_string + chr(ord(character)-32)

    return output_string


print(capitalise("helloWorld"))

This should behave as expected. It works because the UTF-8 character encoding that Python uses stores characters as numbers, and uppercase letters are 32 places below their lowercase counterparts.

In this example, we've used a simple test case, and we're printing it to the screen to check manually. We can get Python to check the test case for us using the unittest module in the following code:

import unittest



def capitalise(input_string):

    output_string = ""

    for character in input_string:

       if character.isupper():

           output_string = output_string + character

       else:

           output_string = output_string + chr(ord(character)-32)

    return output_string



class Tests(unittest.TestCase):

    def test_1(self):

        self.assertEqual("HELLOWORLD", capitalise("helloWorld"))
        

if _  _name_  _ == '_  _main_  _':

    unittest.main()

This does more or less what the previous code did. If you run it, it'll check one string to make sure "helloWorld" goes to "HELLOWORLD". At this level, it's not much better or worse than just having a print statement.

When you run unittest.main(), Python runs every method in subclasses of unittest.TestCase that start with test_. In this case it's just test_1. The real advantage of using unit tests is that you can combine lots of tests in order to check at a glance whether things have worked properly.

You can add a second test case that checks that the string "hello world" capitalises to "HELLO WORLD":

   def test_2(self):

        self.assertEqual("HELLO WORLD", capitalise("Hello World"))

If you run this, you should get the following output:

FAIL: test_2 (_  _main_  _.Tests)
-------------------------------------------------------------

Traceback (most recent call last):

  File "capitalise.py", line 17, in test_2

    self.assertEqual("HELLO WORLD", capitalise("Hello World"))

AssertionError: 'HELLO WORLD' != 'HELLOx00WORLD'

- HELLO WORLD
? ^

+ HELLOWORLD
? ^

Oh dear, it looks like the test failed. You can see that the space wasn't properly dealt with. If you go back to the original code, you can see that the problem is that everything that isn't an uppercase character gets 32 taken off its UTF-8 value. Since space isn't uppercase, this happens to it too, but this isn't what the program should do.

Test cases should be designed to try a wide range of valid inputs. For example:

class Tests(unittest.TestCase):

    def test_1(self):

        self.assertEqual("HELLOWORLD", capitalise("helloWorld"))

        

    def test_2(self):

        self.assertEqual("HELLO WORLD", capitalise("Hello World"))

        

    def test_3(self):

        self.assertEqual('!"£$%^&*()_+-=',

                         capitalise('!"£$%^&*()_+-='))

     

    def test_4(self):

        self.assertEqual("1234567890", capitalise("1234567890"))

        

    def test_5(self):

        self.assertEqual("HELLO WORLD", capitalise("HELLO WORLD"))
     
    def test_6(self):

        self.assertEqual("`¬#~'@;:,.<>/?",

                         capitalise("`¬#~'@;:,.<>/?"))

If you run these, you'll see that most fail. The problem is a flaw in the program logic. The code leaves it alone if it's an uppercase letter and changes it otherwise. However, what you want is for the code to change it if it's a lowercase letter, and leave it alone otherwise.

If you change the capitalise function to the following, it'll do this:

def capitalise(input_string):

    output_string = ""

    for character in input_string:

       if character.islower():

           output_string = output_string + chr(ord(character)-32)

       else:

           output_string = output_string + character

           

    return output_string

Now if you run the code, you should find that it passes all the tests.

By default, unittest will give you details only if one or more tests fail. Otherwise, it just returns an overall OK. For most purposes, this is what you want, but you can specify how verbose you want the output to be in two ways. If you're running the script from the command line, you can add the -v flag for more output. So, for example, if you've saved the program as capitalise.py, you can run the tests with verbose output using:

python3 capitalise.py -v

Alternatively, you can specify that you want a more verbose output in the code itself by changing:

if _  _name_  _ == '_  _main_  _':

    unittest.main()

to:

if _  _name_  _ == '_  _main_  _':

    unittest.main(verbosity=2)

Before going any further, we should point out that the capitalise() function is here for this example. If you actually need to capitalise text, you should use upper() method of the string class. For example:

>>> 'hello world'.upper()

Getting More Assertive

All these tests have a call to self.assertEqual(). This line tells the unit test module what the output of the test should be. That is, the test should pass if the two values passed are the same, and fail if they're different. This covers a large proportion of cases, but you may wish to check different things. There are a number of different assert methods that you can use in your tests.

These check that various structures are the same:

• assertSequencesEqual(sequence1, sequence2)
• assertListEqual(list1, list2)
• assertTupleEqual(tuple1, tuple2)
• assertSetEqual(set1, set2)
• assertDictEqual(dict1, dict2)

With these structures, you may want to check that the value is in the structure rather than if two structures are the same. The following methods check if a value is in a structure:

• assertIn(value, structure)
• assertNotIn(value, structure)

Strings are a special type of structure and have their own method:

• assertMultiLineEqual(string1, string2)

You can also check values using tests other than equality using these assert methods:

• assertNotEqual(value1, value2)
• assertGreater(value1, value2)
• assertGreaterEqual(value1, value2)
• assertLess(value1, value2)
• assertLessEqual(value1, value2)

There are also a few that allow a margin of error:

• assertAlmostEqual(value1, value2)
• assertNotAlmostEqual(value1, value2)

These check that the two values differ (or not) by less than 0.000001. These are useful if you're testing floating-point functions that might have small rounding errors that are acceptable.

You can also test anything that you can reduce to a True or False value using:

• assertTrue(value)
• assertFalse(value)

Whatever you want to check, each test_ method should have one assert method call that is used to determine the success or failure of that particular test.

You can use these unit tests in a number of ways. There is a style of development called test-driven development that says that the tests are the first thing you should write and then you use those tests as the specifications for the code. Most programmers, though, write the tests towards the end of development to make sure everything's working properly.

Using Test Suites for Regression Testing

Writing programs isn't usually a single effort. You don't usually sit down, create software, and then stop and go on to do something else. Instead, you generally code some of the features, distribute it to users, then fix bugs and add new features in later versions.

There is a risk of breaking things that once worked as you add new features, so it's important to test not only newly created things, but also older things that have worked. Testing older code is called regression testing, and having a properly ordered set of tests makes it really easy.

To make sure that you're not introducing bugs into previously working code, you should rerun the tests after you make any changes. However, as your programs become bigger, you'll end up with more and more tests. Eventually, you'll get to the point where it's not practical to run every test every time. You can group tests together into test suites. These allow you to test just particular areas of your program at a time.

Using the previous code, you can change the final code block to:

if _  _name_  _ == '_  _main_  _':

    letters_suite = unittest.TestSuite()

    symbols_suite = unittest.TestSuite()

    letters_suite.addTest(Tests("test_1"))

    symbols_suite.addTest(Tests("test_2"))

    symbols_suite.addTest(Tests("test_3"))

    symbols_suite.addTest(Tests("test_4"))

    symbols_suite.addTest(Tests("test_5"))

    symbols_suite.addTest(Tests("test_6"))

    all_suite = unittest.TestSuite()

    all_suite.addTest(letters_suite)

    all_suite.addTest(symbols_suite)

    unittest.TextTestRunner(verbosity=2).run(all_suite)

This bit of code by itself does exactly what the previous code block did. That is, it runs all the tests. However, it has grouped them into different test suites. There's letters_suite that runs the test that checks letters, symbols_suite that runs the tests that check symbols, and all_suite that combines both of them. You can use the final line to run any of these three suites.

Using this code, you should find it quite easy to build a simple testing menu to help you make sure that everything's running smoothly.

Testing the Whole Package

Unit testing is great because you can automate it, and quickly check that everything's running properly. However, it doesn't cover everything. Even though everything seems to work properly by itself, you may still find that there are problems when everything comes together.

In commercial software development, after the unit tests have been done, the code will be passed to the quality assurance team to make sure everything's working fine. This team will outline a series of test cases that cover how the software will be used. It should check every aspect of the program and test it with a variety of inputs to make sure it behaves as expected. This is sometimes done manually with testers interacting with the software just as users would, and sometimes by specialist testing software that can simulate mouse and keyboard input.

Of course, it's unlikely that you'll have a quality assurance team to help you with your software, but there are some things you can take from the professional approach. You should be methodical. Before you start testing, make a list of everything that the program does, and come up with test input and expected outputs. You can then go through this list and make sure it's all functioning correctly.

It's probably a bit excessive to do this after every code change, but you should do it periodically, and especially before any big releases.

Making Sure Your Software's Usable

By the time you've finished a program, you know everything there is to know about it. You know how to interact with it, how to get the best out of it, and what all the various options are. Your users, however, don't have any of this knowledge. Your software has to help them understand it and provide enough information for them to know what to do. After all, it doesn't matter how awesome your features are if the users don't know how to invoke them.

User testing is the area of testing devoted to making sure this is possible. In an ideal world, you'd get a room full of people, sit them down in front of your software, and ask them to perform certain tasks and see how they get on. Again, you're unlikely to be able to do this. Sometimes you may be able to persuade a friend or relative to help you out, but the more programs you create, the fewer volunteers you seem to find. The only real solution to this is to listen to people using the software, and make sure to ask for feedback.

How Much Should You Test?

There's an old saying about software bugs that goes, “Absence of proof isn't proof of absence.” Basically, no matter how much you test your software, there's no way of ever proving that there aren't any bugs in it. In fact, it's almost impossible to write software that doesn't have any bugs in it. The purpose of testing isn't to make perfect software, but to make software that's good enough. What “good enough” means will vary from project to project. The more important the software, the more you should test it, but all software deserves at least some testing. It's not as glamorous as implementing new features, but most of the time it is more important to have a few features that are properly tested than have loads that are buggy, so it's worth spending some time writing unit tests and making sure everything's working properly. After all, it could well be your data that the program loses when there's a problem.

Summary

After reading this chapter, you should understand the following a bit better:

• Debugging is the process of removing any problems from the code.
• Judicious use of print() statements can help you find out what the problems are.
• It sometimes helps to have a way of switching these print() statements on and off so you can reuse them if you find more problems.
• Testing is the process of finding bugs in code.
• Unit tests are the most basic form of testing and can be automated using the unittest module.
• Test cases can be grouped together into test suites to help you test particular areas of a program.
• It's easy to accidentally introduce new bugs when you add features, so you should always regression test after you make changes to your code.
• Unit tests won't pick up all problems though, so you should also test at a complete-system level.
• Usability problems are also bugs, so you need to listen to your users to make sure they are addressed.

This brings us to the end of the book. Hopefully, you're now confident and knowledgeable enough to create your own programs. Don't worry if you feel you don't know everything about every aspect of Python, very few people do. If you ever get stuck, you can always refer back to this book, or the Python documentation at http://docs.python.org/3/.

Hopefully you've seen that programming isn't overly complex, and if you break up a problem into small steps, it's usually quite straightforward to code. The main thing to remember is that programming should be fun! Find an area that interests you and explore it. Despite its small size, there's very little you can't do with a Raspberry Pi.