6.3. System or Regression Testing

Perl comes with code for testing perl itself, and you can use these same tools for your regression tests. They are deceptively simple. The Test::Harness module maintained by Andreas König exports a routine runtests that takes as arguments the names of programs to run. It looks for a header line of the form “1..max” and output lines that say either “ok n ” or “not ok n ”, where n is between 1 and max; all other output is ignored and discarded. When it's done with all the tests, it prints a summary containing either the magic word “successful” or the dreaded word “FAILED.”

But wait, there's more. The Test module^[2] by Joshua Nathaniel Pritikin makes it easy for your programs to output the “ok/not ok” lines. It exports the routine ok, which evaluates its arguments to determine which kind of line to output (based on a whole slew of convenient possibilities depending on the number and type of arguments), and the subroutine plan, for outputting the header containing the number of tests.

^[2] Both these modules are considerably more powerful than indicated in our brief explanations; we're just hitting the highlights.

Therefore Test provides assertion mechanisms for outputting success/failure information that can then be logged from multiple scripts by Test::Harness, which will output a summary telling you whether the entire test passed or identifying which subtests failed.

As if that weren't enough—these Perl people really are lazy^[3]—Perl comes with yet more utilities to automate the construction of the test framework. The h2xs program creates a skeleton directory tree for a new Perl module (although in a pinch it could be adapted to an application instead), including a file Makefile.PL and a skeleton test script test.pl.

^[3] Remember, that's a virtue.

When you run this file through perl (which you typically need to modify somewhat to reflect additional files and dependencies that your module requires), it generates a much larger makefile (although a size that is typical for today's makefiles). That makefile includes a test target that runs Test::Harness::runtests over test.pl. If you create a subdirectory called t before you create the makefile, make test includes every file in that directory with a name ending in .t in the runtests.

Here's a quick example to show how all these things play together. Let's say that we're going to create a module that exports a function to convert the names of colors to their numerical values. On many Unix systems there is a conversion table in the file rgb.txt, often stored in /usr/lib/X11, so we'll just appropriate that for our use. Let's call this package ColorConv, so we start out with h2xs:

% h2xs -XA -n ColorConv
Writing ColorConv/ColorConv.pm
Writing ColorConv/Makefile.PL
Writing ColorConv/test.pl
Writing ColorConv/Changes
Writing ColorConv/MANIFEST

Then we'll descend into the new directory ColorConv and start editing ColorConv.pm. You'll notice that it's already been populated with documentation stubs and a framework which we leave out of the following version in which we show just our code:

package ColorConv;
use strict;

my %COLOR;
if (open COLORS, '/usr/lib/X11/rgb.txt')
   {
   while (<COLORS>)
      {
      next unless my ($r, $g, $b, $name) =
           /^s*(d+)s+(d+)s+(d+)s+(.*)/;
      $name =~ tr/ //d;
      $COLOR{lc $name} = sprintf "#%02X%02X%02X", $r, $g, $b;
      }
   close COLORS;
   }
   else
   {
   warn
     "Can't open color file ($!), translations impossible
";
   }

sub conv
   {
   $COLOR{lc shift} || 'Unknown';
   }

1;

You'll notice there's also a test.pl created for you. Although it doesn't use Test.pm by default, we'll alter it so that it does:

#!/usr/bin/perl -w
use strict;
use ColorConv;
use Test;

my %color = (bisque  => '#FFE4C4',
     tomato1 => '#FF6347',
     red     => '#FF0000',
     green   => '#00FF00',
     blue    => '#0000FF',
     plum    => '#DDA0DD',
     puce    => 'Unknown'),

plan tests => scalar keys %color;

for (keys %color)
   {
   ok(ColorConv::conv($_), $color{$_});
   }

Now when we run this, the output is as follows:

1..7
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7

And if we were to create the makefile right now (with perl Makefile.PL), that's what we'd see if we typed make test. However, if we create a subdirectory t first and put test.pl in t/testcol.t instead, we can get a higher level perspective with Test::Harness when we run make test:

% make test
PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/
lib/perl5/5.6.0/i586-linux -I/usr/lib/perl5/5.6.0 -e 'use
Test::Harness qw(&runtests $verbose); $verbose=0; runtests
@ARGV;' t/*.t
t/testcol...........ok
All tests successful.

In this case, testcol.t was the only .t file in the t directory. If we were to put more there, each of them would be run in turn, so we could build up a suite of regression tests.

How should you craft regression tests? A deceptively simple question. Jarkko Hietaniemi's advice is, “Try to break your own code. Give functions/programs too few/many arguments, arguments of the wrong type (e.g., a directory when a file is expected). Give undef or empty or huge scalars, strings when numbers are expected, and vice versa. Think Evil.” You'll notice we took that advice to heart when we added a test that the absent (unfairly, in our opinion) color puce caused the right kind of error. Your regression tests should exercise as many as possible of the failure modes your code can produce.

Perl runs approximately 12,000 tests using this framework when you type make test for the perl distribution itself, so the capability can support large projects.

6.3.1. Coverage Analysis

At this point we might want to determine whether we're really testing all the code in our application or whether some is getting missed. The CPAN module Devel::Coverage by Randy J. Ray does just this. It's a debugger plugin that counts the number of times each executable line of code is visited.^[4] You can then look for any that has a count of zero.

^[4] Since it can't distinguish among multiple statements on the same line, don't put multiple statements on the same line if one might be executed but not another.

Here's an example. (In practice, this tool would be most useful on much larger programs.) Here we've included our test data in the program itself to make it more self-contained.

 1  #!/usr/bin/perl -wd:Coverage
 2  use strict;
 3
 4  my (%scientific, %float, %integer);
 5  while (<DATA>) {
 6      chomp;
 7      if (/(-?d+.d?E[-+]?d+)/i) {
 8          $scientific{$1}++;
 9      }
10      elsif (/(-?d+.d?)/) {
11          $float{$1}++;
12      }
13      elsif (/(-?d+)/) {
14          $integer{$1}++;
15      }
16  }
17
18  __END__
19  4
20  4.5
21  -3E3
22  5
23  6
24  3.2

When we run this program, it puts its output in a file with the same name but with .cvp appended. This is in a raw format that is converted to something appealing by running it through the coverperl program that comes with Devel::Coverage:

3    line 2
1    line 4
7    line 5
6    line 6
6    line 7
0    line 8
2    line 11
4    line 14

(Only the executable lines are listed.) We can quickly see that line 8 was never executed, and a quick check of our code shows that the regex didn't match the input data; either the regex or the data must be at odds with the requirements.