Tools You Cannot Do Without

The following sections provide a short introduction to several kinds of tools, give an overview of their interrelation, and highlight their use. Not intended to be a tutorial, they provide some basic information needed in later parts of the book. For more information on specific tools, refer to the manuals provided with those tools. On the UNIX operating system, these man pages can be accessed via the man command by typing man toolname .

Users of Microsoft's Developer Studio can access many tools via its main menu; however, some tools are supplied only with the Professional or Enterprise Edition—or early editions—of the Developer Studio.

Whether the tools are used via an integrated GUI environment or not, they all serve the same basic purpose. As such, this chapter can be of interest regardless of the preferred system of use.

The Compiler

The compiler is a tool that transforms a program written in a high-level programming language into an executable binary file. Examples of high-level programming languages are Pascal, C, C++, Java, Modula 2, Perl, Smalltalk, and Fortran. To make an executable, the compiler makes use of several other tools: the preprocessor, the assembler, and the linker. Although these tools are often hidden behind the user interface of the compiler itself, it can be quite useful to know more about them, perhaps even to use them separately.

Listing 4.1 shows an example of compiler input. This is a C program that prints the text Hello, Mom! onscreen.

Code Listing 4.1. Listing Hello.c

#include <stdio.h>

int main(int argc, char *argv[])
{
    printf("Hello, Mom! n");
}

This chapter uses the GNU C Compiler (gcc) as the example compiler. It is available for free under the GNU General Public License as published by the Free Software Foundation . There also is a C++ version of this compiler (often embedded in the same executable) which can be called with the g++ command. In all this chapter's examples, the C compiler can be substituted by the C++ compiler.

The command gcc hello.c -o hello will translate the source file hello.c into an executable file called hello. This means the whole process—from preprocessor to compiler to assembler to linker—has been carried out, resulting in an executable program. It is possible, however, to suppress some phases of this process or to use them individually. By using different compiler options, you can exert a high degree of control over the compilation process. Some useful examples are

gcc -c hello.c

This command will preprocess, compile, and assemble the input file, but not link it. The result is an object file (or several object files when more than one input file is specified) which can serve as the input for the linker. The linker is described later in this chapter in the section "The Linker." Object files generally have the extension .o. The object file generated by this example will be called hello.o.

gcc -c hello.s

When the compiler receives an Assembly file as input, it will translate this as easily into an object file. There is no functional difference between object files generated from different source languages, so a hello.o generated from a hello.s will be handled the same by the linker as a hello.o generated from a hello.c.

gcc -S hello.c

This option suppresses the compilation process even earlier, resulting not in an object file but in an Assembly file: hello.s (this file could be used as input in the previous example).

Listing 4.2 provides the assembly output generated by the compiler for an Intel 80x86 processor, using the hello.c example as input. Depending on the compiler used, output may differ somewhat.

Code Listing 4.2. The Assembly Version of hello.c

        .file    "hello.c"
        .version    "01.01"
gcc2_compiled.:
        .section    .rodata
.LC0:
        .string    "Hello, Mom! n"
.text
        .align 16
.globl main

        .type     main,@function
main:
        pushl %ebp
        movl %esp,%ebp
        pushl $.LC0
        call printf
        addl $4,%esp
.L1:
        movl %ebp,%esp
        popl %ebp
        ret
.Lfe1:
        .size     main,.Lfe1-main
        .ident    "GCC: (GNU) egcs-2.91.66 19990314 
							
							
							
							
							 (egcs-1.1.2 release)"

This Assembly code can be turned into an object file by invoking the assembler separately. This way, it is possible to fine-tune the Assembly code generated by the compiler. The assembler is described in section "The Assembler," later in this chapter.

gcc -E hello.c

This command will only preprocess the input file. Because the output generated by the preprocessor is rather lengthy and involved, it is therefore discussed separately in the section, "The Preprocessor," which can be found later in this chapter.

Figure 4.1 shows the process by which an executable is obtained.

Note that no output file has been specified for the precompiler. When the precompiler is used separately from the compiler, it sends its output to screen. This output can, of course, be redirected to a file (gcc -E hello.c >> outputfilename.txt).

A final remark concerning compilers: Most compilers will simply take any intermediate file you throw at them and translate it into an executable. For instance, gcc hello.o -o hello will just link the provided object file and generate an executable, and gcc hello.s -o hello will assemble and link the provided Assembly file into an executable.

The Preprocessor

As you discovered in the previous section, the -E option of the compiler command allows us to use the precompiler separately. Doing its name proud, the preprocessor takes care of several processes before the compiler starts its big task. The following sections describe the main tasks performed by the preprocessor.

Include File Substitution

The #include statements are substituted by the content of the file they specify. The include line from the example

#include <stdio.h>

will be replaced with the contents of the file stdio.h. Within this file, further includes will be found which in turn will be substituted. This is the reason care should be taken not to define circular includes. If stdio.h were to include myinclude.h and myinclude.h were to again include stdio.h, the preprocessor would be told to substitute indefinitely. Most preprocessors can deal with these kinds of problems and will simply generate an error. It is wise, however, to always use conditional statements when handling includes (see the section "Conditional Substitution," later in this chapter).

The <> characters tell the preprocessor to look for the specified file in a predefined directory. Using double quote characters ("") tells the preprocessor to look in the current directory:

#include "myinclude."

In practice, you will find that you use double quotes when including the header files you have written yourself and <> for system header files.

Macro Substitution

Macro substitution works in a way which is very similar to include file substitution. When you use a macro, you define a symbolic name (called the macro) for what is in effect a string. The preprocessor hunts down all the macros and replaces them with the strings they represent. This can save the programmer the trouble of typing the string hundreds of times. An often-used macro is the following:

#define MAX(A,B) ((A) > (B) ? (A) : (B))

Anywhere in the code where you use the macro MAX(variable1, variable2) , this code will be substituted by the preprocessor. The compiler will find the following string:

 ((variable1) > (variable2) ? (variable1) : (variable2))

Note that this macro happens to look like a function call. The downside to this approach (using a macro instead of a real function) is that the code is found everywhere in the program where the macro is used. The upside is that there is no function call overhead. The use of macros during optimization will be discussed in detail in Chapter 8, "Functions."

Conditional Substitution

The preprocessor can also be used to conditionally substitute pieces of code. This means the preprocessor determines which pieces of code the compiler will get to see (and translate). An example of this with which you are probably familiar is the following conditional statement:

#ifndef __HEADERFILENAME__
#define __HEADERFILENAME__
~
#endif

This piece of code tells the preprocessor that only if the symbol __HEADERFILENAME__ is not yet defined, it can continue. The next statement the preprocessor encounters is one defining the very same symbol. This means that if the preprocessor encounters the file containing this piece of code again, it will skip the rest of the text until it has passed the #endif statement. When this piece of code can be found in the file myinclude.h, including this file twice will not cause the code represented by the ~ character to be substituted more than once.

Another often-used example of conditional substitution is generating different versions of executables from the same source file:

~
a += a + 1;

#ifdef DEBUG
printf("The value of a = %d", a);
#endif

b += a;
~

Compiling this piece of code can generate a debug version which will print the value of a onscreen (when #define DEBUG is found somewhere by the preprocessor) or a normal version which does not print a; when DEBUG is defined nowhere in the program. Be sure, however, that the definition of DEBUG is encountered by the preprocessor before any conditional DEBUG statements are encountered.

Symbolic Substitution

Symbolic substitution is basically a simplified macro; again a symbolic name is replaced by a string or a value:

#define DEBUG 1
#define FILENAME "input.c"

When the preprocessor is used separately, it becomes possible to determine whether compiler errors are generated from within the code or from somewhere in the include file hierarchy. You can study the preprocessor output to determine whether or not the macros are expanded the way they are expected to be.

The Assembler

An assembler is a tool that translates human-understandable mnemonics into microprocessor instructions (executable code). Mnemonics are in fact nothing more than symbolic text representations of those microprocessor instructions. For instance, the mnemonic ADD A,20 (which adds 20 to the value of register A of the microprocessor) will be translated by the assembler into the following numbers: 198 and 20. It should not surprise you to see that microprocessor instructions (or machine instructions) are numbers; the computer memory holds nothing but numbers. This example shows that mnemonics are highly machine dependent, as every mnemonic instruction represents a microprocessor instruction for a very specific microprocessor (the example is taken from Z80 mnemonics). An Assembly program written for a 68000 microprocessor cannot simply be assembled to run on an 80386. The program is not portable and has to be completely rewritten. Another quality of this close proximity to the actual hardware is that Assembly is very useful for optimizing programming routines for speed/footprint.

Listing 4.3 shows a small piece of a mnemonic listing for a 68000 microprocessor :

Code Listing 4.3. An 68000 Assembly File

         move.l     (a0), d0
loop:    jsr        subroutine
         sub        #1, d0
         cmpi       #400, d0
         bne        loop
         move.l     (a1), d0

This example will be fairly easy to read for someone familiar with 68000 mnemonics. However, Assembly listings (programs) tend to become lengthy and involved. Often, the goal of a piece of code does not become apparent until a large part has been studied, because a single higher programming language instruction will be represented by many Assembly instructions. Development time is therefore high and looking for bugs in Assembly listings is quite a challenge. Also, working with Assembly requires the programmer to have a high level of system-specific technical knowledge.

In the early days of computer programming (even until the late 1980s), the use of Assembly languages was quite common. High-level languages, such as Pascal or C, were not that common, and their predecessors were slow (in both compiling and executing). The generated code was far from optimal. Nowadays, compilers can often do a far better job of optimizing Assembly than most developers could, so Assembly is only used where software and hardware interact closely. Think of parts of operating systems, such as device drivers, process schedulers, exception handlers and interrupt routines. In practice, when you find Assembly, it is most often embedded within sources of a higher programming language. Listing 4.4 shows the use of Assembly in a C++ listing:

Code Listing 4.4. A Listing Mixing 68000 and C/C++

void f()
{
    printf("value of a = %d", a);
}
#asm
    label:     jsr       pushregs
               add.l     #1, d0
               rts
#endasm

It might surprise you to see that languages can be mixed and matched like this. If so, it will surprise you even more that C and C++ can even be mixed to take advantage of their independent merits; there's more about this in the section Mixing Languages.Note that mixing Assembly with a higher-level language makes the source less portable because the Assembly will still only work on one specific microprocessor.

In the section "The Compiler," you saw how to make an executable from an Assembly file. It is, however, also possible to generate an object file from an Assembly file. This is done by calling the assembler with the -o option:

as hello.s -o hello.o

The generated object file can serve as input for the linker, which is the subject of the next section.

The Linker

Using the linker is the final phase in the process of generating an executable program. The linker takes the libraries and object files that are part of the project and links them together. Often, the linker will also add system libraries and object files to make a valid executable. This might not always seem obvious, but an example will make that clear; because stdio.h is included in the hello.c program, it can use the printf command. This command, however, is not part of some ROM library, so the code implementing the function has to come from somewhere else. This is why the linker will have to add a system library to the executable.

To use the linker separately, use the command

ld hello.o -o hello <system options + libraries>

The previous sections show how to transform the hello.c program into an executable with a single command and how to break this command up into separate steps. Listing 4.5 is a short example of using the separate steps together to complete the whole process:

Code Listing 4.5. Separate Steps to Create an Executable

gcc -S hello.c
as hello.s -o hello.o
ld hello.o -o hello <system options + libraries>

As expected, the result of this example is the same as that of the command:

gcc hello.c -o hello

The Make Utility

The Make utility is very closely related to the chain of processes that creates an executable from a higher-level source file. As the previous sections show, you can use a single command to create the executable of the example program hello.c. But this program is simple; it does not use multiple modules or have complex project interdependencies. As programs grow, they are often split into functional modules, use third-party libraries, and so on. Compiling such a program (project) can be made complicated by the interdependencies between modules (and even between source files!). The Make utility aids in checking up on dependencies and recompiling all objects that are affected by a change in one or more source files, making it possible to rebuild the entire software tree with one single command. This section discusses only the most important Make options because there are enough to fill entire chapters.

The Make utility uses an input file (called a makefile ) that describes the project (program) is has to make.

This makefile defines the project interdependencies to be checked by the utility. Listing 4.6 shows the makefile that can be used for the example program.

Code Listing 4.6. A Makefile

#This is the makefile for Hello.c

OBJS = hello.o
LIBS =

hello: $(OBJS)
   gcc -o hello $(OBJS) $(LIBS)

You can now build the example program by simply typing make on the command line when the current directory contains the makefile. It is, of course, possible to put the project definition in a file with a different name, but the Make utility then has to be told explicitly which file to look for:

make -f mymakefile

This is useful when you want to create more than one project definition in the same directory (perhaps to build different versions of the software from the same directory of source files).

Let's look at Listing 4.6 line by line and see what is happening:

Line 1: This simply is a comment; it could have been any text that might be useful.

Line 2: This defines the objects needed for the target; it actually defines a variable OBJS which will be used in the makefile from now on.

Line 3: This defines the variable LIBS, which again can be used in the remainder of the makefile. Because the hello program is so simple, it is not actually needed.

Line 5: This is the definition of a rule that states that the target hello depends on the content of the variable OBJS.

Line 6: This line actually lists the command to build the target. There can be multiple command lines, as long as they always start with a tab.

For more makefile commands, you are referred to the appropriate manuals. and the section "The Profiler," later in this chapter, where a slightly more complex makefile is explained.

The Debugger

Bug searching is an activity every programmer will have to do sooner or later. And when saturating the sources with print statements has finally lost its charm, the next logical step is using the debugger. The power of the debugger lies in the fact that it allows the user to follow the path of execution as the program runs. And at any time during the execution, the user can decide to look at (and even change) the value of variables, pointers, stacks, objects, and so on. The only precondition is that the executable contains special debugging information so that the debugger can make connections between the executable and the sources; it is more useful to look at the source code while debugging a program than to be presented with the machine instructions or the Assembly listings. Adding debug information will make the executable larger, so when a program is ready to be released, the whole project should be rebuilt once more but without debug information.

Adding debug information to the executable can be done by adding the -g option to the compile command :

gcc -g hello.c -o hello

The generated executable can now be run by the debugger. To start the GNU debugger, type

gdb hello

Countless debugging tools are available, but the terminology used is generally the same. The following sections provide an overview of important terms.

Run

Run starts the program execution. It is important to note that run will continue running the program until the end or until a breakpoint (see "Breakpoint," later in this chapter) is encountered. In general you set breakpoints and then run the program, or single step (see Step) into the program to start debugging activities.

Step

A step executes a single source file statement. As opposed to run, a step executes only one statement and then waits for further instructions. Executing part of a program this way is called single-stepping .

Step Into/Step Out Of

When single-stepping through a program, you can decide between seeing a call to a function as a single instruction (and stepping, in effect, over the function call) or directing the debugger to step into the function and debug it. Once inside a function, it is possible to skip the rest of the statements of the function and return to the place where it was called. In that case, you decide to step out of the function.

Breakpoint

A breakpoint is a predefined point in a source file where you want the debugger to stop running and wait for your instructions. You generally set a breakpoint on a statement close to where you think the program will do something wrong. You will then run the program and wait for it to reach the breakpoint. At that point, the debugger will stop and you can check variables and single-step (see Step) through the rest of the suspicious code.

Conditional Breakpoint

A conditional breakpoint is a breakpoint that is seen by the debugger only when a certain condition has been satisfied (i > 10 or Stop != TRUE). Instead of simply stopping at an instruction, you can tell the debugger to stop at that instruction only when a certain variable reaches a certain value. For example, say a routine goes wrong after 100 iterations of a loop. You would need 100 run commands to get to that program state using a normal breakpoint. With the conditional breakpoint, you can tell the debugger to skip the first 100 iterations.

Continue

Sometimes this is also called Run . The continue is a run from whatever place in the code you might have reached with the debugger. Say you stop on a breakpoint and take a few single steps, you can decide to continue to the next breakpoint (or the end of the program if the debugger encounters no more breakpoints).

Function Stack/Calling Stack

The calling stack is the list of functions that was called to arrive in the program state that you are examining. When the debugger arrives at a breakpoint it might be useful to know what the path of execution was. You might be able to arrive in function A() via two different calling stacks: Main() -> CheckPassword() -> A() and Main()->ChangePassword()-> A(). Sometimes it is even possible to switch to the program state of a previous call, making it possible to examine variables before the routine which is under investigation is called. By going back to CheckPassword() or even Main() you can see, for instance, what went wrong with the parameters received by A().

Watch

A watch is basically a view on a piece of memory. During debugging, it is possible to check the value of variables, pointers, arrays, and even pieces of memory by putting a watch on a specific memory address. Some debuggers will even allow you to watch the internal state of the microprocessor (registers and so on).

Effects When Debugging

A popular philosopher once remarked that it is impossible to observe anything without influ encing that which is being observed, and sadly this is also true when debugging programs. A program that is being debugged will not behave exactly the same as when it is running normally. One reason for this is the fact that the binary image is different; because of the addition of debug information, the variables, constants, and even functions will reside at different relative addresses within the executable. This means that overwriting memory boundaries and using corrupted (function) pointers will have a different effect during debugging. The bug is still there, but its effect might be more (or even less) noticeable. Another reason why a program can behave differently during a debugging session is that single stepping through the code affects its timing. This is especially important to remember when debugging multithreaded programs. But even during the running of the program, the timing will be slightly different because of the debugging overhead.

A final remark concerning debugging is the suggestion of using the debugger to do preventive maintenance, as mentioned in Chapter 3, "Modifying an Existing System." By single stepping through a few iterations of a complex routine, bugs might be found before they have a chance to go out into the world and celebrate.

The Profiler

The profiler is the most important tool related to performance optimizations. This tool can determine how often each function of a program is called, how much time is spent in each function, and how this time relates (in percentages) to the other functions. In determining which parts of a program are the most-important candidates for optimization, such a tool is invaluable. There is no point in gaining optimizations in functions which are not on the critical path, and it is a waste of time to optimize functions which are seldom called or are fairly optimal to begin with. With the profiler, you look for functions that are called often and that are relatively slow. Any time won in these functions will make the overall program noticeably faster. An important side note is that one should be careful in trying to optimize user input functions. These will mostly just seem slow because they are waiting for some kind of user interaction!

As with the debugger, the use of the profiler warrants some preparations made to the executable file. Follow these steps to use the profiler:

Listing 4.7 will be run through the specified steps. Note that the new example program used is more interesting for the profiler.

Code Listing 4.7. Program to Profile

#include <stdio.h>

long mul_int(int a, int b)
{
        long i;
        long j=0;

        for (i = 0 ; i < 10000000; i++)
           j += (a * b);

        return(j);
}

long mul_double(double a, double b)
{
        long i;
        double j=0.0;

        for (i = 0 ; i < 10000000; i++)
           j += (a * b);

        return((long)j);
}

int main(int argc, char *argv[])
{
        printf("Testing Int    : %ld n", mul_int(1, 2));
        printf("Testing Double : %ld n",
							
							 mul_double(1, 2));
        exit(0);
}

This program contains two almost-identical functions. Both functions perform a multiplication; however, they use different variable types to calculate with: mul_int uses integers and mul_double uses doubles. The profiler can now show us which type is fastest to calculate with.

Compile this program with the following command line:

gcc -pg test.c -o test

And run it with the command line:

test

The following output should appear onscreen:

Testing Int        : 20000000
Testing Double     : 20000000

The output file that was generated during the run is called gmon.out. It contains the results of the profile action and can be viewed with the following command line:

gprof test > test.out

This command will redirect the output of the profiler to yet another file called test.out, which is a normal text file that can be viewed with any editor. Because the test.out file contains a lot of information and explanation, only the most-important part is shown in Listing 4.8 and discussed in this section.

Code Listing 4.8. Profile Output

Each sample counts as 0.01 seconds.
  %     cumulative     self                self     total
 time     seconds     seconds  calls    us/call    us/call   name
 62.50      0.10       0.10      1    100000.00   100000.00  mul_double
 37.50      0.16       0.06      1     60000.00    60000.00  mul_int
  0.00      0.16       0.00      1         0.00   160000.00  main

 %             the percentage of the total running time of the
time           program used by this function.

cumulative     a running sum of the number of seconds accounted
 seconds       for by this function and those listed above it.

 self          the number of seconds accounted for by this
seconds        function alone.  This is the major sort for this
               listing.

calls          the number of times this function was invoked, if
               this function is
							 profiled, else blank.

 self          the average number of milliseconds spent in this
ms/call        function per call, if this function is profiled,
               else blank.

 total         the average number of milliseconds spent in this
ms/call        function and its descendents per call, if this
               function is profiled, else blank.

name           the name of the function.  This is the minor sort
               for this listing. The index shows the location of
               the function in the gprof listing. If the index is
               in parenthesis it shows where it would appear in
               the gprof listing if it were to be
							
							 printed.

The first column tells us that 62.5% of runtime was spent in function mul_double() and only 37.5% in the function mul_int(); the value for main seems to be negligible, and this is not so surprising as the main function only calls two other functions. The numbers are reflected again in column 3 but this time in absolute seconds. Note that the output is sorted by the third column, so it is fairly easy to find the most time-consuming function. It is up to the developer to decide whether this usage of time is acceptable or not.

Listing 4.9 shows what the steps for profiling look like when transformed into a makefile.

Code Listing 4.9. Makefile with Different Targets

#This is the makefile for test.c

PROG    = ./test
PROFOPT = -pg
PROFCMD = gprof
COPT    = -O2
OBJS    = $(PROG).o
LDLIBS  = -L/usr/local/lib/


test:  $(OBJS)
        gcc $(COPT) -o $(PROG) $(OBJS) $(LDLIBS)

profile:$(OBJS)
        gcc $(COPT) $(PROFOPT) $(PROG).c -o $(PROG) $(LDLIBS)
        $(PROG)
        $(PROFCMD) $(PROG)

clean:
        rm $(PROG).o gmon.out $(PROG)

execute:
       
							
							
							 $(PROG)

The makefile presented here has different targets, depending on how the Make is invoked:

As with the debugger, the profiler will influence the program it profiles. This time, however, the implications are not as far reaching because all functions will be influenced almost exactly the same way. The relations between the functions are therefore useable.

The Runtime Checker

The runtime checker is used to detect irregularities during program execution. As with the profiler, the runtime checker will add routines to your program that gather and output information on your program while it is running. Again, it gathers information only on parts of the program that are actually activated during the run, so the test cases are very important here also.

Typical problems a runtime checker will detect are

• Memory leaks (memory is allocated but never released)

• Overflow of the stack (overly recursive function calls)

• File descriptors left open

• Reading from or writing to released memory

• Releasing already released memory

• Reading or writing beyond the bounds of an array or structure

• Using uninitialized memory

• Reading or writing via null pointers

Quite a few commercial runtime checkers are available, and there are freeware checkers available also: Electric Fence, mcheck, and mpr.

The Static Source Code Analyzer

Both the runtime checker and the debugger help you find bugs and irregularities while your program is running. This means you generally have a high level of control over the checking procedure (changing variable values during execution in the debugger, for instance), but you do only check the parts of the program that are activated. The static Source Code Analyzer differs in this respect. In essence, it looks at the source code (all the source code!) and remarks on constructions that it finds suspicious. This is very much like the task the compiler performs, and a lot of analyzer functionality can in fact be invoked by raising the warning levels of most compilers, but the analyzer goes beyond syntactical correctness of the code.

The following example will pass the compilation process with flying colors because it is syntactically correct; in fact, you might have even written this listing on purpose:

for (i = 0; i < 10; i++);
    DoWop();

But it is more that likely that the ; at the end of the first line was unintentional and you wanted the DoWop() function to be called 10 times instead of just once. The Static Source Code Analyzer would remark on this. Other constructions the Analyzer can warn us about are

• Program statements which are never reached:

  {
      ~
      return pResult;
      pResult++;
  }
  

• Variables which are used before being initialized:

  int c;
  return c++;
  

• Logical expressions with a constant value:

  if (a = 1) { ~}
  

It is also possible to compile your program with the -Wall option (warnings all) to get analyzer type functionality:

gcc -Wall hello.c -o hello

Beware that the output from the analyzer can be quite overwhelming, and not all its remarks will be justified. You will have to evaluate each remark on its own merits.

Your Own Test Program

One of the most important development tools you can use is most likely your own test pro gram. Only this piece of software can be equipped with all the information needed for a good, non-generic test. It should be created early in the development process and be able to call all I/O functions of your program and evaluate all returned data/program states. Often this last part is forgotten, but it is as useful, if not more useful, to test the reaction to faulty data and conditions as it is to test whether the program can work under perfect conditions. Where possible, test programs should be able to run automatically, without needing constant user interaction, and even use a mechanism for regression tests (some kind of scripting, perhaps). The idea behind this is that when you add new functionality to a program, you should always retest all the old functionality. If it is possible to do this at night via an automatic script, all the better.

Other advantages to automatic testing:

• Running a test twice will guarantee it is conducted the exact same way; this means incurred problems are highly reproducible, which is important in finding out if a new build has solved the identified problem.

• Automatic testing can take place overnight and no development time needs to be invested.

• The test output can be logged and compared (automatically) to other test outputs; this is ideal to check regression tests.

• Tests can easily be expanded to include new test cases.

There are, of course, tools to aid you in creating test applications, but there is nothing wrong with spending time on writing a C or C++ program. The invested time will definitely pay itself back.