Define the Granularity of Your Tests

Writing unit tests takes time, there is no way around this. Software developers are often crunched for time. To reach deadlines, developers tend to skip writing unit tests, because they think they will finish faster that way. Unfortunately, this thinking does not take the whole picture into account. Omitting unit tests will back-fire in the long run. The earlier a bug is detected in the software development process, the less it costs. If a developer finds a bug during unit testing, it can be fixed immediately, before anyone else encounters it. However, if the bug is discovered by QA, then it becomes a much costlier bug. The bug can cause an extra development cycle, requires bug management, has to go back to the development team for a fix, and then back to QA to verify the fix. If a bug slips through the QA process and finds its way to the customer, then it becomes even more expensive.

The granularity of tests refers to their scope. As the following table illustrates, you can initially unit test a database class with just a few test functions, and then gradually add more tests to ensure that everything works as it should.

As you can see, each successive column brings in more-specific tests. As you move from large-grained tests to more finely grained tests, you start to consider error conditions, different input data sets, and different modes of operation.

Of course, the decisions you make initially when choosing the granularity of your tests are not set in stone. Perhaps the database class is just being written as a proof-of-concept and might not even be used. A few simple tests may be adequate now, and you can always add more later. Or perhaps the use cases will change at a later date. For example, the database class might not initially have been written with international characters in mind. Once such features are added, they should be tested with specific targeted unit tests.

Consider the unit tests to be part of the actual implementation of a feature. When you make a modification, don’t just modify the tests so that they continue to work. Write new tests and re-evaluate the existing ones. When bugs are uncovered and fixed, add new unit tests that specifically test those fixes.

Brainstorm the Individual Tests

Over time, you will gain an intuition for which aspects of a piece of code should turn into a unit test. Certain methods or inputs will just feel like they should be tested. This intuition is gained through trial and error, and by looking at unit tests that other people in your group have written. It should be pretty easy to pick out which programmers are the best unit testers. Their tests tend to be organized and frequently modified.

Until unit test creation becomes second nature, approach the task of figuring out which tests to write by brainstorming. To get some ideas flowing, consider the following questions:

1. What are the things that this piece of code was written to do?
2. What are the typical ways each method would be called?
3. What preconditions of the methods could be violated by the caller?
4. How could each method be misused?
5. What kinds of data are you expecting as input?
6. What kinds of data are you not expecting as input?
7. What are the edge cases or exceptional conditions?

You don’t need to write formal answers to those questions (unless your manager is a particularly fervent devotee of this book or of certain testing methodologies), but they should help you generate some ideas for unit tests. The table of tests for the database class contained test functions, each of which arose from one of these questions.

Once you have generated ideas for some of the tests you would like to use, consider how you might organize them into categories; the breakdown of tests will fall into place. In the database class example, the tests could be split into the following categories:

• Basic tests
• Error tests
• Localization tests
• Bad input tests
• Complicated tests

Splitting your tests into categories makes them easier to identify and augment. It might also make it easier to realize which aspects of the code are well tested and which could use a few more unit tests.

Create Sample Data and Results

The most common trap to fall into when writing unit tests is to match the test to the behavior of the code, instead of using the test to validate the code. If you write a unit test that performs a database select for a piece of data that is definitely in the database, and the test fails, is it a problem with the code or a problem with the test? It’s often easier to assume that the code is right and to modify the test to match. This approach is usually wrong.

To avoid this pitfall, you should understand the inputs to the test and the expected output before you try it out. This is sometimes easier said than done. For example, say you wrote some code to encrypt an arbitrary block of text using a particular key. A reasonable unit test would take a fixed string of text and pass it in to the encryption module. Then, it would examine the result to see if it was correctly encrypted.

When you go to write such a test, it is tempting to try out the behavior with the encryption module first and see the result. If it looks reasonable, you might write a test to look for that value. Doing so really doesn’t prove anything, however! You haven’t actually tested the code; you’ve just written a test that guarantees it will continue to return that same value. Often times, writing the test requires some real work; you would need to encrypt the text independently of your encryption module to get an accurate result.

Write the Tests

The exact code behind a test varies, depending on what type of test framework you have in place. One framework, the Microsoft Visual C++ Testing Framework, is discussed later in this chapter. Independent of the actual implementation, however, the following guidelines will help ensure effective tests:

• Make sure that you’re testing only one thing in each test. That way, if a test fails, it will point to a specific piece of functionality.
• Be specific inside the test. Did the test fail because an exception was thrown or because the wrong value was returned?
• Use logging extensively inside of test code. If the test fails someday, you will have some insight into what happened.
• Avoid tests that depend on earlier tests or are otherwise interrelated. Tests should be as atomic and isolated as possible.
• If the test requires the use of other subsystems, consider writing stubs or mocks of those subsystems that simulate the modules’ behavior so that changes in loosely related code don’t cause the test to fail.
• Ask your code reviewers to look at your unit tests as well. When you do a code review, tell the other engineer where you think additional tests could be added.

As you will see later in this chapter, unit tests are usually very small and simple pieces of code. In most cases, writing a single unit test will take only a few minutes, making them one of the most productive uses of your time.

Run the Tests

When you’re done writing a test, you should run it right away before the anticipation of the results becomes too much to bear. The joy of a screen full of passing unit tests shouldn’t be minimized. For most programmers, this is the easiest way to see quantitative data that declares your code useful and correct.

Even if you adopt the methodology of writing tests before writing code, you should still run the tests immediately after they are written. This way, you can prove to yourself that the tests fail initially. Once the code is in place, you have tangible data that shows that it accomplished what it was supposed to accomplish.

It’s unlikely that every test you write will have the expected result the first time. In theory, if you are writing tests before writing code, all of your tests should fail. If one passes, either the code magically appeared or there is a problem with the test. If the code is done and tests still fail (some would say that if tests fail, the code is actually not done), there are two possibilities: the code could be wrong, or the tests could be wrong.

Running unit tests must be automated. This can be done in several ways. One option is to have a dedicated system that automatically runs all unit tests after every continuous integration build, or at least once a night. Such a system must send out e-mails to notify developers when unit tests are failing. Another option is to set up your local development environment so that unit tests are executed every time you compile your code. For this, unit tests should be kept small and very efficient. If you do have longer-running unit tests, put these separate, and let these be tested by a dedicated test system.

Introducing the Microsoft Visual C++ Testing Framework

Microsoft Visual C++ comes with a built-in testing framework. The advantage of using a unit testing framework is that it allows the developer to focus on writing tests instead of dealing with setting up tests, building logic around tests, and gathering results. The following discussion is written for Visual C++ 2017.

To get started with the Visual C++ Testing Framework, you have to create a test project. The following steps explain how to test the ObjectPool class:

1. Start Visual C++, create a new project, select Visual C++ ➪ Test ➪ Native Unit Test Project, give the project a name, and click OK.
2. The wizard creates a new test project, which includes a file called unittest1.cpp. Select this file in the Solution Explorer and delete it, because you will add your own files.
3. Add empty files called ObjectPoolTest.h and ObjectPoolTest.cpp to the newly created test project.
4. Add an #include "stdafx.h" as the first line in ObjectPoolTest.cpp. (This line is required for the precompiled header feature of Visual C++.)

Now you are ready to start adding unit tests to the code.

The most common way is to divide your unit tests into logical groups of tests, called test classes. You will now create a test class called ObjectPoolTest. The basic code in ObjectPoolTest.h for getting started is as follows:

#pragma once
#include <CppUnitTest.h>

TEST_CLASS(ObjectPoolTest)
{
    public:
};

This code defines a test class called ObjectPoolTest, but the syntax is a bit different compared to standard C++. This is so that the framework can automatically discover all the tests.

If you need to perform any tasks that need to happen prior to running the tests defined in a test class, or to perform any cleanup after the tests have been executed, then you can implement an initialize method and a cleanup method. Here is an example:

TEST_CLASS(ObjectPoolTest)
{
    public:
        TEST_CLASS_INITIALIZE(setUp);
        TEST_CLASS_CLEANUP(tearDown);
};

Because the tests for ObjectPool are relatively simple and isolated, empty definitions will suffice for setUp() and tearDown(), or you can simply remove them altogether. If you do need them, the beginning stage of the ObjectPoolTest.cpp source file is as follows:

#include "stdafx.h"
#include "ObjectPoolTest.h"

void ObjectPoolTest::setUp() { }
void ObjectPoolTest::tearDown() { }

That’s all the initial code you need to start developing unit tests.

Writing the First Test

Because this may be your first exposure to the Visual C++ Testing Framework, or to unit tests at large, the first test will be a very simple one. It tests whether 0 < 1.

An individual unit test is just a method of a test class. To create a simple test, add its declaration to the ObjectPoolTest.h file:

TEST_CLASS(ObjectPoolTest)
{
    public:
        TEST_CLASS_INITIALIZE(setUp);
        TEST_CLASS_CLEANUP(tearDown);

        TEST_METHOD(testSimple);  // Your first test!
};

The implementation of this test uses Assert::IsTrue(), defined in the Microsoft::VisualStudio::CppUnitTestFramework namespace, to perform the actual test. In this case, the test claims that 0 is less than 1. Here is the updated ObjectPoolTest.cpp file:

#include "stdafx.h"
#include "ObjectPoolTest.h"

using namespace Microsoft::VisualStudio::CppUnitTestFramework;

void ObjectPoolTest::setUp() { }
void ObjectPoolTest::tearDown() { }

void ObjectPoolTest::testSimple()
{
    Assert::IsTrue(0 < 1);
}

That’s it. Of course, most of your unit tests will do something a bit more interesting than a simple assert. As you will see, the common pattern is to perform some sort of calculation, and then assert that the result is the value you expect. With the Visual C++ Testing Framework, you don’t even need to worry about exceptions; the framework catches and reports them as necessary.

Building and Running Tests

Build your solution by clicking Build ➪ Build Solution, and open the Test Explorer (Test ➪ Windows ➪ Test Explorer), shown in Figure 26-4.

After having built the solution, the Test Explorer automatically displays all discovered unit tests. In this case, it displays the testSimple unit test. You can run the tests by clicking the “Run All” link in the upper-left corner of the window. When you do that, the Test Explorer shows whether the unit tests succeed or fail. In this case, the single unit test succeeds, as shown in Figure 26-5.

If you modify the code to assert that 1 < 0, the test fails, and the Test Explorer reports the failure, as shown in Figure 26-6.

The lower part of the Test Explorer window displays useful information related to the selected unit test. In case of a failed unit test, it tells you exactly what failed. In this case, it says that an assertion failed. There is also a stack trace that was captured at the time the failure occurred. You can click the hyperlinks in that stack trace to jump directly to the offending line—very useful for debugging.

Negative Tests

You can write negative tests, tests that do something that should fail. For example, you can write a negative test to test that a certain method throws an expected exception. The Visual C++ Testing Framework provides the Assert::ExpectException() function to handle expected exceptions. For example, the following unit test uses ExpectException() to execute a lambda expression that throws an std::invalid_argument exception. The template type parameter for ExpectException() specifies the type of exception to expect.

void ObjectPoolTest::testException()
{
    Assert::ExpectException<std::invalid_argument>(
        []{throw std::invalid_argument("Error"); },
        L"Unknown exception caught.");
}

Adding the Real Tests

Now that the framework is all set up and a simple test is working, it’s time to turn your attention to the ObjectPool class and write some code that actually tests it. All of the following tests will be added to ObjectPoolTest.h and ObjectPoolTest.cpp, just like the earlier initial test.

First, copy the ObjectPool.h header file next to the ObjectPoolTest.h file you created, and then add it to the project.

Before you can write the tests, you’ll need a helper class to use with the ObjectPool class. The ObjectPool creates objects of a certain type and hands them out to the caller as requested. Some of the tests will need to check if a retrieved object is the same as a previously retrieved object. One way to do this is to create a pool of serial objects—objects that have a monotonically increasing serial number. The following code shows the Serial.h header file defining such a class:

#include <cstddef>  // For size_t

class Serial
{
    public:
        Serial();
        size_t getSerialNumber() const;
    private:
        static size_t sNextSerial;
        size_t mSerialNumber;
};

And here are the implementations in Serial.cpp:

#include "stdafx.h"
#include "Serial.h"

size_t Serial::sNextSerial = 0;  // The first serial number is 0.

Serial::Serial()
    : mSerialNumber(sNextSerial++) // A new object gets the next serial number.
{
}

size_t Serial::getSerialNumber() const
{
    return mSerialNumber;
}

Now, on to the tests! As an initial sanity check, you might want a test that creates an object pool. If any exceptions are thrown during creation, the Visual C++ Testing Framework will report an error:

void ObjectPoolTest::testCreation()
{
    ObjectPool<Serial> myPool;
}

Don’t forget to add a TEST_METHOD(testCreation); statement to the header file. This holds for all subsequent tests as well. You also need to add an include for "ObjectPool.h" and for "Serial.h" to the ObjectPoolTest.cpp source file.

A second test, testAcquire(), tests a specific piece of public functionality: the ability of the ObjectPool to give out an object. In this case, there is not much to assert. To prove validity of the resulting Serial reference, the test asserts that its serial number is greater than or equal to zero:

void ObjectPoolTest::testAcquire()
{
    ObjectPool<Serial> myPool;
    auto serial = myPool.acquireObject();
    Assert::IsTrue(serial->getSerialNumber() >= 0);
}

The next test is a bit more interesting. The ObjectPool should not give out the same Serial object twice (unless it is released back to the pool). This test checks the exclusivity property of the ObjectPool by retrieving a number of objects from the pool. The retrieved objects are stored in a vector to make sure they aren’t automatically released back to the pool at the end of each for loop iteration. If the pool is properly dishing out unique objects, none of their serial numbers should match. This implementation uses the vector and set containers from the Standard Library. Consult Chapter 17 for details if you are unfamiliar with these containers. The code is written according to the AAA principle: Arrange, Act, Assert; the test first sets up everything for the test to run, then does some work (retrieving a number of objects from the pool), and finally asserts the expected result (all serial numbers are different).

void ObjectPoolTest::testExclusivity()
{
    ObjectPool<Serial> myPool;
    const size_t numberOfObjectsToRetrieve = 10;
    std::vector<ObjectPool<Serial>::Object> retrievedSerials;
    std::set<size_t> seenSerialNumbers;

    for (size_t i = 0; i < numberOfObjectsToRetrieve; i++) {
        auto nextSerial = myPool.acquireObject();

        // Add the retrieved Serial to the vector to keep it 'alive',
        // and add the serial number to the set.
        retrievedSerials.push_back(nextSerial);
        seenSerialNumbers.insert(nextSerial->getSerialNumber());
    }

    // Assert that all retrieved serial numbers are different.
    Assert::AreEqual(numberOfObjectsToRetrieve, seenSerialNumbers.size());
}

The final test (for now) checks the release functionality. Once an object is released, the ObjectPool can give it out again. The pool shouldn’t create additional objects until it has recycled all released objects.

The test first retrieves ten Serial objects from the pool, stores them in a vector to keep them alive, records the serial number of the last retrieved Serial, and finally clears the vector to return all retrieved objects back to the pool.

The second phase of the test again retrieves ten objects from the pool and stores them in a vector to keep them alive. All these retrieved objects must have a serial number less than or equal to the last serial number retrieved during the first phase of the test. After retrieving ten objects, one additional object is retrieved. This object should have a new serial number.

Finally, two assertions assert that all ten objects were recycled, and that the eleventh object had a new serial number.

void ObjectPoolTest::testRelease()
{
    ObjectPool<Serial> myPool;
    const size_t numberOfObjectsToRetrieve = 10;

    std::vector<ObjectPool<Serial>::Object> retrievedSerials;
    for (size_t i = 0; i < numberOfObjectsToRetrieve; i++) {
        // Add the retrieved Serial to the vector to keep it 'alive'.
        retrievedSerials.push_back(myPool.acquireObject());
    }
    size_t lastSerialNumber = retrievedSerials.back()->getSerialNumber();
    // Release all objects back to the pool.
    retrievedSerials.clear();

    // The above loop has created ten Serial objects, with serial
    // numbers 0-9, and released all ten Serial objects back to the pool.

    // The next loop first again retrieves ten Serial objects. The serial
    // numbers of these should all be <= lastSerialNumber.
    // The Serial object acquired after that should have a new serial number.

    bool wronglyNewObjectCreated = false;
    for (size_t i = 0; i < numberOfObjectsToRetrieve; i++) {
        auto nextSerial = myPool.acquireObject();
        if (nextSerial->getSerialNumber() > lastSerialNumber) {
            wronglyNewObjectCreated = true;
            break;
        }
        retrievedSerials.push_back(nextSerial);
    }

    // Acquire one more Serial object, which should have a serial
    // number > lastSerialNumber.
    auto anotherSerial = myPool.acquireObject();
    bool newObjectCreated =
        (anotherSerial->getSerialNumber() > lastSerialNumber);

    Assert::IsFalse(wronglyNewObjectCreated);
    Assert::IsTrue(newObjectCreated);
}

If you add all these tests and run them, the Test Explorer should look like Figure 26-7. Of course, if one or more tests fail, you are presented with the quintessential issue in unit testing: is it the test or the code that is broken?

Debugging Tests

The Visual C++ Testing Framework makes it easy to debug unit tests that are failing. The Test Explorer shows a stack trace captured at the time a unit test failed, containing hyperlinks pointing directly to offending lines.

However, sometimes it is useful to run a unit test directly in the debugger so that you can inspect variables at run time, step through the code line by line, and so on. To do this, you put a breakpoint on some line of code in your unit test. Then, you right-click the unit test in the Test Explorer and click Debug Selected Tests. The testing framework starts running the selected tests in the debugger and breaks at your breakpoint. From then on, you can step through the code however you want.

Basking in the Glorious Light of Unit Test Results

The tests in the previous section should have given you a good idea of how to start writing professional-quality tests for real code. It’s just the tip of the iceberg, though. The previous examples should help you think of additional tests that you could write for the ObjectPool class. For example, you should include a test that acquires and releases objects multiple times, and tests whether such objects are properly recycled.

There is no end to the number of unit tests you could write for a given piece of code, and that’s the best thing about unit tests. If you find yourself wondering how your code might react to a certain situation, that’s a unit test. If a particular aspect of your subsystem seems to be presenting problems, increase unit test coverage of that particular area. Even if you simply want to put yourself in the client’s shoes to see what it’s like to work with your class, writing unit tests is a great way to get a different perspective.

Sample Integration Tests

Because there are no hard-and-fast rules to determine what integration tests you should write, some examples might help you get a sense of when integration tests are useful. The following scenarios depict cases where an integration test is appropriate, but they do not cover every possible case. Just as with unit tests, over time you will refine your intuition for useful integration tests.

Methods of Integration Testing

When it comes to actually writing integration tests, there is often a fine line between integration and unit tests. If a unit test is modified so that it touches another component, is it suddenly an integration test? In a way, the answer is moot because a good test is a good test, regardless of the type of test. I recommend that you use the concepts of integration and unit testing as two approaches to testing, but avoid getting caught up in labeling the category of every single test.

In terms of implementation, integration tests are often written by using a unit testing framework, further blurring their distinction. As it turns out, unit testing frameworks provide an easy way to write a yes/no test and produce useful results. Whether the test is looking at a single unit of functionality or the intersection of two components hardly makes a difference from the framework’s point of view.

However, for performance and organizational reasons, you may want to attempt to separate unit tests from integration tests. For example, your group may decide that everybody must run integration tests before checking in new code, but be a bit laxer on running unrelated unit tests. Separating the two types of tests also increases the value of results. If a test failure occurs within the JSON class tests, it will be clear that it’s a bug in that class, not in the interaction between that class and the file API.