Simply writing your C++ application and hoping it works as expected without any testing is guaranteed to result in reliability-, stability-, and security-related bugs. This recipe is important because testing your applications prior to release ensures that your applications execute as expected, ultimately saving you time and money in the future.
There are several different ways to test your code, including system-level, integration, long-term stability, and static and dynamic analysis, among others. In this recipe, we will focus on unit testing. Unit testing breaks an application up into functional units and tests each unit to ensure that it executes as expected. Typically, in practice, each function and object (that is, class) is a unit that should be tested independently.
There are several different theories as to how unit testing should be performed, with entire books written on the subject. Some believe that every line of code within a function or object should be tested, leveraging coverage tools to ensure compliance, while others believe that unit testing should be requirement-driven, using a black-box approach. A common development process called test-driven development states that all tests, including unit tests, should be written before any source code is written, whereas behavioral-driven development takes test-driven development a step further with a specific, story-driven approach to unit testing.
Every testing model has its pros and cons, and which method you choose will be based on the type of application you are writing, the type of software development process you adhere to, and any policies you may or may not be required to follow. Regardless of this choice, unit testing will likely be a part of your testing scheme, and this recipe will provide the foundation for how to unit test your C++ applications.
Although unit testing can be done with standard C++ (for example, this is how libc++ is unit tested), unit-test libraries help to simplify this process. In this recipe, we will leverage the Catch2 unit-test library, which can be found at
https://github.com/catchorg/Catch2.git.
Although we will be reviewing Catch2, the principles that are being discussed apply to most of the unit-test libraries that are available, or even standard C++, if you choose not to use a helper library. To leverage Catch2, simply execute the following:
> git clone https://github.com/catchorg/Catch2.git catch
> cd catch
> mkdir build
> cd build
> cmake ..
> make
> sudo make install
You can also use CMake's ExternalProject_Add, as we did in our examples on GitHub to leverage a local copy of the library.
To find out how to use Catch2, let's look at the following simple example:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
TEST_CASE("the answer")
{
CHECK(true);
}
When this is run, we see the following output:
In the preceding example, we start by defining CATCH_CONFIG_MAIN. This tells the Catch2 library that we want it to create the main() function for us. This must be defined before we include the Catch2 include statement, which we did in the preceding code.
The next step is to define a test case. Each unit is broken up into test cases that test the unit in question. The granularity of each test case is up to you: some choose to have a single test case for each unit being tested, while others, for example, choose to have a test case for each function being tested. The TEST_CASE() takes a string that allows you to provide a description of the test case, which is helpful when a test fails as Catch2 will output this string to help you identify where in your test code the failure occurred. The last step in our simple example is to use the CHECK() macro. This macro performs a specific test. Each TEST_CASE() will likely have several CHECK() macros designed to provide the unit with a specific input and then validate the resulting output.
Once compiled and executed, the unit-test library will provide some output text describing how to execute tests. In this case, the library states that all of the tests passed, which is the desired outcome.
To better understand how to leverage unit testing in your own code, let's look at the following, more complicated example:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <vector>
#include <iostream>
#include <algorithm>
TEST_CASE("sort a vector")
{
std::vector<int> v{4, 8, 15, 16, 23, 42};
REQUIRE(v.size() == 6);
SECTION("sort descending order") {
std::sort(v.begin(), v.end(), std::greater<int>());
CHECK(v.front() == 42);
CHECK(v.back() == 4);
}
SECTION("sort ascending order") {
std::sort(v.begin(), v.end(), std::less<int>());
CHECK(v.front() == 4);
CHECK(v.back() == 42);
}
}
Like the previous example, we include Catch2 with the CATCH_CONFIG_MAIN macro and then define a single test case with a description. In this example, we are testing the ability to sort a vector, so this is the description we provide. The first thing we do in our test is to create a vector of integers with a predefined list of integers.
The next thing we do is use the REQUIRE() macro to test, making sure that the vector has 6 elements in the vector. The REQUIRE() macro is similar to the CHECK() as both check to make sure that the statement inside the macro is true. The difference is that the CHECK() macro will report an error and then continue execution while the REQUIRE() macro will stop the execution, halting the unit test. This is useful to ensure that the unit test is properly constructed based on any assumptions that the test might be making. The use of REQUIRE() is important as unit tests mature over time, and other programmers add to and modify the unit tests, ensuring that bugs are not introduced into the unit tests over time, as there is nothing worse than having to test and debug your unit tests.
The SECTION() macro is used to further break up our tests with better descriptions and provide the ability to add common setup code for each test. In the preceding example, we are testing the sort() function for a vector. The sort() function can sort in different directions, which this unit test must validate. Without the SECTION() macro, if a test failed, it would be difficult to know whether the failure was from sorting in ascending or descending order. Furthermore, the SECTION() macro ensures that each test doesn't affect the results of other tests.
Finally, we use the CHECK() macro to ensure that the sort() function worked as expected. Unit tests should check for exceptions as well. In the following example, we will ensure that exceptions are thrown properly:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <vector>
#include <iostream>
#include <algorithm>
void foo(int val)
{
if (val != 42) {
throw std::invalid_argument("The answer is: 42");
}
}
TEST_CASE("the answer")
{
CHECK_NOTHROW(foo(42));
REQUIRE_NOTHROW(foo(42));
CHECK_THROWS(foo(0));
CHECK_THROWS_AS(foo(0), std::invalid_argument);
CHECK_THROWS_WITH(foo(0), "The answer is: 42");
REQUIRE_THROWS(foo(0));
REQUIRE_THROWS_AS(foo(0), std::invalid_argument);
REQUIRE_THROWS_WITH(foo(0), "The answer is: 42");
}
As with the previous example, we define the CATCH_CONFIG_MAIN macro, add the includes that we require, and define a single TEST_CASE(). We also define a foo() function that is thrown if the input to the foo() function is invalid.
In our test case, we first test the foo() function with a valid input. Since the foo() function doesn't have an output (that is, the function returns void), we check to ensure that the function has executed properly by ensuring that no exception has been thrown using the CHECK_NOTHROW() macro. It should be noted that, like the CHECK() macro, the CHECK_NOTHROW() macro has the equivalent REQUIRE_NOTHROW(), which will halt the execution if the check fails.
Finally, we ensure that the foo() function throws an exception when its input is invalid. There are several different ways to do this. The CHECK_THROWS() macro simply ensures that an exception has been thrown. The CHECK_THROWS_AS() macro ensures that not only has an exception been thrown, but that the exception is of the std::runtime_error type. Both must be true for the test to pass. Finally, the CHECK_THROWS_WITH() macro ensures that an exception has thrown and that the what() string returned what we expect from the exception matches. As with the other version of CHECK() macros, there are also REQUIRE() versions of each of these macros.
Although the Catch2 library provides macros that let you dive into the specific details of each exception type, it should be noted that the generic CHECK_THROWS() macro should be used unless the exception type and string are specifically defined in your API requirements—for example, the at() function is defined by the specification to always return an std::out_of_range exception when the index is invalid. In this case, the CHECK_THROWS_AS() macro should be used to ensure that the at() function matches the specification. The string that this exception returns is not specified as part of the specification, and therefore, the CHECK_THROWS_WITH() should be avoided. This is important, as a common mistake when writing unit tests is to write unit tests that are over-specified. Over-specified unit tests must often be updated when the code under test is updated, which is not only costly, but prone to error.
Unit tests should be detailed enough to ensure that the unit executes as expected but generic enough to ensure that modifications to the source code do not require updates to the unit tests themselves, unless the API's requirements change, resulting in a set of unit tests that age well while still providing the necessary tests for ensuring reliability, stability, security, and even compliance.
Once you have a set of unit tests to validate that each unit executes as expected, the next step is to ensure that the unit tests are executed whenever the code is modified. This can be done manually or it can be done automatically by a continuous integration (CI) server, such as TravisCI; however, when you decide to do this, ensure that the unit test returns the proper error code. In the previous examples, the unit test itself exited with EXIT_SUCCESS when the unit tests passed and printed a simple string stating that all of the tests passed. For most CIs, this is enough, but in some cases it might be useful to have Catch2 output the results in a format that can be easily parsed.
For example, consider the following code:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
TEST_CASE("the answer")
{
CHECK(true);
}
Let's run this with the following:
> ./recipe01_example01 -r xml
If we do this, then we get the following:
In the preceding example, we created a simple test case (the same as our first example in this recipe), and instructed Catch2 to output the results of the test to XML using the -r xml option. Catch2 has several different output formats, including XML and JSON.
In addition to output formats, Catch2 can also be used to benchmark our code. For example, consider the following code snippet:
#define CATCH_CONFIG_MAIN
#define CATCH_CONFIG_ENABLE_BENCHMARKING
#include <catch.hpp>
#include <vector>
#include <iostream>
TEST_CASE("the answer")
{
std::vector<int> v{4, 8, 15, 16, 23, 42};
BENCHMARK("sort vector") {
std::sort(v.begin(), v.end());
};
}
In the preceding example, we create a simple test case that sorts a vector with predefined vector numbers. We then sort this list inside a BENCHMARK() macro, which results in the following output when executed:
As shown in the preceding screenshot, Catch2 executed the function several times, taking on average 197 nanoseconds to sort the vector. The BENCHMARK() macro is useful to ensure that the code not only executes as expected with the proper outputs given specific inputs, but also that the code executes given a specific amount of time. Paired with a more detailed output format, such as XML or JSON, this type of information can be used to ensure that as the source code is modified, the resulting code executes in the same amount of time or faster.
To better understand how unit testing can truly improve your C++, we will conclude this recipe with two additional examples designed to provide more realistic scenarios.
In the first example, we will create a vector. Unlike an std::vector, which in C++ is a dynamic, C-style array, a vector in mathematics is a point in n-dimensional space (in our example, we limit this to 2D space), with a magnitude that is the distance between the point and the origin (that is, 0,0). We implement this vector in our example as follows:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <cmath>
#include <climits>
class vector
{
int m_x{};
int m_y{};
The first thing we do (besides the usual macros and includes) is to define a class with x and y coordinates:
public:
vector() = default;
vector(int x, int y) :
m_x{x},
m_y{y}
{ }
auto x() const
{ return m_x; }
auto y() const
{ return m_y; }
void translate(const vector &p)
{
m_x += p.m_x;
m_y += p.m_y;
}
auto magnitude()
{
auto a2 = m_x * m_x;
auto b2 = m_y * m_y;
return sqrt(a2 + b2);
}
};
Next, we add some helper functions and constructors. The default constructor makes a vector with no direction or magnitude as x and y are set to the origin. In order to create vectors that have a direction and magnitude, we also provide another constructor that allows you to provide the vector's initial x and y coordinates. To get the vector's direction, we provide getters that return the vector's x and y values. Finally, we provide two helper functions. The first helper function translates the vector, which in mathematics is another term for changing a vector's x and y coordinates given another vector. The final helper function returns the vector's magnitude, which is the length of the vector's hypotenuse if the vector's x and y values were used to construct a triangle (that is, we must use Pythagoras's theorem to calculate a vector's magnitude). Next, we move on to adding operators, which we do as follows:
bool operator== (const vector &p1, const vector &p2)
{ return p1.x() == p2.x() && p1.y() == p2.y(); }
bool operator!= (const vector &p1, const vector &p2)
{ return !(p1 == p2); }
constexpr const vector origin;
We add some equivalence operators, which can be used to check whether two vectors are equal. We also define a vector that represents the origin, which is a vector whose x and y values are 0.
To test this vector, we add the following tests:
TEST_CASE("default constructor")
{
vector p;
CHECK(p.x() == 0);
CHECK(p.y() == 0);
}
TEST_CASE("origin")
{
CHECK(vector{0, 0} == origin);
CHECK(vector{1, 1} != origin);
}
TEST_CASE("translate")
{
vector p{-4, -8};
p.translate({46, 50});
CHECK(p.x() == 42);
CHECK(p.y() == 42);
}
TEST_CASE("magnitude")
{
vector p(1, 1);
CHECK(Approx(p.magnitude()).epsilon(0.1) == 1.4);
}
TEST_CASE("magnitude overflow")
{
vector p(INT_MAX, INT_MAX);
CHECK(p.magnitude() == 65536);
}
The first test ensures that a default constructed vector is in fact the origin. Our next test ensures that our global origin vector is the origin. This is important because we should not assume that the origin is constructed by default—that is, it is possible for someone in the future to accidentally change the origin to something other than 0,0. This test case ensures that the origin is in fact 0,0, so that in the future, if someone accidentally changes this, this test will fail. Since the origin must result in x and y both being 0, this test is not over-specified.
Next, we test both the translate and magnitude functions. In the magnitude test case, we use the Approx() macro. This is needed because the magnitude that is returned is a floating point, whose size and precision depend on the hardware, and is irrelevant to our test. The Approx() macro allows us to state the level of precision to which we want to validate the result of the magnitude() function, which uses the epsilon() modifier to actually state the precision. In this case, we only wish to validate to one decimal point.
The last test case is used to demonstrate how all inputs to these functions should be tested. If a function takes an integer, then valid, invalid, and extreme inputs should all be tested. In this case, we are passing INT_MAX for both x and y. The resulting magnitude() function does not provide a valid result. This is because the process of calculating the magnitude overflows the integer type. This type of error should either be accounted for in the code (that is, you should check for possible overflows and throw an exception) or the API's specification should call out these types of issues (that is, the C++ specification would likely state that the result of this type of input is undefined). Either way, if a function takes an integer, then all possible integer values should be tested, and this process should be repeated for all input types.
The results of this test are as follows:
As shown in the preceding screenshot, the unit fails the last test. As stated previously, to fix this issue, the magnitude function should be changed to either throw when an overflow occurs, find a way to prevent the overflow, or remove the test and state that such input is undefined.
In our final example, we will demonstrate how to handle functions that do not return a value, but instead, manipulate an input.
Let's start this example by creating a class that writes to a file and another class that uses the first class to write a string to said file, as follows:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <string>
#include <fstream>
class file
{
std::fstream m_file{"test.txt", std::fstream::out};
public:
void write(const std::string &str)
{
m_file.write(str.c_str(), str.length());
}
};
class the_answer
{
public:
the_answer(file &f)
{
f.write("The answer is: 42 ");
}
};
As shown in the preceding code, the first class writes to a file called test.txt, while the second class takes the first class as an input and uses it to write a string to the file.
We test the second class as follows:
TEST_CASE("the answer")
{
file f;
the_answer{f};
}
The problem with the preceding test is that we do not have any CHECK() macros. This is because, other than CHECK_NOTHROW(), we have nothing to check. In this test, we are testing to make sure that the the_answer{} class calls file{} classes and the write() function properly. We could open the test.txt file and check to make sure that it was written with the right string, but this is a lot of work. This type of check would also be over-specifying as we are not testing the file{} class—we are only testing the the_answer{} class. If in the future we decide that the file{} class should write to a network file and not a file on disk, the unit test would have to change.
To overcome this issue, we can leverage a concept called mocking. A Mock class is a class that pretends to be the class that is inputted, providing the unit test with seams that allow the unit test to verify the result of a test. This is different from a Stub, which provides fake input. Sadly, C++ does not have good support for mocking when compared to other languages. Helper libraries, such as GoogleMock, attempt to fix this issue at the expense of requiring all of your mockable classes to contain a vTable (that is, inheriting pure virtual base classes) and define each mockable class twice (once in your code and a second time in your test, using a set of APIs defined by Google). This is far from optimal. Libraries such as Hippomocks attempt to address these issues at the expense of some vTable black magic that only works in certain environments and is nearly impossible to debug when things go wrong. Although Hippomocks is likely one of the best options (that is, until C++ enables native mocking), the following example is another method for mocking using standard C++, with its only downside being verbosity:
#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <string>
#include <fstream>
class file
{
std::fstream m_file{"test.txt", std::fstream::out};
public:
VIRTUAL ~file() = default;
VIRTUAL void write(const std::string &str)
{
m_file.write(str.c_str(), str.length());
}
};
class the_answer
{
public:
the_answer(file &f)
{
f.write("The answer is: 42 ");
}
};
As with our previous example, we create two classes. The first class writes to a file while the second class uses the first class to write a string to said file. The difference is that we added the VIRTUAL macro. When the code is compiled into our application, VIRTUAL is set to nothing, meaning that it is removed from the code by the compiler. When the code is compiled in our test, however, it is set to virtual, which tells the compiler to give the class a vTable. Since this is only done during our tests, the added overhead is fine.
Now that our class supports inheritance in our test case, we can create a subclassed version of our file{} class as follows:
class mock_file : public file
{
public:
void write(const std::string &str)
{
if (str == "The answer is: 42 ") {
passed = true;
}
else {
passed = false;
}
}
bool passed{};
};
The preceding class defines our mock. Instead of writing to a file, our mock checks to see whether a specific string is written to our fake file and sets a global variable to true or false, depending on the results of the test.
We can then test our the_answer{} class as follows:
TEST_CASE("the answer")
{
mock_file f;
REQUIRE(f.passed == false);
f.write("The answer is not: 43
");
REQUIRE(f.passed == false);
the_answer{f};
CHECK(f.passed);
}
When this is executed, we get the following:
As shown in the preceding screenshot, we can now check to make sure that our class writes to the file as expected. It should be noted that we use the REQUIRE() macro to ensure that the mock is in the false state prior to executing our test. This ensures that if our actual test registered as having passed, that it actually has passed, instead of registering as a pass because of a bug in our test logic.