Chapter 8. Testing Your Application

As you near the completion of your first application, it's time to turn your attention to what's often called "the other 80 percent"—that is, the work that remains after you've done the first 80 percent. It doesn't need to be overwhelming, but the truth is that there's more to delivering a quality application than just the design and the code. As you wrap up application development, you should be thinking about testing and integration with other systems and even simple things like establishing your application's brand message through the artwork and copy you'll submit to the Ovi Store (which we discuss in the next chapter).

In this chapter, we examine application testing, giving you some tips and tricks as to ways you can best use Nokia tools to support your testing. These include QTest, Qt's test framework, which as you'll see, you can use throughout your development cycle to help you reach your quality goals. We close the chapter with an example using QTest to show you how easy it is to verify application quality as you go through the use of unit tests.

Preparing to Test

Although there has been a lot of discussion in recent years about the nature of testing that's appropriate in software development, there's no question some of it is absolutely necessary. When you plan your testing, you should be sure to think about the kinds your application will require and how much time and effort they will take.

You should begin with a test plan, a concise list of test cases that you automatically or manually execute at regular intervals (say, daily or weekly). Each test case should describe a single test, including the initial configuration, the steps to perform the test, and the expected results. A good test plan, coupled with investigative work, can give you an idea of the test coverage—that is, the percentage of the application's functionality that is testable and how much of that testing you can automate.

Your testing may be functional, as well as non-functional. Functional testing includes everything to do with the actual operation of your application—things like its business logic to ensure that it operates correctly. Non-functional testing includes all areas of your application that have to do with its performance outside of correct operation. That's a lot: out-of-memory conditions, signal loss, position/GPS loss, servers not being available due to downtime or unplanned load, and so on. Non-functional testing is also often thought of as adversarial testing. It explores how your application performs in adverse situations. When planning your tests, be sure to include at least as much adversarial testing as functional testing, because it's impossible to really know just what situations your application will encounter. The execution environment, after all, is mobile, will both move in and out of signal coverage, and experience periods of heavy and light use, leading to periods of heavy heap and persistent storage use.

What most people think of testing is dynamic—that is, testing that occurs while the application is running. Dynamic tests include:

  • Manual testing, in which you or a tester puts your application through its paces with the guidance of a test plan.

  • Unit tests, which exercise a single class programmatically (more on doing this with Qt later in this chapter in the section "Using Unit Tests to Verify Functionality").

  • Integration tests, which examine your application as it works with other portions of your application's system, such as back-end servers.

  • Analysis tools like valgrind (http://valgrind.org) that explore run-time performance with regard to memory and other resource usage (not available for Symbian, however).

Equally—and perhaps more—important is static testing, which includes any nonexecution examination of your source code. Buddy checks or code reviews are a great way to catch common programming errors and cross-train participants on your developer team, as well as think through tricky problems together. Tools abound for verifying your application as part of the compile cycle; an easy one is the compiler's own warnings.

Note

Unfortunately, as we write this, the Symbian build environment is apt to emit warnings of its own when you compile, making a zero-warnings policy difficult. One strategy is to use compiler pragmas to turn off certain warnings, but we find it easier to periodically (say, daily) skim the warnings from an otherwise working build and make sure nothing has crept in. If you've got a lot of files in your project, once the number of files is stable, you can capture the output and use a tool like diff to verify that the warnings remain the same. With luck and effort on the part of Nokia, this problem will improve soon.

Using Qt's Test Framework

Inspired by the various unit testing frameworks evangelized by the extreme programming community, Qt provides a framework for implementing unit tests for classes using Qt. This framework, called QTest, is a small, self-contained library that invokes tests provided by a class that you write and lets you test components in your application. With a bit of creativity, you can extend your tests within QTest to test not just single classes as recommended by the unit test paradigm, but also test classes in combination or perform simple integration tests. Implementing both the test runner and the basic primitives for result verification and benchmarking, QTest is a lightweight, self-contained thread- and type-safe library you can use to quickly create tests for your application.

Introducing the QTest Test Framework

QTest's test runner uses Qt's meta-object protocol to introspect the methods in a class you provide to determine what must happen at run time. This makes test definition very easy for you. You only need to provide a class derived from QObject that defines slots implementing the tests that you want to run. QTest treats several slots in your test class in special ways to determine what slots correspond to tests and give you a way to control the test environment setup and teardown:

  • The test harness invokes the slot initTestCase before any tests are run.

  • The test harness invokes the slot cleanupTestCase after all tests are run.

  • The test harness invokes the slot init before each test is run.

  • The test harness invokes the slot cleanup after each test is run.

  • The test harness invokes each slot in turn, invoking first the slot init, and then the first slot it encounters, and then the slot cleanup, and then init, the next slot, and then cleanup, and so on, until all slots are run.

By design, your individual test cases should be independent; a test case should not rely upon the performance of a previous or subsequent test case. This is a key reason for providing the initTestCase and init methods, where you can separate key initialization for all test cases, and clean up after a single or all test cases using cleanup and cleanupTestCase.

Warning

Don't confuse init with initTestCase (or cleanup with cleanupTestCase). It's easy to get confused, because you tend to think of each test in your class as an individual test case—so there's the temptation to write initTestCase for per-test initialization. It's exactly the reverse, however.

Listing 8-1 shows perhaps the most trivial of tests.

Example 8.1. A trivial test class

#include <QtTest/QtTest>
class TrivialTest: public QObject
{
    Q_OBJECT
private slots:
    void () trivialTest
    { QVERIFY(1 == 1); }
    void anotherTrivialTest()
    { QVERIFY(0 != 1); }
};
QTEST_MAIN(TrivialTest)
#include "trivialtest.moc"

Before we look at the test slots, let's pause briefly and look at the additional stuff after the test. The compiler will expand the QTEST_MAIN macro to provide an entry point function and invoke your test methods. The second line with the #include includes the output from the meta-object compiler, required anytime you define a QObject derivative in a C++ class instead of a header. Not shown in Listing 8-1 is the .pro file; in addition to including the source file for TrivialTest.cpp, it needs to include the QTest configuration. If you've got the qmake executable in your path, an easy way to make a .pro file for a QTest test class is to use qmake on the command line, like this:

C:BookTests>qmake -project "CONFIG += QTest"
C:BookTests>qmake
C:BookTests>make

In project mode, qmake will make a .pro file for you that includes the source files in the current directory, as well as the CONFIG variable, including the QTest libraries and headers.

If you don't, you can create one using Qt creator by performing the following in the Nokia Qt SDK:

  1. Choose "File>New File or Project"...

  2. Choose "Other Project" from the upper left-hand pane of the window that appears.

  3. Choose "Qt Unit Test" from the list in the upper right-hand pane.

  4. Click "Choose..."

  5. Enter a name and path for the unit test and click "Next."

  6. Choose at least the Simulator Qt options for your build system, and optionally choose device targets as well and click "Next."

  7. Choose any modules upon which your unit tests will depend, such as QtNetwork for networking and click "Next."

  8. Fill out the form describing the first unit test in your test cases and click "Next."

  9. Click "Finish."

Returning to the body of the test functions, the QVERIFY macro is one of several provided by QTest to facilitate instrumentation for test passes and failures. These are:

  • QVERIFY verifies that a condition is true and causes the test to fail if it's not.

  • QVERIFY2 operates the same as QVERIFY, but includes a verbose message to be output if the condition fails.

  • QCOMPARE performs a type-safe comparison of an actual value to an expected value. When comparing floating-point (single or double precision), it uses the Qt function qFuzzyCompare to better support approximate comparisons using floating-point representations.

  • QSKIP stops execution of the current test without adding a failure to the test log. It does, however, indicate that the test was skipped for the reason you provide when invoking the macro.

  • QBENCHMARK benchmarks the code block that immediately follows the macro, running it multiple times if necessary to obtain benchmark data. You can require the benchmarked code only be run once by using QBENCHMARK_ONCE, although the elapsed time may be reported as zero, if the execution time is too short to be measured by the benchmark system.

Unit Testing the QuakeEvent Class

Let's take a look at an actual unit test, one we wrote for the QuakeEvent class. Listing 8-2 shows the unit test itself.

Example 8.2. The unit test for the QuakeEvent class

#include <QtCore/QString>
#include <QtTest/QtTest>
#include <QDebug>
#include "quakeevent.h"

class TestQuakeEvent : public QObject
{
    Q_OBJECT

public:
    TestQuakeEvent();

private:
    QuakeEvent *mEvent;

private Q_SLOTS:
    void initTestCase();
    void cleanupTestCase();

    void init();
    void cleanup();

    void testConstructor();
    void testSetGet();
    void testIsEmpty();
    void testClear();
    void testComparator();

    void testId();
    void testSummary();
    void testWhen();
    void testWhere();
    void testMagnitude();
    void testPosition();
    void testElevation();
    void testHtml();
    void testDistanceTo();
};
TestQuakeEvent::TestQuakeEvent()
{
}

void TestQuakeEvent::initTestCase() {
    mEvent = new QuakeEvent();
}

void TestQuakeEvent::cleanupTestCase() {
    delete mEvent;
}

void TestQuakeEvent::init() {
    mEvent->clear();
    mEvent->set("title",   "M 2.6, Hawaii region, Hawaii");
    mEvent->set("point",   "19.9770 −156.8687");
    mEvent->set("elev",    "−7900");
    mEvent->set("summary", "<img src="http://earthquake.usgs.gov
/images/globes/20_-155.jpg" alt="19.977&#176;N 156.869&#176;W"
 align="left" hspace="20" /><p>Monday, September  6, 2010 15:
19:09 UTC<br>Monday, September  6, 2010 05:19:09 AM at epicenter<
/p><p><strong>Depth</strong>: 7.90 km (4.91 mi)</p>");
}

void TestQuakeEvent::cleanup() {
    mEvent->clear();
}

void TestQuakeEvent::testConstructor() {
    QuakeEvent *e = new QuakeEvent();
    QVERIFY(e->isEmpty());
    delete e;
}

void TestQuakeEvent::testSetGet() {
    mEvent->set("arbitrary", "value");
    QVERIFY(mEvent->get("arbitrary")=="value");
}

// Failures may indicate a problem with either
// isEmpty or clear
void TestQuakeEvent::testIsEmpty() {
    QVERIFY(!mEvent->isEmpty());
    mEvent->clear();
    QVERIFY(mEvent->isEmpty());
}

// Failures may indicate a problem with either
// isEmpty or clear
void TestQuakeEvent::testClear() {
    QVERIFY(!mEvent->isEmpty());
    mEvent->clear();
    QVERIFY(mEvent->isEmpty());
}
void TestQuakeEvent::testComparator() {
    QuakeEvent *e = new QuakeEvent();
    e->set("summary",      "<img src="http://earthquake.usgs.gov
/images/globes/20_-155.jpg" alt="19.977&#176;N 156.869&#176;W"
 align="left" hspace="20" /><p>Monday, September  6, 2010 15:
19:09 UTC<br>Monday, September  6, 2010 05:19:09 AM at epicenter<
/p><p><strong>Depth</strong>: 7.90 km (4.91 mi)</p>");
    QVERIFY(*mEvent < *e);
    delete e;
}

void TestQuakeEvent::testId() {
    mEvent->set("arbitrary", "123456789");
    QVERIFY(mEvent->get("arbitrary")=="123456789");
}

void TestQuakeEvent::testSummary() {
    QVERIFY(mEvent->summary() == "M 2.6, Hawaii region, Hawaii");
}

void TestQuakeEvent::testWhen() {
    // Ideally this would test a number of dates and times
    QDateTime when(QDate(2010, 9, 6),
                   QTime( 15, 19, 9), Qt::UTC);

    QVERIFY(mEvent->when() == when);
}

void TestQuakeEvent::testWhere() {
    QVERIFY(mEvent->where() == "Hawaii region, Hawaii");
}

void TestQuakeEvent::testMagnitude() {
    float mag = (float)mEvent->magnitude();

    QCOMPARE(mag, (float)2.60);
}

void TestQuakeEvent::testPosition() {
    qDebug() << mEvent->position();
    QCOMPARE((float)mEvent->position().first,  (float)19.977);
    QCOMPARE((float)mEvent->position().second, (float)-156.869);
}

void TestQuakeEvent::testElevation() {
    QVERIFY(qFuzzyCompare(mEvent->elevation(), −7900.0));
}

void TestQuakeEvent::testHtml() {
    QVERIFY(mEvent->html() ==
                           "<img src="http://earthquake.usgs.gov
/images/globes/20_-155.jpg" alt="19.977&#176;N 156.869&#176;W"
 align="left" hspace="20" /><p>Monday, September  6, 2010 15:
19:09 UTC<br>Monday, September  6, 2010 05:19:09 AM at epicenter<
/p><p><strong>Depth</strong>: 7.90 km (4.91 mi)</p>");
}
void TestQuakeEvent::testDistanceTo() {
    qreal distance = mEvent->distanceTo(
        QPair<qreal, qreal>(37.0, −122.0));
    QCOMPARE((float)distance, (float)3870.68);
}

QTEST_APPLESS_MAIN(TestQuakeEvent);
#include "tst_quakeevent.moc"

These tests show a common pattern in unit tests, in which there's at least one test per method of the object under test. Our tests are self-explanatory, so rather than walking line-by-line, we'll just call out a few of the high points.

First, with a few exceptions, the test reuses a single QuakeEvent object, rather than creating one for each case. This increases performance (fewer memory allocations), but requires that the basic object recycling interface that the QuakeEvent::clear method promises be working correctly. As each test case starts, the init method initializes the unit test's mEvent field with known content hand-scraped from the USGS web site's feed. When the test is completed, the cleanup method clears the mEvent field, ensuring that each test enters and concludes with the recycled object in a known state.

Second, tests that require their own object—either one that hasn't been initialized, like the testConstructor test, or one that requires more than one QuakeEvent object, such as testComparator—simply create a second object and initialize them as the logic behind the test case requires.

Third, the test for QuakeEvent::when is probably a little underpowered; you remember from Chapter 5 that it scrapes the date from a text string and requires exact matching of month names; a good test case would probably set several different date strings on the QuakeEvent and check that each date gets parsed correctly. (Even better would be one that includes invalid dates and protects against crashes or bizarre failures.) However, the code we wrote was reviewed, which provides some confidence, and such an example could become tedious for you to read very quickly.

Finally, unit tests that compare floating-point numbers are notoriously finicky to get exactly right. Thanks to the vagaries of floating-point arithmetic, you should always declare the precision you desire and use either qFuzzyCompare or QCOMPARE (which uses qFuzzyCompare under the hood). If you don't, you can get all kinds of odd results, especially when computing with double-precision floating-point numbers or when matching single-precision and double-precision arithmetic. Here, given the nature of the input and results, single-precision arithmetic is ample, so it's all we do.

Testing Signals and Slots Using QTest

Many times when you're writing a test for a class, you realize that what you want to test is a signal's emission, not just the results of a function. For example, if you write a new model, you want to ensure that your model correctly emits the QAbstractItemModel's signals like dataChanged as the contents of the model changes.

Fortunately, Qt has a class that enables you to examine the results of any signal emission, QSignalSpy. Implemented as a list of QVariant lists, each signal it catches appends a list of signal arguments to its main list, letting you eavesdrop on the signal process itself. Listing 8-3 shows a typical use.

Example 8.3. Using QSignalSpy to test signal emission

QSignalSpy spy(myObject, SIGNAL(something(QString, int)));
// trigger signal emission
myObject ->emitsSomething();
// Check the resulting types
QList<QVariant> arguments = spy.takeFirst();
QVERIFY(arguments.at(0).type() == QVariant::QString);
QVERIFY(arguments.at(1).type() == QVariant::Int);
// Check the resulting values
QVERIFY(arguments.at(0).value<int>() == 1);
QVERIFY(arguments.at(1).value<QString>() == "hello");

The code begins by connecting a stack-stored QSignalSpy instance to a custom QObject that emits the something signal, passing a QString and an int. It next calls the hypothetical method emitsSomething, which presumably emits the something signal. The arguments to this signal are stored in the QSignalSpy as its first element; each of the signal's arguments is stored as a separate QVariant instance (which we first mentioned in Chapter 5) in a QList of arguments.

The code performs the tests themselves on the arguments to the signal, first verifying the type of each signal argument, and then verifying the value. The QVariant's type method returns the C++ type of a QVariant as a Qt-enumerated value, while its templated value function returns the value, optionally coerced to the specific type you pass as a template argument.

Testing User Interface Code Using QTestEventList

If you're interested in performing automated testing on user-interface classes, little we've shown you so far provides much help. As your user interface should primarily connect to your business logic through signals and slots, writing unit tests that use QSignalSpy lets you create an instance of pieces of your user interface (say, a custom widget or a panel of related widgets) and test that they emit appropriate signals.

However, to do this, when you test a user interface class, you want to simulate user events, such as key or mouse events. This is especially true if you're creating your own widget or if you want to script an interaction with a component such as a view in your application. QTest also includes the QTestEventList class, a class that lets you create a list of events and then pass them one at a time to a child object of QWidget. At its heart, it's simply a QList of test event objects, which get invoked on the widget one at a time when you invoke its simulate method. QTestEventList provides methods so that you can add following events to the list:

  • A key click or clicks (by Qt::Key code, ASCII character, or QString) by calling addKeyClick or addKeyClicks (for multiple key clicks).

  • A key press by calling addKeyPress (passing either the Qt::Key code or an ASCII character).

  • A key release by calling addKeyRelease (passing either the Qt::Key code or an ASCII character)

  • A mouse click by calling addMouseClick and passing which mouse button (Qt::MouseButton), any keyboard modifiers (Qt::KeyboardModifiers), and the point where the click should be simulated (QPoint).

  • A mouse double-click by calling addMouseDClick and passing which mouse button (Qt::MouseButton), any keyboard modifiers (Qt::KeyboardModifiers), and the point where the click should be simulated (QPoint).

  • A mouse button press or release event by calling addMousePress or addMouseRelease and passing which mouse button (Qt::MouseButton), any keyboard modifiers (Qt::KeyboardModifiers), and the point where the click should be simulated (QPoint).

  • A mouse movement by calling addMouseMove, passing the point to which the mouse cursor should move as a QPoint.

Each of these can optionally include a delay in milliseconds, or you can call addDelay to add a delay to the simulated event stream.

Listing 8-4 shows how you might append the text "Hello world" with some extraneous mouse movements to a QLineEdit in your unit test:

Example 8.4. Simulating events to a QLineEdit using QTestEventList

QTestEventList events;
QLineEdit *lineEdit = new QLineEdit(this);
events.addKeyClicks("Hello world", 100);
events.addMouseMove(QPoint(qrand() % 256, qrand % 256), 25);
events.addMouseMove(QPoint(qrand() % 256, qrand % 256), 25);
events.addMouseMove(QPoint(qrand() % 256, qrand % 256), 25);
// simulate all the events
events.simulate(lineEdit);

The code creates first the events for the discrete key click events to type "Hello world," and then pauses for 100 milliseconds. Next, it creates three random mouse movement events to random coordinates bounded by the rectangle (left: 0, top: 0, right: 256, bottom: 256), pausing for 25 milliseconds between each move. Finally, it simulates these events on the line editor instance lineEdit.

If you need only simulate an event or two, it may be simpler to use the static QTest methods to do so. There's one corresponding to each of the kinds of events you can simulate using the QTestEventList class, namely:

  • keyClick and keyClicks to simulate key clicks.

  • keyEvent to simulate a specific key event.

  • keyPress and keyRelease to simulate a single key press or release.

  • mouseClick and mouseDClick to simulate a single- or double-click.

  • mousePress and mouseRelease to simulate a single mouse button press or release.

  • mouseMove to simulate a single mouse movement event.

Each of these takes the widget to which the event should be sent.

Finally, the QTest class has a few other static methods that can come in handy. Under the qSleep method, the process sleeps for the number of milliseconds you specify, blocking test execution and leaving events unprocessed (in other words, your test is non-responsive during this time). The qWait method waits for the number of milliseconds you specify, letting the test application still process events and handle network communication. The toHexRepresentation takes an array of bytes and returns a character string of space-separated hex characters to facilitate hex dumps, and the toString method returns a human-readable string representation of times, dates, byte arrays, points, sizes, rectangles, and other fundamental types that Qt defines.

Wrapping Up

We opened this chapter with the types of testing (dynamic vs. static, unit vs. integration) that you can perform to help flush out bugs in your application. Most important, we urged you to follow the best practices of test-driven development, including daily builds, buddy code reviews, treating compiler warnings as errors, and frequent execution on real devices.

Next, we gave you an in-depth look at Qt's test framework QTest, which provides a small self-contained library with a test harness and utilities ideal for unit testing and small integration or system-level tests. Using QTest, you can create small executables that run tests you write on a class (or a collection of classes). Defining the tests as private slots of a QObject subclass, you instrument your tests using helper macros such as QVERIFY and QCOMPARE to test for successful situations or test expected vs. actual results. Using the QSignalSpy class, you can also test your classes' emission of signals, ensuring that they emit the correct arguments and values for the situations you create for the object under test. You can even simulate sequences of common events like key strokes and mouse events, either as a collection using QTestEventList or singly with the static methods provided by the QTest class.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset