At this point, you’ve probably gotten your application working and eliminated most of the big security holes in your code. Congratulations—that’s a huge step! Before you rip yourself away from the keyboard for a well-deserved break, there’s one last thing you need to take care of: testing your application.
We know—it’s not the most glamorous part of programming. Let’s be honest: Most programmers consider testing a royal pain in the neck and a task to be handed off to someone—anyone—else. That’s what this chapter is all about: delegating the hassle and monotony of running tests to the computer. Let’s face it; humans are great programmers, but we’re pretty lousy at doing repetitive, monotonous tasks over and over exactly the same way. We’re wired to be creative, not to be mindlessly repetitive. That’s why we invented computers in the first place!
Most of us can agree that testing is a good idea in theory. But why are we devoting a whole chapter to it in a book on securing application code? There are two major benefits to automated testing, from a security standpoint:
• Alerts if the application suddenly starts misbehaving
• The capability to constantly test boundary conditions
During a security incident, time is crucial. The quicker you catch an attack, the less harm it can do. Let’s just say that an attack on your application is launched at 1:52 on a Saturday morning. You’ve set up your automated testing framework to run every hour, and it kicks off promptly at 2 a.m. At 2:03 a.m. your pager goes off, alerting you that some crucial tests have failed—something is definitely wrong. You log in to the server and realize that you’re under a denial-of-service attack. You shut down the port that is under attack and go back to bed by 2:30 a.m. The night’s excitement was a hassle, but nothing your users will ever know about.
Let’s look at the alternative scenario, where you’ve manually tested your application, made sure it was working, and opened it up to users. The same denial-of-service attack begins at 1:52 Saturday morning, and instead of being woken up by your pager 11 minutes later, you blissfully sleep through the night, until your phone starts ringing at 6 a.m. on Sunday. You answer, still half-asleep, and spend the next half hour talking one of your users down from a full-blown panic attack. You stumble into the kitchen for a cup of coffee and the phone rings again. And again. You don’t actually get to drink that cup of coffee before it gets cold because your users are calling you constantly, asking when the application will be back online, what happened, and why. You finally take the phone off the hook and quit answering e-mails for an hour while you figure out the source of the attack and shut down the port, reboot the server, and bring the application back online around 9:30 Sunday morning. Then you send out an e-mail to all your users letting them know that everything is back online and working properly, and you reluctantly put the phone back on the hook.
All told, you’ve just spent three and a half hours dealing with panicked users and the aftermath of a full-blown attack. With a solid automated testing framework, you could have spent half an hour shutting down an attack in progress and gone back to sleep as if nothing had happened. Because you could catch the attack in progress, your users wouldn’t be affected and would never know anything happened.
Just having a testing framework isn’t enough, of course. You have to write tests that exercise both typical situations and boundary conditions. Those boundary conditions are what hackers attack, so it just makes sense to pay special attention to the most unreasonable, illogical data that can be thrown at your application. We’ll talk about choosing test data later in this chapter.
Automated testing requires two parts: the tests themselves and the testing framework. We’ll discuss the actual tests in a moment. In order to place the tests in context, let’s focus on the framework first.
Any testing framework has one specific purpose: to run tests. It impersonates a live tester, who probably has better things to do than manually launch test scripts 15 times a day. In this sense, a series of cron jobs (for those in the *nix universe) or scheduled tasks (for the Windows side of existence) qualifies as a testing framework.
Most true frameworks also include functions designed to analyze and report the results of each test. For example, SimpleTest (www.lastcraft.com/simple_test.php), the framework we used to test the guestbook application, includes 17 unit test functions that you can use to determine whether your tests succeed or fail:
• assertTrue($x)
• assertFalse($x)
• assertNull($x)
• assertNotNull($x)
• assertIsA($x, $type)
• assertNotA($x, $type)
• assertEqual($x, $y)
• assertNotEqual($x, $y)
• assertWithinMargin($x, $y, $margin)
• assertOutsideMargin($x, $y, $margin)
• assertIdentical($x, $y)
• assertNotIdentical($x, $y)
• assertReference($x, $y)
• assertClone($x, $y)
• assertPattern($pattern, $x)
• expectError($x)
• assert($expectation)
To use these functions, you would set up a situation, then assert that the situation you created produced the results you expect. In this example, we create a user object, then test whether the object we created is actually of the type "User":
$user = new User('autotester77982'),
$this->assertIsA($user, "User");
In these two lines of code, we created a user object using the constructor in the User class, then we used the built-in assertIsA() function from the testing framework to prove that the variable $user actually refers to an object of type "User".
There are two basic types of tests:
• Unit tests
• System tests
Unit tests are the easiest type of tests to write, because they test a single part of the program in isolation from the rest. Most unit tests focus on a single function. The example in the previous section was a partial unit test. Ideally, you should include at least one test for each function in your class. For example, here is the constructor from our User class:
function User($username, $email = NULL, $isAdmin = "N") {
$this->_username = $username;
$this->_email = $email;
$this->_isAdmin = $isAdmin;
return $this;
}
Three basic functions are happening here: We set the _username, _email, and _isAdmin private variables in the object. We will have four basic unit tests for this constructor:
• Assert that the constructor actually created an object.
• Assert that the _username private variable was set correctly.
• Assert that the _email private variable remains NULL (since we didn’t supply a value for it).
• Assert that the _isAdmin private variable holds the default value of "N".
Here is the actual unit test for this constructor:
// test constructor, basic scenario
$user = new User('autotester77982'),
$this->assertIsA($user, "User", "did not create user object");
$this->assertEqual($user->_username, 'autotester77982', "did not set username");
$this->assertEqual($user->_isAdmin, "N", "did not set isAdmin to N");
$this->assertNull($user->_email, "email is not null");
You’ll notice that we put a third argument into the assertion functions. This is a note to the developer that will be included in the test report if the assertion fails to help diagnose the problem.
It may seem as if these tests are absurdly simplistic, like dropping a fork on the floor 15 times every day just to be sure the law of gravity is still working, but that’s what unit tests are all about—proving the basic assumptions we make about the universe of our program.
Unit tests prove that the basic building blocks of an application work as expected. They are concerned purely with the inner workings of an application, without regard to the user experience. System tests prove that all those building blocks work together to create a cohesive whole and often approach the application from the front end.
A good system test for the guestbook application will replicate the user experience. It begins with replicating a browser, which then loads the Web page containing the application, then tests various assumptions about the Web page content, such as the page title (to ensure that the correct page was loaded). It will also test the actual content of the page against a regular expression pattern to be sure that the page “looks” the way you expect it to.
System tests are crucial for ensuring that your users experience the application correctly. They are also important from a security perspective because Web site defacement is probably the single most common type of Web site hack. System tests will alert you to a defacement as soon as it’s carried out—ideally before your users ever see it.
So far we’ve only discussed tests that prove the application is working properly. It’s also important to beat on your application to ensure that it can handle the oddest situations users throw at it. Using the code from the User class constructor from the beginning of this chapter, we’ve already tested the constructor using typical data—a valid username. We also need to test the boundary conditions and other special cases.
Boundary conditions are the most extreme cases you can think of. When we test the constructor, which takes a username as data, the following may be useful boundary conditions:
• NULL data
• Length exceeding the size of the variable
• Data including ASCII control characters
• Data including special characters, such as & and *
• Data that replicates an injection attack, such as ;drop table users;
• Any other extreme data you can think of
These are tests that you expect to fail—in fact, if they don’t fail, you know that you need to go back and harden your code some more.
This has been a really brief introduction to the concepts of automated testing and testing frameworks. We’ve covered the two basic types of tests and discussed how to choose good test data. We highly recommend that you investigate automated testing further and experiment with automated tests of your own applications. We’ve listed several good references and tutorials on automated testing in the Appendix, “Additional Resources.”