Chapter 4. Automated testing
- Increasing confidence
- Evaluating existing tests
- Integrating existing tests
- Automating test execution
We’re now three chapters into a book on software development and we have yet to write any code. We have a good reason: many brownfield applications become contaminated because the proper foundation wasn’t laid out before the team, past or present, started coding. On other projects, it was laid out, but subsequently ignored in the face of growing deadlines and spiraling requirements.
This chapter continues that theme. Now that we’ve gone through the first chapters and the source control, build, deployments, and releases have been automated, it’s time to look at our testing strategies. Perhaps the project already has some degree of unit testing. Maybe the project has no unit testing at all; you appreciate its relevance but haven’t implemented any due to a lack of time. Finally, it’s possible that the project has no unit testing and there’s strong opposition to its use on the project.
Whatever the case, over the course of this chapter we’ll continue laying groundwork for the eventual code changes in later chapters. We’ll start with a discussion on why we write tests before delving into the various types. By the end, we hope to convince you not only that you can’t live without tests but that implementing them isn’t nearly as cumbersome as you first thought. The first step is understanding what issues you may be seeing in your projects. For that, let’s look at some of the common pain points that relate to automated testing.
4.1. Pain points
It isn’t hard to come up with examples of pain caused by a lack of tests. Take any large application, pick an outstanding bug, and ask a developer to fix it. In such cases the pain will often manifest itself physically on the developer’s face.
Any time you write a piece of branching logic into your application, which is common in the form of if/else and switch statements (or select in Visual Basic), how is each branch verified to work as expected? Do you manually configure the application to run each branch independently, taking the time to personally interact with it while this happens? If so, how long does that verification process take you? After you have verified that the logic is working as expected, how do you perform regression tests on those scenarios 10 minutes, 10 hours, 10 days, or 10 months from now?
Figure 4.1 shows how branching logic can quickly get out of hand.
Figure 4.1. An example of branching logic within your code. Each path through the code represents a potential point of failure and should be tested.
Let’s consider another example. After writing a number of small, interrelated components, you’ll eventually need to verify that they work together. Now is when you find out if the address retrieval component works with the email construction component. Again we (and you should) ask, “How do I test this? Is the process time consuming? Does it depend on the consistency of human interaction? How will it be repeated the next time that regressions need to be run?”
These two pain points, verifying the correctness of your code and ensuring components work together, apply when developing new features. What about once the application has moved into maintenance mode and has bugs?
Note
Maintenance of the application doesn’t begin once it has been released into production. From a developer’s perspective, each line of code you write enters maintenance mode immediately after it’s written.
The act of fixing bugs, at any point in the application’s life cycle, can be a pain point in and of itself. To repair the bug, first you need to be able to reproduce it. Typically, this means launching the application and navigating to the screen where the bug occurs. Often, preconditions need to be met, such as appropriate data in the database.
Then, as you fix the bug you must continuously launch the application, navigate to the screen again, and test the latest code you’ve done. Depending on the complexity of the bug, you could spend most of your time in the application rather than in the code. Even when the bug is fixed for the specific scenario you tested, an exhaustive test to make sure it hasn’t introduced other bugs is usually deferred, or handed off to a QA/testing team.
You may have noticed that we’re dancing around the solution for these pain points: automated tests. We’ll remedy that shortly, but first let’s move on to the underlying motive behind automated tests.
4.2. Living in fear of change
Confidence. It is one of, if not the primary reason for tests of any kind. Without tests, all we have to go on to prove that our application does what it’s supposed to is the gut feelings of the development team. A large codebase without tests is akin to a trapeze artist with no net. Most of the time, things hum along smoothly, but when something goes wrong, such a spectacular failure it is.
Once an application reaches any substantial size, the lack of tests starts to have an impact on your attitude toward it. It starts with a slight hesitation when the client requests a new feature. Then words like spaghetti and minefield start to enter the project’s lexicon.
As the brittleness of the codebase increases, you begin to fear change. Designers, business analysts, testers, and management join with the developers in their hesitation to modify areas of the application. Eventually it will get to the point where users find workarounds for major bugs because they either fear the ramifications of reporting them or have just plain lost confidence in the development team. Even small codebases suffer from a lack of confidence. Let’s look at a hypothetical case.
Let’s assume we have an online restaurant locator service, and it’s expanding outside the United States into the Caribbean, specifically the Bahamas. In order to ease the transition, they’ve purchased the data for all the local restaurants in the country and imported it into their system.
One of the first things you discover is that the Bahamian users are complaining about the screen used to find local restaurants. On this screen, users enter their address and the application displays a list of participating restaurants in the area on the screen.
Now there’s something unique about Bahamian addresses: there’s no zip code or postal code. Instead, the code for the post office that services each address is baked into the address itself. But like 90 percent of applications built today, we’ve made the zip code a required field on our search screen.
It’s an easy enough bug to fix. We modify the search parameter screen so that the zip code is optional and diligently test to make sure the database allows nulls for the field. Soon, however, we start getting reports of errors in the search results page itself. It seems the mechanism in which we locate restaurants involves looking up the user’s zip code, retrieving the coordinates for it, and finding restaurants within a certain radius. And now that the zip code is optional, this service is failing spectacularly for our increasingly annoyed Bahamian patrons.
Tales from the trenches: The pinnacle of fear
We worked on a project with one particular module that was the source of a great deal of fear. At one point, the business rules changed and this module needed to be altered. When this requirement was mentioned to the team, developers, analysts, testers, and management began to show the signs of fear. People stuttered while trying to discuss the pending change. Noncoding solutions were suggested. Many team members literally broke into an anxious sweat.
Instead of diving straight into a solution, we decided to spend time researching the current state of the affected code. The reason for the fear quickly became apparent. Although there were many things wrong with the code structure, the root of the fear was driven by a lack of reproducible test scenarios. With over 100 code execution paths, the testing required to manually verify the complete set of code executions was unmanageable. Many scenarios were being missed or ignored, which resulted in the release of many new and/or regression defects.
After some work, automated tests were created for all the code execution scenarios. Subsequent requests for change from the business were met with far less trepidation by the developers. Over time and a number of releases without regression defects, management, analysts, testers, and the business clients gained confidence as well.
Now, one might argue that you didn’t do your due diligence when incorporating the new postal codes—and one would be right. After all, when doing something as major as accommodating a whole new country, it should be obvious that it would affect the page that searches based on location. But perhaps the developer who fixed the bug didn’t think that page searched based on zip or postal code. Or maybe the application is much bigger in scope and the developer made a valiant effort to catch all the scenarios but developers are only human and it was the end of the day and he had a bit of a cold and...
The point is, if every developer did comprehensive testing of bug fixes and didn’t miss any possible scenarios, there wouldn’t be any bugs in the first place.
What this contrived example illustrates is that fixing one bug can often introduce others. Often the new bugs are in areas you might not even have realized were related. For example, you might not imagine a lot of pain involved in fixing a bug in a timesheet entry system whereby we don’t allow entries after noon on the cutoff date. But what if we move the server from Toronto (where all the employees are) to Bangladesh and forget to account for the shift in time zones?
The underlying argument of this discussion is that without tests, you’ll eventually become hesitant, then reluctant, then outright fearful about fixing bugs or adding features to the codebase. You become afraid that even the slightest modification to a seemingly benign business rule will have far-reaching implications. We have yet to meet a single developer who enjoys having bugs he thought were fixed come back to haunt him in some other form.
It’s not just the development team that needs confidence in the application. If you’re afraid to work with the codebase, you can imagine how the management team and the client and users feel about it. It takes only one bad experience to create animosity between the business and the IT department. This animosity and distrust can reach the point where, when an application is needed, the customer will go outside the company to have it built. Needless to say, rebuilding such a relationship takes a long time. It is possible, though, so let’s take a look at some of the techniques.
4.2.1. Regaining confidence and control
We slid down a slippery slope a little at the end of the last section. Automated tests are not the be-all, end-all of your relationship with your clients (the end-all perhaps, but certainly not the be-all).
Let’s revisit our earlier scenario on zip codes in the Bahamas and change the parameters a bit. Now assume we have a full suite of unit and integration tests backing up our code. We hit the reset button, go back in time, and move into the Bahamas again for the first time with no changes to the application other than automated tests.
As before, say the system fails when customers try to enter an address with no zip code. But now, instead of diving in and fixing the code right away, we write some tests to expose the bug. We start with a unit test that creates an account with a typical Bahamian address, which fails, as we’d expect it to.
Challenge your assumptions: Write a test before fixing a bug
When you’re tasked with fixing a defect in a brownfield system, what’s your first step? Do you search out a person who you know worked on that area of the application and can provide you with some background? Do you just start debugging the code as you attempt to reproduce and resolve the reported issue?
Chances are that you’re working in a large codebase with areas you’re not intimately familiar with. Because of that, you’ll probably have to spend some time narrowing down the area where the bug resides. Once you know approximately where the problem is, you can begin to use our recommended technique: writing a test.
If you’ve adequately narrowed the point of failure, you should be able to write a test that exposes the bug. That is, don’t write a test that passes. Instead, write one that tests what you want the code to do but what it isn’t currently doing. Write a test that should pass but doesn’t.
Sure, it seems odd to intentionally create this failing test, but what you’re doing is preparing for the bug fix. You’re creating a harness that will confirm when you have the application working correctly. The test serves not only to notify you when you’ve done enough to resolve the issue, but it also will stay on as part of your testing suite to ensure that no future changes to the code will create the same bug again.
There’s a caveat: sometimes the code exists in a state that makes it hard to write a good test prior to fixing a bug. You may look at it and think that you have to refactor until it’s testable. If these are your thoughts, hesitate. By performing that refactoring, what would you introduce? Do you have sufficient tests available now to verify that your refactoring was successful? If not, do you want to introduce the risk of more defects at that point in time?
So do the best you can. Any verification that you can provide to yourself after working on the bug will be better than none.
So we write the code to accommodate the new address format and the test passes.
We’re not done yet, as we haven’t run the new code through its paces in integration tests yet. This task shouldn’t be too difficult; presumably, we already have integration tests that run the gamut for a test account using a U.S. zip code. We have tests that go through our location process end to end with a standard U.S. address. All we need to do is run through our suite of integration tests using a Bahamian address instead.
And lo! We get a failure. It seems that when we get to the test that retrieves the data from the web service based on the zip code, we get an error: unrecognized zip code. We can now write a test to expose this bug and fix it before going any further.
There’s something subtle at play here. Notice that in both the no-test and the full-test scenarios, we haven’t changed anything to do with how we address either bug. The code to fix it is identical in both cases. All we’ve done is shifted when and by whom the bug was discovered. Rather than releasing the code after fixing the first bug and having paying customers discover it, we’ve found it ourselves and can now fix it as part of the current release cycle. We’ve closed the feedback loop.
Note
Recall the feedback loop from section 3.2 in chapter 3. By having tests in place for your application, you increase the speed with which bugs are discovered. This is a good thing. Your application has bugs. If you can find them quickly, there’s less chance they’ll still exist when the application is released to an unsuspecting public.
Every step further away from the developer a bug is allowed to travel without being found increases the cost of repairing the bug.[1] The best way to minimize this cost is to catch defects in the application at development time. Automated tests can help with that job. Better yet, running these tests means we can announce with confidence that we haven’t introduced any new bugs while fixing something.
1 This concept is taken from Code Complete, 2nd Edition (Microsoft Press, 2004), by Steve McConnell.
Back to our example whereby we have a full suite of tests and you breathe a little sigh of relief that the integration tests saved you while the management team congratulates you on a job well done. Chances are the client won’t offer you any kudos for not introducing unforeseen bugs, but when it comes time to have another application built, at least they won’t say, “Whatever we do, don’t call the IT department.”[2]
2 With credit to Steve Martin from the movie, Roxanne
Although this kind of confidence is a benefit derived from automated test suites, there are other reasons to use them as well. Let’s spend some time looking at them.
4.2.2. The need for tests
The point of this discussion is that your application needs tests. Here are a few reasons why:
- To ensure bugs aren’t repeat offenders
- To ensure new development is of an acceptable quality
- To preserve and build on the confidence the various parties have in the applications
- Most importantly, to increase the speed of the feedback loop
However, ours is a brownfield application and possibly one of substantial size. If you don’t have any tests in place, we’re not so unreasonable as to assume you’ll halt all new development and bug fixes for the next 3 months so that you can retrofit unit and integration tests into your code.
But tests should always be near the forefront of your mind going forward. No new code should be checked in without being supported by tests. If you’re fixing bugs, have at least one test in place to ensure that the same bug does not get reintroduced inadvertently when another one is fixed. And new features don’t get added if they aren’t adequately covered by unit tests.
Note
Bugs are expensive and should be fixed once and only once. Writing a unit test to first expose the bug, then to verify it has been fixed, is a good way to ensure it doesn’t recur.
In this way, you ensure the important parts of your application are the focus for your tests. Yes, if there are currently no bugs in a section of code, it will never be covered by tests. But then, if there are no bugs, one might argue it doesn’t need tests.
So with that as our rallying cry, let’s turn our attention to specific types of tests, starting with unit tests.
4.3. Types of automated tests
We have no desire for this section to be a comprehensive discussion of techniques and practices for writing tests. There are entire books on the subject, including Roy Osherove’s excellent The Art of Unit Testing.[3] Instead of delving deep into the subject, we’ll give a brief overview of the two types of unit test to set the context, and then in the next section, discuss them within the context of a brownfield application. For the most part, however, we’ll assume you’re familiar with the concept of unit tests in general, if not the implementation. One key point to remember about your automated tests (unit or otherwise) is that they’re a strong form of documentation. Well-written tests that pass will provide invaluable feedback to future developers about how you intend the codebase to work.
3 Roy Osherove, The Art of Unit Testing (Manning, 2009)
What’s a unit?
The key to writing unit tests is defining the unit that’s being tested. In general terms, this is simply the smallest piece of code that can be tested.
The smallest piece of code that can be tested isn’t always obvious. If you’re testing whether a property on a class can be set properly, there isn’t a lot of ambiguity. But if you’re testing whether someone’s account was created successfully, things aren’t as obvious. Are you testing whether it was added to the database successfully? Whether it was sent to an account creation service? Whether the new account can be successfully retrieved (which may or may not be from the database)?
The good news is that practice makes perfect and help is only a web search away. Although you may experience early pain when defining your units, take heart that it does get easier with practice.
Assuming you’re able to define the unit to test, the next step is to test it. There are two broad ways of doing this testing, which we’ll discuss next.
4.3.1. State-based tests
The most basic and intuitive type of test is state based. In state-based tests, you verify that the state of an object, an object property, or a variable is as you expect it to be. Most often you do so by executing code in a method, followed by verifying some result. Depending on the code being exercised, the result may be a value returned from the method call. Other times you may verify values associated with an object.
State-based tests have no knowledge of the inner workings of the code being exercised. All we know is that it performs some action and we know what effect that action should have. As the name implies, the state-based test is based on the state of an object before and after an action is taken (see figure 4.2).
Figure 4.2. State-based testing doesn’t require any knowledge of the code under test.
Let’s look at an example. Listing 4.1 shows a sample state-based test using the NUnit unit-testing framework.
Listing 4.1. State-based test
As you can see in listing 4.1, there are three components to this simple test. First, 
shows the setup of information that is required to execute the test. Following the setup, 
highlights the execution of the code that we’re testing. Once the code under test has been executed, 
shows where the test verifies that the output is as expected.
Challenge your assumptions: Naming your tests
Don’t be afraid to be verbose when naming your tests. You’re not designing an API, nor are you creating methods that will be called by anything other than an automated testing framework. The main criterion for a good test name is that it should be easy for a developer to read and understand what the test is doing without having to parse the code. Better yet, a test name should enlighten the reader on the functionality of the application and, more precisely, the specific code under test.
In listing 4.1, we’re testing to ensure that the FullName method for Customer will concatenate the FirstName and LastName as we expect (LastName, comma, space, FirstName).
Note
We have no hint as to how the FullName method does its job. All we know is that it returns a string. Maybe it uses string concatenation, maybe it uses a StringBuilder, or maybe it sends the data off to another service. All we care about is that it returns a string and should be in a certain format.
Although listing 4.1 is a simple test, it’s important. With this test in place, if we ever decide to go back and alter the code in the Customer class, we can be reasonably confident that the FullName method will still behave as expected.
When this test fails, one of two things has happened: the output of the FullName method has unexpectedly changed, or the business requirements of FullName have changed, making this test no longer correct.
State-based tests are often referred to as black box tests because we see the inputs and verify the outputs, but the inner workings are a black box to the test. In contrast, interaction tests know how the code they’re testing works. We’ll look at interaction tests next.
4.3.2. Interaction tests
An interaction test is one where we test how two or more objects interact with each other. It’s best described after looking at an example.
Listing 4.2 shows a simple example using the Rhino Mocks framework. Don’t worry about the syntax or the explanations too much; we’ll clarify them shortly.
Listing 4.2. Interaction-based test
In listing 4.2, the object we’re testing is an OrderService . Specifically, we’re testing the effects of placing an order. What this test tells us is that when we call Load on an OrderService object 
, we expect the FetchById method to be called on an IOrderRepository object that’s passed into the OrderService object through the constructor call.
For developers new to interaction testing, there’s a lot to take in with this example. Like many interaction tests, it makes use of mock objects, which are proxy objects that take the place of real implementations. When we run this test, the mocking framework will autogenerate a class that implements the IOrderRepository, and then create an instance of the autogenerated class.
The reason we go through this exercise is because we don’t care how IOrderRepository works in the context of this test. We’re concerned with testing only the OrderService. Whether FetchById goes to a database, the file system, or a web service is irrelevant for the purpose of this test. All we care about is that the OrderService is going to call this method at some point.
So we create a mock implementation of IOrderRepository that we can manipulate in a way that facilitates our test. We set expectations on which calls the OrderService will make on the IOrderRepository object when we exercise our test.
In this way, we test how the OrderService interacts with its dependencies (the OrderRepository).
The Rhino Mocks syntax
In listing 4.2, you can see a number of things that are part of the Rhino Mocks framework. The first is the MockRepository.CreateMock<T> syntax, which will create a mock object that we can manipulate for our test. The returned object is a fake object that implements the IOrderRepository interface.
After creating the mock object, the next Rhino Mocks–specific syntax is the Expect (m =>...) call. Here we make assertions about what we expect to happen during our test. This code doesn’t execute at this time; it’s recorded so that we know how to react when the OrderService makes this call.
One of the nice things about Rhino Mocks is that its fluent interface for expectations is easy to understand. In this example we’re stating that we expect a call on the OrderRepository’s FetchById method. When that call happens, we don’t care about the argument being passed to the FetchById method, but we do expect that it will only be called one time. And when it’s called, we’d like it to return an empty List<Order>() object.
If you’re interested in learning more about Rhino Mocks, we suggest you look through the resources at www.ayende.com. Also, The Art of Unit Testing gives a good overview of Rhino Mocks as well as other mocking frameworks.
You don’t necessarily need to use a mocking framework to do interaction tests. You could create the fake objects manually (for instance, a FakeOrderRepository that implements IOrderRepository) if you’re having a hard time wrapping your head around the concept. This approach can be a good way to try out interaction tests without having to dive in head first. It doesn’t take long before you start experiencing pain from hand-rolled fake objects. In our experience, switching over to true mock objects sooner rather than later will save you more time over the long run.
Although far from a comprehensive discussion on interaction tests, this section has provided a quick introduction to set the stage for our later work on adding these tests to our codebase.
Specification-based tests
Another type of test that’s gaining momentum is specification-based tests or, more commonly, specifications.
A specification is a way of describing tests in a more natural language. In the agile world, specifications flow directly from the user stories. In conjunction with unit testing, the developers (in conjunction with the business users) write specifications about how the application should behave. Developers then write code with the goal of satisfying those specifications.
Specifications are written in support of an overall business requirement. Here’s a sample specification for the code that we’re testing in listing 4.2:
As a Customer Support Representative,
I can save an order,
So that the order can be fulfilled by the restaurant.The “As a/I can/So that” syntax is important because it lends itself to automated frameworks as well as reporting tools so that the gap can be closed between developer and business user. Indeed, we encourage you to write specifications alongside the business user.
Writing specifications, like unit tests, takes practice. If you feel you’d like to try your hand at them, we suggest using them for new code only, rather than trying to retrofit them over existing code. They’re generally more useful for outlining behavior that hasn’t been written rather than verifying the correctness of existing code. For the latter case, state-based or interaction tests would be better suited.
That’s about as wet as our feet are going to get on the theory behind unit tests. It’s time to get practical and determine how to apply automated testing to an existing codebase. We’ll start by addressing any existing test projects you may already have.
4.4. (Re-)integrating existing test projects
Your application may already contain unit tests. Perhaps the previous team had good intentions and started out ensuring all code was tested, but then they fell by the wayside in the face of mounting time pressures. Another possibility is that partway through the previous version of the application, it dawned on someone that unit tests might be a Good Thing and a halfhearted attempt was made to retrofit some in.
Whatever the reason, many brownfield applications will include a test project that’s incomplete at best and incorrect at worst. If you’re among the lucky ones, the existing tests will even pass, assuming they haven’t been ignored, commented out, or just plain removed from all the compile configurations in Visual Studio.
The underlying steps we’ll follow to revisit this test project are outlined in figure 4.3.
Figure 4.3. Steps to integrate an existing test project back into your process
With that, let’s look at your current tests.
4.4.1. Examining existing test projects
The first step in analyzing an existing test project is simply to see if it compiles. If the testing project isn’t currently part of your standard solution structure, leave it that way and work in a new solution for now. At this point in our analysis, the testing project (or projects, as it’s not uncommon to find more than one) will be volatile. Keep in mind we now have a continuous integration (CI) environment going. Adding instability into the environment of all the other developers would potentially impose a great deal of friction on them if we start breaking the build.
Getting test code to compile in isolation sounds well and good; all you have to do is go in and fix things until the project compiles, right? There’s that word right again. In this case, it means there are other things to think about once each test is compiling.
First, is the test project worth compiling? Are the tests valid? The tests probably haven’t been maintained for some time, so it’s possible that they no longer make sense in terms of the current code structure. Worse yet, the test code may look technically applicable, but business requirements may have changed so that the code is no longer relevant.
There are three things you can do with an invalid test:
- Delete it —If the test is no longer technically appropriate, delete it from the test suite. If you’ve started at the front of this book, you’ll already have this code in a version control system (VCS), which will allow you to retrieve it if the need ever arises. (Hint: It never does.)
- Refactor it —If the test is still potentially relevant, consider refactoring it. The problem with this approach is that it’s often faster just to delete the test and start from scratch. From personal experience, refactoring existing tests can be an arduous mental exercise. It often spirals into confusion about what should be tested, when, and in what manner. As a general rule, save yourself the headache and start fresh if you’re thinking of refactoring.
- Ignore it —Most testing frameworks have attributes that will allow you to keep the test in the suite but not have it execute during a test run. These attributes can be useful if the test still has business value but is currently failing and needs reworked.
Don’t ignore your tests
Don’t ignore your tests for too long. Ignored tests are like rotten apples in a box. If you have one, pretty soon you have two, then six, and then fifteen. Before long the stink they cause will catch your attention. What do you do when one day you realize that you have allowed 10 or 20 percent of your tests to become ignored? At that point you’re back where you were when we started this chapter: analyzing your test suite, fixing outdated tests, and deciding if you should keep them, ignore them, or delete them. Why bother going through this effort if you’re going to perpetuate this kind of analysis work for yourself?
The simple rule is that tests can’t be ignored for long.
It’s surprising how many test projects have been omitted from the overall solution simply because it became too much work to keep the code in sync with the application’s code. Test projects like this are at the extreme end of staleness. If you haven’t already done so, take the time to get the tests compiling, review them, and rewrite or purge them as required.
Once the tests are all compiling, the next step is simply to add the test project back into the solution. Because you’ve already read chapter 2, you’re using a VCS and the addition of the test project should be distributed to the rest of your team the next time they retrieve the latest version from it.
Now we come to an important step. Watch what happens when developers start to retrieve the latest version. Is the automated build now broken? Does the test project add friction to the developers’ work in any way?
If the answer to either question is yes, pull the test project back out as fast as you can. You need to address the issues, then add the project to the solution again. It’s important to act on the appearance of this (and, indeed, any) friction. If you ignore it, people will adapt. And they’ll probably adapt in ways that are detrimental to the project’s overall well-being.
For example, let’s say we add the test project to our solution and it includes a prebuild command that copies a file into the testing directory. And suppose that in some circumstances, like, say, when the application is running in Internet Information Services (IIS), the file is locked and can’t be copied. The prebuild command will fail intermittently, not allowing the project to be built.
In this scenario, we’ve added significant friction to one or more developers’ work. Now they have to remember to shut down the application in IIS while they’re compiling. Perhaps this leads to them not running the application in IIS anymore and using the built-in Cassini web server from Visual Studio. This could lead to issues when the application is deployed in IIS because you don’t use it regularly while developing the app.
This is the type of workaround you want to avoid. You’re changing a developer’s behavior to deal with an issue that should’ve been corrected early on: you should remove the test project at the first sign of friction, fix the problem, and then reintegrate the test project.
Tip
Don’t allow friction to remain in your work process long enough for you to find a workaround for it.
Analyzing existing tests and reintegrating them into the project can be a tedious first step. Once it’s done, your work isn’t over. Just because you have added back existing tests to the project, nothing says that those tests are valid. Even if all of the reintegrated tests are running and passing, they may not be verifying the code under test correctly. In the next section we’ll look at how we can review those tests and determine whether they’re working as we need them to.
4.4.2. Address existing tests
Assuming that you have been able to reintegrate the test project, you now have all your production and testing code in one environment. With all the tests compiling and included in the solution, the next step is to start adding tests, riiiiiight?
Well, yes, the next step is to start adding tests, and we’ll go into that later in the chapter. But at the same time, you should do something else in parallel: review the existing tests.
Regardless of your path to get here, you have tests from the past in your solution now. One of the questions for you and the rest of your team is, “How valid or correct are the existing tests?” Just because you have tests doesn’t mean that they’re providing any value to you.
Unit tests can be invalid or incorrect for any number of reasons. In general, an incorrect test is any unit test that gives the illusion of validating a piece of code but doesn’t provide any significant or meaningful verification of the code it’s exercising. Tests like these provide a false sense of security. They build confidence that’s misplaced and set to crumble. These aren’t good traits for a project or a codebase to have.
Although you may not be able to determine all the reasons that unit tests can be incorrect, they’ll fall into three broad categories, as shown in figure 4.4:
- They don’t test a unit.
- The expectations aren’t meaningful.
- They don’t reflect business requirements.
Figure 4.4. There are many reasons a test may be invalid.
Let’s examine each of these categories.
Tests That Don’t Verify a Single Unit
The first category, tests that don’t address a unit, means your test is verifying more than one single unit—for example, a test that checks whether a customer is valid, then checks to see if it was saved to the database.
That we consider this “incorrect” may seem confusing. After all, we need to test how the pieces work together, don’t we? Of course we do. But the problem with tests like these is that they usually present several code execution paths that require verification. More paths to verify mean more permutations in your test. More permutations mean more test configuration. And more test configuration means brittle tests.
Instead of trying to test multiple things in a single test, it’s much easier to test each piece (or execution path) separately. Then you can have a separate test altogether that verifies how the pieces interact with one another. One effective method is to test each unit of code thoroughly by itself using state-based tests. This approach will allow you to verify that the results are as you expect for each of the code paths. If the unit of code being tested is small enough, the setup required to exercise all the execution paths will be quite manageable.
Many brownfield applications will contain integration tests that claim to be unit tests. These are usually easy to spot. They are long-winded and have complicated setups compared to what they’re testing. Often they span multiple layers of your application and even connect up to a database or read a configuration file. In any case, these aren’t unit tests. But neither are they to be discarded outright. More likely, they are integration tests, which we’ll discuss later. Evaluate them to see if they qualify as valid integration tests, and if so, move them to your integration test area.
You still need to verify that the pieces of code will interact with one another as expected. This point is where interaction testing and mocking frameworks come into play. Again, because the unit of code under test is smaller, configuring the tests for each permutation of interaction should require minimal coding effort.
Getting a brownfield application to the point where you can adequately test interactions between pieces of code can take some time. To effectively do interaction testing, you must have clearly defined seams and logical layers (we’ll discuss layering in chapter 8) in your code. Without this, interaction testing can be enormously frustrating to the point of being impossible.
If you find tests that require a lot of setup before the code under test is executed, this can be a sign of poor tests. It can also be a sign of code that doesn’t adhere to the single responsibility principle (discussed in chapter 7). If this is the case, the solution is to refactor the existing code into smaller pieces (units) and write both state- and interaction-based tests for all of them.
Tests That Don’t Set Meaningful Expectations
Even if you’re testing a cohesive unit of code, the tests may not be of any value. As you can see in listing 4.3, a test can pass, but not fully test the execution of the code it’s exercising.
Listing 4.3. Expectation lacking meaning
[Test]
public void VerifyMenuItemImport()
{
string sampleItem =
"Item(Name:Chicken Vindaloo, Price:$11.95, Category:Entree)";
IImportMenuItem menuItemParser = new ImportMenuItemParser();
MenuItem menuItem = menuItemParser.For(sampleItem);
Assert.That(menuItem, Is.Not.Null);
}
The expectations and assertions that our tests make should be clear and rigid whenever possible. In listing 4.3, the intent of the code being exercised isn’t clear enough. All we know from this test is that the method call should be returning a non-null MenuItem object. Think about this for a second. What’s the minimum amount of code that you need to write to make that test pass?
The code in listing 4.4 will pass our test. But few would agree that it’s valid code. The code in listing 4.4 is probably wrong—we all can guess that—but the test in listing 4.3 isn’t explicit enough to give us confidence to say that it’s definitely wrong. Based on the name of the test alone, it would be easy for us to get a warm, fuzzy feeling that everything was okay.
Listing 4.4. Minimum code to pass test
public class ImportMenuItemParser : IImportMenuItem
{
public MenuItem For(string itemToBeParsed)
{
return new MenuItem();
}
}
So what would give us that confidence then? We have to write a test that sets the minimum expectation that we have for the code as clearly as possible. In this case we expect to have a MenuItem returned to us that has a Name, Price and Category. The first thing we need to do is change the name of the test to reflect our goal (see listing 4.5).
Listing 4.5. Intention-revealing test name
[Test]
public void ImportMenuItemParser_Should_Transform_Input_Into_MenuItem()
{
string sampleItem =
"Item(Name:Chicken Vindaloo, Price:$11.95, Category:Entree)";
IImportMenuItem menuItemParser = new ImportMenuItemParser();
MenuItem menuItem = menuItemParser.For(sampleItem);
Assert.That(menuItem.Name, Is.EqualTo("Chicken Vindaloo"));
Assert.That(menuItem.Price, Is.EqualTo(11.95));
Assert.That(menuItem.Category, Is.EqualTo("Entree"));
}
That’s better. You now can say with much more certainty that the ImportMenuItemParser must parse the input string and assign values from it into the properties on the MenuItem object. The minimal amount of code required to make this test pass has changed significantly. If we were to exercise the code in listing 4.4 with the test in listing 4.5, the test would fail.
Vagueness in testing, such as we just displayed, tends to be rampant in brownfield applications. Over time, developers may have become frustrated with the complexity of testing their codebase. Instead of refactoring the codebase so that it adheres to principles like single responsibility and separation of concerns (see chapter 7), they’ve relaxed their testing discipline. Not only has this increased the fragility of the test suite, it has also reduced its effectiveness. This is a classic sign of a brownfield application and is a direct cause of the contamination that defines it.
Tests That Don’t Reflect Business Requirements
Another set of tests that are prime candidates for deletion are those that are no longer testing functionality that meets business requirements. Like the code that these tests are exercising, they’ve been forgotten. The code may work fine, the test may pass, and the test may be testing all the right things, but the code being tested may not be used in the application in any meaningful way. In fact, it’s not uncommon for the tests to be the only place in the application that calls the code being tested.
Of all the incorrect test types that you’re going to search for in your brownfield application, tests that don’t reflect business requirements will be the hardest to find. Until you have knowledge about the application’s inner workings, you won’t be able to recognize that these tests aren’t calling useful code.
Most likely, you’ll find them by accident. You’ll probably stumble into a piece of long-retired code, delete it, and then find that one or more tests will no longer compile.
These tests aren’t hurting anything by existing. At worst they’re inflating your code coverage statistics (see chapter 6). At best, they’re protecting a unit of code that you may, on some mythical day, start using again. It’s our opinion, however, that if you encounter code that’s no longer being used in the application, you should delete it. You may feel that you’ll need it at some point, but that’s your VCS’s job: to maintain that history and keep it hidden from you on a daily basis. Code and tests, in this circumstance, are adding to the maintenance workload that your project is creating. Deleting them helps a little in reducing that load.
Tip
We suggest not going out of your way to search for tests that test irrelevant code. The reward-to-effort ratio is far too low. As you refactor the application, you’ll come across these tests, and they can be deleted at that time along with the defunct code.
In this section we explained that as part of addressing existing tests, you’ll identify and categorize tests as either unit or integration tests. Because these serve different purposes, let’s look at how we can separate them for easier management.
4.4.3. Separate unit tests from integration tests
The final step in the process of adding inherited unit tests back into the solution is to segregate true unit tests from integration tests. We’ll talk later in this chapter about what defines integration tests, but for now think of them as tests that execute more than one unit of code at a time. Because integration tests run slower than unit tests, separating them can reduce test execution friction. We talked a little about how to overcome this in chapter 3.
Physically separating these two types of tests is usually a good practice. By “physically,” we mean into entirely separate folders or even separate assemblies. Having integration test classes intermingled with unit test classes in the same folder will make it more difficult to separate the two of them in the build script for your CI process, which is something that might become necessary at some point during the project.
Why would we want to do this? Well, integration tests run much slower than unit tests. It’s their nature. They’re testing how components, such as your code, databases, and web services, integrate. Instead of forcing the developers to always run slow integration tests, split them so that the option of running the slow tests can exist. Having a process for developers to run only unit (fast) tests when appropriate can help to increase the speed at which they work.
So at the very least, we recommend creating two folders at the root of your test project: one for integration tests and the other for unit tests (see figure 4.5). Within each of these folders, you can use whatever subfolder or file structure you like. But as long as they’re separated at some root level in the file system, they can be easily separated within the build script should the need arise.
Figure 4.5. Physically separating unit and integration tests in the folder structure allows for build scripts to easily create separate testing assemblies.
Another option is to have one test project for integration tests and one for unit tests. Although this will work, this solution is overkill. It’s effectively the same as segregating the tests into separate folders in one project. So unless there’s a technical need for the two types of tests to be separated (and compiling them into separate assemblies isn’t one of those needs), it’s easier to keep the tests within a single project.
Tales from the trenches: Testing at a snail’s pace
We joined a project that had a reasonably sized suite of unit and integration tests. As development continued, developers began to complain that the amount of time to run the automated build script, which included test execution, was slowing down their development progress.
In this case, the script took over 30 minutes, sometimes over 40, to execute completely. If one unit test failed during execution of that script, developers would have to fix the problem and rerun the build script, which cost them another 30 minutes of effective working time.
When we looked at the tests, it was apparent that unit and integration tests needed to be separated. A semblance of separation existed, but some integration tests existed in the unit-testing folders, and vice versa. After taking some time to whittle through the tests and moving those running slowly into the integration area, we set about altering the build script.
Instead of developers running one script to execute all tests, we configured it to execute the script for unit tests alone, integration tests alone, or both combined. By allowing the developers to run the fast tests at times, we cut the wait time for test execution from 30 to 40 minutes every time down to 5 to 6 minutes most times. In the end, the team still complained that 5 to 6 minutes was too long to wait, but at least we had them working at a much faster pace and we could concentrate on trying to improve the speed.
Regardless of how you organize your tests, you should always be looking at the tests as a way to expose current or potential future issues in your code. As we move into the next sections, we’ll talk more about this.
4.4.4. Reexamining your tests
We’ve gone through a lot of techniques for identifying tests that you may want to include, exclude, and eliminate from your brownfield test suite. While you’re going through this process, there’s one thing that you should be watching for at all times. Look for areas of brittleness in the test suite. When you’re writing tests, note the ones that take a lot of effort to write. When tests are failing, watch for areas that seem to be breaking on a regular basis and gather information about the points of failure.
That information is something that you should file away and look to address in the future. It may be useful when making code design decisions like creating or modifying the logical layers in the code. Having gone through the test re-inclusion exercise will have given you some idea as to what areas of the application should be attended to and in what order. Tests are client code to your application. Looking at it from the point of view that a client would will give you a different view of the code’s state.
Tip
Any code that you write will have a client. At a minimum this client will be a developer (most likely you) who is writing tests to exercise the code. Possibly, the client will be a developer who’s consuming the code in another part of the application. On the far end of the spectrum, the client could be a developer who has purchased your third-party utility or control and is adding it into their application. Writing code with the concerns of your client in mind will increase its usability and readability. One effective technique for achieving this is test-driven design (see section 4.6). Because it’s concerned with creating and designing code from the code client’s perspective, and doing so before writing the actual code, you’ll be writing code as a client on a regular basis.
Writing code from its consumer’s perspective is a good practice. Unfortunately, writing code this way doesn’t provide verification that the code written is functional. Because automated tests are one of the clients for your code, executing those tests can provide some of that verification. Let’s spend some time on this topic now.
4.4.5. Executing the tests in your build script
Having completed a review of the inherited tests (and reworking as necessary), the final step of the process of reintegrating existing tests to your brownfield application is to have the build process execute them. By adding the execution of the tests to the automated build process, you’re ensuring that every time the build script is executed, either by a developer on their local machine or on the CI server, the suite of tests is executed as well.
If you’ve followed our advice from chapter 3, you’ll already have an automated build script in place. So the next step is to add a task to the script that will exercise your tests. How this is implemented will depend on the CI software and unit-testing framework you’re using (and even then, there’s often more than one way of skinning this cat). Here’s a simple NAnt target that will execute all tests within a given assembly using NUnit:
<target name="test" depends="init">
<nunit2 failonerror="true">
<formatter type="Plain" />
<test assemblyname="${build.dir}/${test.assembly}" />
</nunit2>
</target>
The sample uses the <nunit2> NAnt task to execute all unit tests within the assembly, ${test.assembly}, which is defined elsewhere in the build script.
An integral part of this sample is the failonerror attribute. This means that if any of the tests fail, the entire build will fail. A failing test has the same status as a compile error. A broken test is the sign of something unexpected in your code, and it indicates that the build process doesn’t have confidence in the compiled code. If the automated build script doesn’t have confidence in the compilation, you shouldn’t either. Code compiled with failing tests shouldn’t be deployed to any environment, test or otherwise.
Note
We can’t recommend strongly enough the importance of ensuring that the overall build fails if one or more tests fail. It may seem harsh at first, but it indicates that a written expectation (your failing test) hasn’t been met. When expectations aren’t being met, the application isn’t working as you’d desired. In this regard, failing tests are as serious as a compilation error because the application isn’t functional in this state.
With your unit-test suite being executed during your build process, you’ve completely reintegrated the inherited tests. You now have a suite of tests that have been vetted for correctness and verified to pass. Along the way you’ll have removed some tests that weren’t providing you with any benefits and modified some to provide more than they did before.
Overall, you’ve increased your confidence in the codebase, and that’s what’s known in the industry as a Good Thing. You could stop here and still be much better off than you were at the beginning of this chapter. We’re certainly breathing a little easier.
But there’s much more to do. There are ways to further increase the confidence that the developers have in the quality of the product. That, in turn, will increase the confidence that QA, management, the client, and other groups have in the product as well. As that happens, your project will become much more enjoyable to work on. Because that’s something we all want, let’s look at what we have to do to add new tests to the project.
4.5. Adding a new test project
For those applications that don’t yet have a test project, you haven’t been forgotten. In this section, we’ll walk through the steps for adding a new test project and integrating it into the CI process.
And, thankfully, there isn’t much you need to do. The steps are a subset of integrating an existing test project, except that you have to create the project first.
So first let’s create a Class Library project in your solution with an appropriate name (such as MyApplication.Tests). Next, add a reference to the unit-testing framework you’re using.
Note
Recall from chapter 2 that the unit-testing framework files you reference, along with all their dependencies, should be stored within the tools folder of your root application folder. To find the files you need to include, check for a subfolder called bin in the folder where you installed or unzipped the unit-testing framework. Most frameworks will store everything you need within this folder, and it’s a matter of copying the entire thing into your tools folder. Because you’ll probably end up with more than one tool, we also recommend an appropriately named subfolder (such as NUnit, MbUnit, or xUnit).
After adding the project, create two folders in it: one for unit tests and one for integration tests. As mentioned in section 4.4.3, it is helpful to have unit tests and integration tests physically separated so that they can be compiled, and ultimately executed, separately if need be.
With the two subfolders created, we can already skip to the end and add the test project to the CI process. This is done in the same way as outlined in section 4.4.5: you add a step to the CI process to exercise the tests in your test assembly (the code snippet in section 4.4.5, which shows a sample NAnt task for integrating NUnit tests into the automated build, applies for new test projects as well as existing test projects).
Note
You may find that after integrating your new test project into the automated build, the build fails because there are no tests to execute. If this is the case, just write a test. Any test. If you can pick off one that makes sense from your code, that would be an ideal starting point. Otherwise, create a class called DeleteMe and add a test that asserts that 1 + 1 = 2. The important part is to get the tests executing in the CI process as quickly as possible. After that, any additional tests will execute with this safety net in place.
As with the existing test project scenario, make sure the build is configured to fail whenever one or more tests in your test suite fail. Once that is done, we can start adding new tests to the codebase.
4.6. Writing new tests
By this point, you should have a test project in place with all the tests within it being executed as part of your automated build in your CI process. We now have the framework in place to write some new tests.
One of the key parts in the definition of a brownfield application is that the codebase is in active development. As part of that work, we expect that there will be refactorings as well as completely new features added. And as we’ve mentioned before, it’s difficult to have confidence in code if it isn’t being tested. After all, how can we be sure that modifying existing code or adding new code won’t break something?
As you might expect with two different scenarios (writing new code vs. modifying existing code), there are multiple ways to approach the test writing. First, let’s look at writing new tests when you’re modifying existing code. It’s this scenario that will offer the most challenges.
4.6.1. Tests on existing code
Modifying existing code to add new or changed functionality is much like fixing bugs. It’s likely you’re going into an area of the application that may not be covered by tests at all. Making changes with confidence in that scenario is hard to do. So how do we resolve that?
The scenario we discuss here is slightly different than when you fix a bug. Here, we’re adding a new feature or changing existing functionality. It’s a subtle difference but the method to attack it, testing, is the same. You should never make a change to code until there’s a test in place first to prove that your change will be successful when you’re done. Don’t worry about testing the current functionality; it’s going to go away. Instead, create tests that verify the functionality you intend to have after making your changes. Write a test as if the code is already doing what you want it to do. This test will fail, but that’s good. Now you’ll know when your modifications are done: when the test passes.
Let’s look at a simple example to explain what we mean. Say you have a service that retrieves the display name for a customer, as we do here:
public string GetDisplayNameFor( Customer customer )
{
string format = "{0} {1}";
return String.Format( format, customer.FirstName, customer.LastName );
}
Because ours is a typical brownfield application, there are no tests for this method. For now, we’re fine with this because no bugs have been reported as yet.
Now, the client says that they’d like to change the format of the customer’s name to LastName, FirstName. After searching through the code, we zero in on this function and...then what?
This is a benign change, so you’d probably just change the format string and be done with it. But let’s rein our instincts in for a moment and try to make sure this issue doesn’t come back to haunt us.
Instead of diving into the code, let’s write a test to verify that this function behaves the way we want it to (listing 4.6).
Listing 4.6. Testing to verify desired functionality
[Test]
public void Display_name_should_show_last_name_comma_first_name( )
{
MyNameService sut = new MyNameService( );
Customer customer = new Customer( "Tony", "Danza" );
string displayName = sut.GetDisplayNameFor( customer );
Assert.That( displayName, Is.EqualTo( "Danza, Tony" );
}
Listing 4.6 is a test that verifies that our name display service returns names in the format LastName, FirstName.
You may be asking yourself, “Won’t this test fail in the current code?” The answer is, “You better believe it’ll fail.” The current implementation of this function displays the name in FirstName, LastName format.
But we know we’re about to change the current functionality, so there’s no point verifying that it’s correct. Rather, we want to make sure that the new functionality is correct. And we do that by writing the test first.
By writing the test for the modified functionality first and having it fail, we’ve now given ourselves a specific goal. Setting it up in this way makes it unambiguous as to when we’re finished. We aren’t done until the test passes.
For completeness, let’s modify the GetDisplayNameFor method so that it passes our test:
public string GetDisplayNameFor( Customer customer )
{
string format = "{1}, {0}";
return String.Format( format, customer.FirstName, customer.LastName );
}
We run the tests in our automated build and voilà: the tests pass. We can now commit both the tests and the newly modified code to our source code repository.
The reality of testing existing code
Our example was just shy of being utopian (read: unrealistic) in its simplicity. In many cases, maybe even most, it will be nearly impossible to test existing code in isolation in this manner. Perhaps you have a single function that (a) retrieves a connection string from the app.config, (b) opens a connection to the database, (c) executes some embedded SQL, and (d) returns the resulting DataSet.
If you need to modify such a function in any way, you’re looking at a fairly significant refactoring just to get this one piece of code properly layered so that you can test each of its tasks.
In part 2 of this book, we’ll go into layering in more detail as well as various other techniques for teasing apart your code to facilitate testing.
In the end, you achieve the modified functionality you wanted, confidence in that functionality, and the ability to perform ongoing regression testing for that functionality if anything changes. Instead of just adding the changes that the client needs, you’ve added long-term reinforcement as well.
With our existing code covered (so to speak), we’ll now talk about approaches for testing against new code.
4.6.2. Tests on new code
When you’re working on your brownfield application, you’ll run into one more new-test scenario: that glorious time when you get to write all new code, such as a new module for your application. More likely than not, you’ll be tying into existing code, but you’ll be adding, not modifying, most of the time. It’s like your own little greenfield oasis in the middle of a brownfield desert.
Compared with testing existing code, writing tests for new code should be a breeze. You have more control over the code being tested and chances are that you’ll run into less “untestable” code.
There are two ways to write unit tests for new code: before the new code is written or after. Traditionally, test-after development (TAD) has been the most popular because it’s intuitive and can be easily understood.
Writing tests before you write your code is more commonly known as test-driven design (TDD). We won’t pretend this topic is even remotely within the scope of this chapter (or even this book), but since we have your attention, it’s our opinion that TDD is worth looking into and adopting as a design methodology.
Test-driven design
Although it has little bearing on brownfield application development specifically, we can’t resist at least a sidebar on test-driven design. It’s worth your effort to evaluate it at your organization and try it on a small scale.
Using TDD allows for the creation of code in response to a design that you’ve determined first. Yes, folks, you read it here: TDD is about designing first. Unlike some schools of thought, this design process doesn’t usually include UML, flow charts, state diagrams, data flow diagrams, or any other design tool supported by a GUI-driven piece of software. Instead, the design is laid out through the process of writing code in the form of tests. It’s still design, but it’s done in a different medium. The tests, while valuable, are more of a side effect of the methodology and provide ongoing benefit throughout the life of the project.
Let it not be said that it’s an easy thing to learn. But there are a lot of resources available in the form of books, articles, screencasts, and blogs. Help is never more than an email away.
Whether you choose to write your tests before or after you write your new code, the important thing is that you write them. And execute them along with your automated build.
As we mentioned, writing tests against new code is similar to how you’d do it in a greenfield project. As such, it’s a topic best left for a book more focused on unit testing.
Now that you’ve mastered creating unit tests in different scenarios, it’s time to move on to integration testing.
4.7. Integration tests
Testing code in isolation is all well and good for day-to-day development, but at some point, the rubber is going to hit the road and you need to know that your application works on an end-to-end basis, from the UI to the database. That’s the role of integration tests.
Note
In the context of integration tests, end-to-end extends from the database on one side up to, but not including, the user interface on the other. We’ll look more at the user interface in the next section, but for now, consider integration tests as “subcutaneous” tests: ones that start right below the surface of the UI.
When unit testing, we often create mock or fake objects to take the place of anything that is extraneous to the object we’re testing. By creating and using mock objects we don’t connect to a database or read a configuration file. Rather, we create objects to mimic these actions so that we can test the real logic.
But the time will come where you want to make sure that you can connect to the database to retrieve the data you want or that you can read the configuration file and that it’s returning the information you expect. More than that, you want to make sure that the Save method on your web service will do everything from start to finish, regardless of how many components are involved: populate it with defaults, validate the object, save it to the database, return the new ID, and so on. And although your unit tests can verify that each of these tasks works in isolation, it isn’t until the integration tests that we can be totally confident that a task works in its entirety.
Integration tests are often easier to write than unit tests. They don’t require a lot of thought with respect to how your components interact. The setup for the tests is usually a lot of busy work to configure databases and web services and such. It’s often not trivial, but it’s intuitive.
A good candidate for starting your integration testing is your data access layer. During unit testing, you usually try to abstract the database to such an extent that you don’t even care if it exists. But at some point, you’ll need to verify that you can, in fact, connect to a database somehow, whether it’s with a direct DbConnection or through an object-relational mapper tool (more on those in section 4.9). At the very least, you need to verify that the permissions are appropriate in your database, which isn’t something that’s easily tested in a unit test.
Challenge your assumptions: Let your users write your acceptance tests
There’s an increasing desire to incorporate an application’s users into the software development process. An extension of this idea is to have your users actively involved in writing the acceptance tests that drive out the application’s behavior. We’ve already touched on this in the sidebar “Specification-based tests” earlier in this chapter.
One way to encourage such collaboration is through the use of a tool that facilitates this kind of interaction. One example of such a tool is FitNesse.[4] With it, users and developers collaborate in defining what the software does as well as the tests that will define whether the software does what it purports to do.
Regardless of whether you use such a tool, it’s not a replacement for anything we define here. Its strengths lay more in the acceptance testing phase of an application. As with any tool, your individual team dynamics will decide whether it has any merit.
Because unit tests require small segregated components for testing, and brownfield applications tend to have few of these, integration tests can be easier to add to existing projects. Adding integration tests to brownfield applications isn’t quite the same ordeal as adding unit tests. The reason is that no matter how convoluted your objects are underneath the covers, you typically test at a higher level. The fact that your mammoth method reads a configuration file, connects to a database, and executes embedded SQL doesn’t matter to the integration test. All the setup required to make that work would need to be in place regardless of how the application is layered.
Writing integration tests for a brownfield app is almost identical to doing it for a greenfield app. In both cases, you write the tests after the code is completed. The difference is that in a greenfield application, you’d presumably write your integration test as soon as possible. In a brownfield application, the code has been in place for some time and you’re essentially retrofitting integration tests into it.
As with unit tests, you don’t necessarily need to add integration tests to existing code just for the sake of it. But similarly, you should add integration tests whenever you modify existing code.
Challenge your assumptions: Execute the integration tests separately
Integration tests can add considerable time to your automated build process. Almost invariably, this extra time results from accessing external resources (such as a file, a network resource, a web service, or a database). Typically, integration tests open the resource, use it, and then close it again. There’s no room for caching in an integration test (unless that’s specifically what you’re testing).
In chapter 3, we discussed a method of chaining the running of your integration tests after your unit tests. Remember when we suggested that you work to segregate your unit and integration tests? Here’s where that effort pays off.
Integration tests can add considerable time to your automated build process. Almost invariably, this extra time results from accessing external resources (such as a file, a network resource, a web service, or a database). Typically, integration tests open the resource, use it, and then close it again. There’s no room for caching in an integration test (unless that’s specifically what you’re testing).
In chapter 3, we discussed a method of chaining the running of your integration tests after your unit tests. Remember when we suggested that you work to segregate your unit and integration tests? Here’s where that effort pays off.
To recap, you could configure your main build process on the CI server to compile the application, run the tests, and do everything else one would expect. When it completes successfully, you can have it trigger a second build process that executes the integration tests. But developers don’t need to wait for the integration tests to complete before continuing with their work.
As mentioned in chapter 3, you must evaluate both the benefits and the risks. The point, as usual, is to make sure you’re reducing friction for your developers, not adding to it.
As we mentioned at the start of this section, we’re looking at integration tests as being subcutaneous in nature, right below the UI. Having tests that exist immediately below the UI doesn’t, however, mandate that there’s no need for testing at the user interface level. Let’s take a look at this type of testing now.
4.8. User interface tests
The thorn in the side of any automated build process is the thing that users most identify with for the applications: the user interface. It’s just not that easy to mimic clicks on a screen in an automated fashion.
In many ways, the jury is still out on how far you take testing of the user interface. On the one hand, a properly layered application won’t have any test-worthy logic in the UI. Using patterns like Model-View-Controller and Model-View-Presenter (see chapter 10), the UI code becomes a series of events that call other, more testable code. Indeed, if you have a method in your Controller class called UpdateJob and it’s fully tested with unit and integration tests, there’s little to be gained by testing the click event for a button that calls this method.
Because ours is a brownfield application, we can’t assume our UI is sufficiently thin that UI tests are totally unnecessary. And with the rise in popularity of JavaScript libraries and Ajax, web UIs are becoming increasingly feature-rich so that they can have complex business and validation logic in them.
In addition, UI testing frameworks have become more mature and easy to use, and many have strong communities where you can get help when you run into trouble. These are valuable advantages because as we’ve said several times, if a process causes friction with developers, they’ll avoid it.
One possible consideration with UI testing is that the UI can be volatile, especially for new features. Converting a ListBox to a RadioButtonList is no longer as trivial a task as it was before you had a full suite of tests covering the page.
Whether or not your application needs UI testing will be a judgment call, like anything else. Most of the time, UI tests aren’t as necessary as unit and integration tests, especially if you’re able to apply comprehensive integration tests close to the UI surface. But if you find that a lot of bugs are being discovered due to UI interactions, UI testing is definitely something worth considering. As always, be sure the tests are integrated into your automated build process and that any failures in them will cause the entire build to fail.
It’s possible to write your own tests to mimic the UI. In many cases, it could be as simple as instantiating an object representing the form you want to test and calling methods or raising events on it. There’s usually much more to this process than that (like ensuring it’s using the proper configuration and handling dialogs), but for simple forms it’s a viable option.
We also mentioned the use of UI testing frameworks. Some popular ones are Watir/WatiN (Web Application Testing in Ruby/.NET) or Selenium for automating web browsers, and NUnitForms for automated Windows Forms applications. In addition, projects such as JsUnit allow you to test your JavaScript in isolation of the UI.
Bear in mind that everything comes with a cost. UI testing isn’t trivial to automate, especially in a CI environment. For example, automating a web application may require the testing framework to physically launch Internet Explorer, Firefox, Opera, Safari, and Chrome (see figure 4.6)—which means that all these browsers need to be installed on the CI server. And browsers were meant to be interacted with. All it takes is a single alert call in your JavaScript to launch a dozen browser instances on your CI server over the course of a day.
Figure 4.6. Many considerations are involved when you’re testing a web interface, not least of which is the fact that you must test on all major web browsers.
All this testing can seem overwhelming at first. You may have started this chapter thinking “Yeah, I know we need to unit test but I don’t know how to get started.” Now, your head is swimming with integration tests and UI tests (and shortly, database tests).
Don’t get too wrapped up in the implementations of these tests. One of the nice things about automated testing in a brownfield application is that anything you do is almost always useful. Pace yourself while you’re getting started, and as always, keep your eye on the pain points. You’ll be an expert tester in no time.
In the meantime, let’s see what we can do about testing your database.
4.9. Managing your database
Most applications require a centralized database of some fashion. And this fact can be a source of friction with developers, especially if they’re all accessing the same database. Having more than one developer actively developing against the same database is akin to both of them trying to write code on the same computer against a single codebase. Eventually, two people will be working in the same area at the same time.
And even if they don’t, at some point you may need to alter the schema. Perhaps a column has been added or you need to modify a stored procedure while you’re modifying a feature. If others are working against the same database, they could have problems because they’re using the modified database schema with an older version of the code, at least until you check in your changes.
Integration tests and automated builds add another level of complexity to this equation. Chances are your integration tests are adding, modifying, and deleting data from the database. Woe betide any developer trying to work against a database when integration tests are running against it.
The advice here shouldn’t be a surprise: each developer should have his or her own copy of the database if it’s logistically possible. Ideally, the local database would run locally on the developer’s own machine, but the next best thing would be a separate instance running on a server elsewhere.
By having individual copies of the database, you not only eliminate the risk of having an updated schema with outdated code, but you can also run the integration tests at any time without fear of interfering with anyone else.
Note
It should go without saying that you should have separate databases for each environment as well. For example, you don’t want your quality assurance department working against the same database as your user acceptance testers. And no one should be looking at the database used by the CI server for the automated build.
Having many different databases floating around your development project introduces a management and maintenance problem. Let’s look at how we can mitigate the risks that it introduces.
4.9.1. Autogenerating the database
When executing your integration tests, it’s best if they run against a clean copy of the database—one built from scratch and populated with appropriate initial data.
If you don’t start with a clean database, you tend to end up with data-dependent tests that fail sporadically due to issues with the data. For example, perhaps your integration tests insert a record into a customer table using a known customer ID for the sake of testing various aspects of the data access layer. If you don’t remove this data after you’ve done your tests, you’ll get a failure the next time you run the tests as they try to insert the same customer with the same ID. This issue wouldn’t occur in production and yet it’s causing problems during testing.
So it’s a good idea to include in your build process a task that will create the database, initialize it with test data, and drop it (see figure 4.7). There may even be cases when you want to do this process several times within a single test run. For example, if you don’t want to use the same test data across the whole suite of integration tests, you may drop, re-create, and populate the database for each functional area of the application. Incidentally, dropping and recreating a testing database is one reason why integration tests can take much, much longer to execute than unit tests.
Figure 4.7. The process for running integrations tests against a known set of data
There are a number of ways to accomplish this sequence of events. The most obvious way is to do it yourself. Creating, populating, and dropping a database are all functions that can be done through SQL queries, and most database applications provide a command-line interface for executing queries against it. With this method, the task in your automated build simply needs to execute a series of command-line calls.
This approach may be adequate for small applications, but it can become cumbersome very quickly. You must manage every aspect of the process, from the data definition language (DDL) scripts, to the data loading scripts, to the commands that execute them.
Luckily, there are other options that make this easier, though in our opinion, still not entirely friction free.
If you use an object-relational mapper tool (described in chapter 10), it may offer the ability to autogenerate your schema, dropping the existing database automatically. That still leaves you with the task of populating the database if that’s something you need to do.
A number of third-party applications and frameworks are available that can help you manage your database. Some are geared toward the process of promoting your application between environments, but they can be easily adapted to an automated build process. Examples of products in this area include
- Microsoft Visual Studio Team System for Database Professionals
- Red Gate’s SQL Toolkit
- Ruby on Rails Migrations
- NDbUnit (a library for putting a database into a known state)
Once your application has an existing version of its database in a production environment, the requirements for how you manage changes to that database’s structure and content change.
4.9.2. Dealing with an existing database
Because ours is a brownfield application, there’s a good chance a database already exists in production. In that case, you no longer have the luxury of being able to deploy the database from scratch. Now you have a versioning issue in database form; how do you manage all the changes made to the database after it’s been rolled out?
Again, you could manage changes yourself, keeping track of modifications to the schema and creating appropriate ALTER TABLE scripts in your VCS. But you almost certainly will run into problems when you make a change to the database, and then need to propagate that change to the other developers—not an impossible task, but certainly unpalatable.
Challenge your assumptions: Migrations for database management
Just because the application is written in .NET doesn’t mean you can’t look to other languages for help. After all, only the application itself is deployed, not your CI process.
Ruby on Rails Migrations allows you to more effectively manage changes to your database schema among your team and makes it easier to roll out these changes to the various environments. And if the thought of learning a new language is off-putting for the sake of database management, consider how much time it takes to version your SQL scripts.
Third-party vendors can come to the rescue here as well, offering tools for comparing one database schema to the next, generating appropriate scripts, and even executing them.
Databases offer some unique challenges to testing. The volatile nature of business data doesn’t lend itself to the consistency required for automated testing. Be sure you give your database testing strategy some serious consideration because it’s easier to add pain to your process than remove it.
4.10. Summary
Phew! Things are getting hectic all of a sudden. We’ve covered a lot of ground this chapter and we finally started looking at some actual code.
As usual, we started off examining some pain points of working on a project that doesn’t have any automated tests. From there we looked at how automated tests can vastly improve the confidence everyone feels in the codebase. By writing tests to expose bugs before we fix them, for example, we close the feedback loop between the time a bug is written and the time it’s discovered.
We talked quite a bit about unit tests, differentiating between state-based tests and interaction-based tests. The latter often require the use of mock or fake objects, but they often require less configuration in order to focus on the specific unit being tested. We also recommended separating your unit tests from your integration tests to avoid potential problems in the future.
After that, we examined any existing test projects that you may have had in your project and what to do with any invalid tests in them (refactor them, delete them, or ignore them). Then it was on to the automated build process and how to incorporate the test project into it.
For those applications with no existing test projects, we talked about how to create one and integrate it into the build process, which is essentially the same process as for existing test projects.
Then it was on to writing new tests. Again, it isn’t necessary to write tests on existing code until you’re addressing a bug or adding a feature that involves the code. At that time, you should write the test assuming the code does what you want it to do rather than verify that it acts like it currently does.
We talked briefly about unit tests for new code before moving on to integration tests with respect to brownfield applications. Then it was on to user interface testing before we closed with some thoughts about how to manage the database during your test process.
We now have a solid foundation. Our version control system is in place and the continuous integration process is humming along. Now that we’ve added a test project for unit and integration tests (and possibly user interface tests) and we have a process in place for writing tests going forward, our confidence is probably higher than it ever has been for our lowly brownfield application. And we haven’t even started writing any real code for it yet.
In the next chapter, it will be time to analyze the codebase to see if we can figure out which areas are in need of the most work. After that, we’ll talk about how to manage the defect-tracking process, and then it’ll be time to dive into the code!