Test at All Levels So Change Is Easy
Think about the kinds of testing your team might do:
-
Automated unit testing for the code to know that we can refactor easily.
-
Automated and exploratory story testing (acceptance testing) for a story to know that we have satisfied the story’s criteria.
-
Automated feature testing for feature sets to know that the feature set looks and feels as if we have a coherent view of it.
-
Automated smoke testing for a build to know if the build is any good.
-
Automated system testing for the entire system, from the API so we have the ability to quickly know if we broke anything.
-
System testing from the GUI, possibly with some automation, so we know the first interaction the user has works.
-
Exploratory testing at the system level so we find the interactions and potential problems we didn’t or couldn’t find with automation.
It sounds like a lot of testing, doesn’t it? However, the more tests the team has—and the more automated they can make those tests—the faster the team can test. A nice effect of all this testing is that the tests support changing the code when the requirements change.
One of the problems with all this testing is that the team might not know how to do it. If the team thinks about automating as it proceeds and building tests even before developing the product, it will find this testing much easier.
Automate as You Proceed
The more the team can automate, the easier it will be for everyone to complete their work. Aside from build and smoke-test automation, do consider asking the testers to automate little tests as they proceed.
Perfect Test Automation Doesn't Exist
by Sherry, QA Manager in a New-to-Agile Organization
Sherry
We were trying to find the "best" or "perfect" test automation tool. In the meantime, all the testers tested manually.
Well, it didn’t take too long before we realized the testers were the bottleneck in finishing features. We had so much testing to do—new and regression testing—that we almost fell into the trap of staggered iterations.
I stopped looking for the perfect test automation tool. Instead, I made a list of reasonable tools and asked people to experiment. Their job wasn’t to find the best tool. Their job was to test as quickly as possible to provide feedback to the developers.
Once they started automating, they actually found more problems. I hadn’t expected that. And problems that cropped up again and again? The testers gave the developers their tests. The developers were happy to use the testers’ tests to understand where they had gone wrong.
We decided to refactor our tests as we proceeded. That worked. Every few months, we need to assess our tests and look for duplicate tests—yes, we still have duplicates. But we have an organization and naming system that works for us. We don’t need the "best" test automation tool. We can refactor our way into good enough.
Use Testing to Drive Product Development
In the agile community, we have advocates for a variety of approaches for using testing to drive product development:
-
Test-driven development (TDD) is when a developer writes a test that fails (red), creates just enough code to make it pass (green), and, when it’s time to add to that code, refactors to clean any messes.
-
Behavior-driven development (BDD) is for specification by example, where given a certain context, when the user behaves in a certain way, then we can expect certain results. See Define Acceptance Criteria for Each Story, for more details.
-
Acceptance test–driven development (ATDD) helps us create tests by thinking of the acceptance tests for the product.
Each serves a different purpose. While you can use all three kinds of testing to think about the product, teams often use TDD for developing and refactoring with unit tests as they proceed, BDD for feature or feature-set testing with examples of behavior, and ATDD for system testing with examples of how the user might use the system.
When the entire team collaborates to write stories with acceptance criteria, as discussed in Create or Refine the Stories as Preparation for Future Work, the team can create examples that guide its testing. Often, the testers will see devious, strange, and horrible possibilities for the story. When the entire team starts to think about “what can go wrong,” the product owner might see ways to create several stories.
Why test first? When we think about the tests first, and what would make a test pass or fail, we think about more possibilities for the product’s design and use. Our thinking is richer. That means we can question and refine the requirements.
For many years, requirements problems have caused problems in the code and the product (see Estimating Software Costs [Jon98]). Anything that helps teams clarify and refine requirements will have a payoff in the future of the project. (See Measure Your Cost to Fix a Defect.)
Here’s the real value of all this testing: one of the big problems with changes to requirements is knowing whether or how the changes will affect the rest of the product. Tests provide support for changes. The team has much more confidence in its code and the changes to that code when using tests as support.
The team can know what “done” means at several levels: a given story, the entire feature set, and the project.
When the product owner realizes the team needs to change a feature or a feature set, the team and the product owner can have a deep conversation. When the team members ask for examples for ATDD and BDD kinds of testing, the product owner clarifies the new actions the product owner expects.
I use a form of ATDD and BDD development to write my articles and books. I find using those approaches clarifies my thinking and therefore my delivery.
