Similar to typical SCM usage, the primary use of source control on test teams is for tracking changes to code written for test tools and test automation. Entire teams, and sometimes teams spanning all of Microsoft, share test tools. When more teams adopt a tool, it becomes more important to track changes so that any bugs or unexpected behavior caused by the change can be more easily identified. This usage is equivalent to SCM usage in development environments, with the fundamental difference being that the “customer” of the tool is another tester or test team in the company.

It is also beneficial to track changes in test automation or test cases. In addition to tracking changes in tests over the lifetime of a product, with SCM systems teams can create and re-create the state of the entire body of source code and related documents at any arbitrarily selected point in time. The most common use of this is to generate snapshots of the system from key milestones in the product life cycle. A common example is creating a snapshot of all product code and test code at the moment the product was released to manufacturing. By having a copy of the tests used to ship the product, sustained engineering teams can make changes to the product code with confidence. This is the same concept adopted by developers who use a suite of unit tests to help ensure that any changes they make to a module do not cause a regression. Sustained engineering teams use the tests to help them assess risk of a code change causing an unforeseen bug in another part of the system.

The Hocus-Focus comic strip by Henry Boltinoff ran in the newspaper my family received while I was growing up. If you’re not familiar with the comic, it was a simple two-panel strip where the two panels looked identical, but the text in between the panels asked the reader to “Find at least six differences in details between panels.” It usually took me only a few minutes to find all six differences, but my strategy was consistent. I cycled through each object or feature in the first picture and compared its shape, size, and other attributes with the same object in the second picture. Sometimes the difference was in part of the object, and sometimes I had to look at an entire area at once to see the difference. Over time, I got better and better (faster and more accurate) until I reached a point where I could spot all six in a few seconds every time.

Similarly, as a software tester, I use a SCM system and diff tools (utilities that show the differences between two files) to isolate coding errors. Figure 12-1 shows the comparison of two files using a diff tool. Because the SCM system records every change made to the product source code, when a tester discovers a bug while testing a scenario that previously worked, he often turns to the SCM system to help isolate the problem. Source control can show all changes made to a file, module, feature, or entire application between any points in time. If a tester discovers a bug that didn’t occur on a test pass two weeks earlier, she can easily identify all source code changes and examine the differences to investigate which change might have caused the error. Note that on some teams, developers can be the team members who go through this process. (See the sidebar titled "Who wears what hat?" that follows for more information on this topic.)

Figure 12-1. Comparing files in Visual Studio.

Monitoring changes isn’t unique to source code. It is also important to track changes made to specifications and other documents. Many applications can track the changes made to a document. The chapters in this book, for example, pass from authors to reviewers to editors and back again. At each stage, a reviewer might make several edits and suggestions. All changes are tracked within Microsoft Office Word so that each of the authors can track progress as well as document and refer to the discussions leading up to wording or content decisions.

Word can also compare two documents even if the reviewer forgets to activate the Track Changes feature, as shown in Figure 12-2. This is convenient for checking the changes made between two versions of a document and for reviewing the changes, for example, when reviewing edits made simultaneously by several reviewers.

Figure 12-2. Comparing documents in Microsoft Word.

Sometimes when I’m playing hocus focus with source code changes I can see the change, but I have no idea why the change was made.

===================================================================
--- math.cs;8  (server)  5/2/2008 5:24 PM
+++ math.cs;9  (server)  5/6/2008 7:25 PM
*************** 
*** 20,26 ****
              }
              else
              {
!                return value;
              }
--- 20,26 ----
              }
              else
              {
!                return value * 2;
              }
===================================================================

In the preceding code, it is obvious what this change does—it changes the returned value by a factor of 2! But why was this change made? Source control systems give testers a few key pieces of information in their detective work. One useful type of information is access to the names of each developer who has changed the code along with exact descriptions and records of the code the individual has changed. If you know who made a change that you don’t understand, you can walk down the hall, call, or e-mail that person and ask him or her about it.

Tracking down a developer for quick communication is often a great solution for understanding recent changes or changes made to a current version of a product, but what do you do if the developer has gone home, has left the group, or has left the company? The source control system often houses many points of highly relevant data, including developer comments and a bug number or link to the issue fixed by the change in the bug management system. When code is committed (a working copy is merged into the master source control system), the author of the change is generally required to fill in several fields in a submission form including the bug fixed by the change (or in the case of new code, an ID associated with the particular feature). Other fields might include such items as the name of the code reviewer and a short description of the details of the change, as shown in Figure 12-3. All of these items are clues that aid testers and developers alike in bug investigations.

Figure 12-3. Check-in forms in Visual Studio Team System.

The use of source control among test teams at Microsoft has evolved and grown with the company. In my early days at Microsoft, source control for the test teams I worked on was quite informal to say the least! Most of the test teams used source control for their test code or data files, but each part of the team used a separate source control server. The fact that we used source control on managed servers ensured that the test source code was backed up and revisions were tracked.

As long as you didn’t want to share or view source from another team, there were no problems with this system. On my team, it was the tester’s responsibility to compile the test source code, and then copy the resulting binaries to a common share where other testers or the automation system could find them when they were needed. This system worked most of the time, but errors occurred occasionally when a copy failed or someone accidentally deleted a file.

Over time, more teams began to consolidate test source into single servers and systems for their entire team. The structure and layout became more formal and far less ad hoc. These days, most teams store test source code next to product source code on the same servers and systems, as shown in Figure 12-4. Then, a build lab (a single person or small team dedicated to creating daily builds for the team) builds product code and test code daily and propagates the test binaries to servers automatically.

Figure 12-4. Typical layout of product and test code in source control.

A source control layout such as the one shown in the figure has several advantages. One of the most important features is that code is easy to find. If a developer wants to run some of the test team’s tests, he knows where to go. Similarly, if a tester wants to understand a bit more about the implementation of the code she is testing, she can find the product code easily. It also makes it much easier for the build team to investigate any build errors caused by inconsistencies between product and test code. Finally, if the build team is building both product and test code as part of the same process, it is simpler for the team to add consistent version information to the test code and product code. Consistent versioning allows for easier management of test binaries and product binaries, and can facilitate better regression analysis.

For most teams, the entire product is compiled from source code at least once a day. Microsoft has performed daily builds for years, and continuous integration (continuous builds along with frequent code check-in) is strongly supported by the Agile community. The build process includes compiling the source (transforming the source code to binary format), linking (the act of combining binaries), and any other steps required to make the application usable by the team such as building the setup program and deploying the build to a release server.

Note

The Windows Live build lab creates more than 6,000 builds every week.

More often than not, the daily build process also includes a suite of smoke tests. A smoke test is a brief test to make sure the basic functionality of the application works. It’s similar to driving a borrowed car around the block to check for obvious faults before driving all the way across town. Test teams usually run a suite of smoke tests. Most often, these are known as build acceptance tests (BATs) or build verification tests (BVTs). Some definitions of these terms denote BATs to be a smaller suite of tests than BVTs, but in most cases, the terms are interchangeable. A good set of BVTs ensures that the daily build is usable for testing. Table 12-2 lists several BVT attributes.

An example of a BVT suite for a simple text editor such as Windows Notepad could include the following:

A daily (or more frequent) build and BVT process reduces the chance of errors caused by large integrations or sweeping changes. Keeping the product in a state where it will build and where it will run every day is critical to a healthy software organization. Test teams at Microsoft have the same need for daily builds as product teams. For this reason, source code for automated tests and test tools are also built daily, typically as part of the same process used to build the product code.

Stopping the Breaks

I don’t know anyone who has worked with daily builds who hasn’t experienced a build break on their team. At Microsoft, we use several techniques to minimize the number and impact of build errors. Two of the most popular and effective techniques are rolling builds and check-in systems.

In its simplest form, a rolling build is an automatic continuous build of the product based on the most current source code. Several builds might occur in one day, and build errors are discovered sooner. The fundamental steps in a rolling build system include these:

• A clean build environment

• Automatic synchronization to the most current source

• Full build of system

• Automatic notification of errors (or success)

The easiest way to implement a rolling build is with a simple Windows command script (cmd) file. Scripting tools such as Sed, Awk, and Perl are also commonly used. An example of what a rolling build script might look like is in Example 12-1.

Example 12-1. Simple Rolling Build Windows Command Script

rem RollingBuild.cmd
rem sync, build, and report errors

rem The following two commands record the latest change number
rem and obtain the latest source changes

:BEGINBUILD
rem clean up the build environment
call cleanbuild.cmd

changes -latest
sync -all

rem build.cmd is the wrapper script used for building the entire product
call build.cmd

rem if ANY part of the build fails, a build.err file will exist
rem notify the team of the rolling build status
IF EXIST "build.err" (
  call reporterror.cmd
) ELSE (
  call reportsuccess.cmd
)

goto BEGINBUILD

Some Microsoft teams also include some or all BVTs in a rolling build and automatically report the results to the team. Teams that conduct several rolling builds each day often pick one—for example, the last successful build before 13:00—and conduct additional testing and configuration on that build before releasing it to the team as the daily build.

Another method of preventing build breaks and increasing code quality is the use of a check-in system. Years ago, when developers had changes to check in to SCM, they would check the code directly into the main source tree. Any mistakes they made—either syntax errors or bugs—would immediately affect every build. You can get away with a system like this for small projects, but for larger software projects, a staged check-in can be quite beneficial.

Figure 12-5 shows the basic architecture of a check-in system. Instead of checking the code directly into the main SCM, when programmers are ready to submit changes, they submit them to an interim system. At a minimum, the interim computer verifies that the code builds correctly on at least one platform, and then submits the code on behalf of the programmer to the main source control system. Most such systems build for multiple configurations and targets—something that is nearly impossible for a single developer to do on her own.

Figure 12-5. Check-in system architecture.

The interim system (sometimes referred to as a gauntlet or gatekeeper) will also typically run a selection of automated tests against the change to watch for regressions. Depending on the implementation of the system, the set of tests can be static, can be selectable by the programmer at the time of submission, or is created dynamically based on which parts of the code are changed.

Any selection of pre- or postbuild tests can be added at any time to the system. To the developer, it feels the same as checking directly into main (the main source tree), but a substantial number of bugs are found before they ever get a chance to cause problems for the entire team.

A number of different tools are available for analysis of native code (that is, code written in C or C++). Traditional tools include commercial products such as PC-Lint,^[2] KlocWork,^[3] and Coverity,^[4] as well as the static Code Analyzer included with Visual Studio Team System.

Every team at Microsoft uses code analysis tools. Since 2001, the primary tool used at Microsoft for native code analysis is a tool named PREfast. This is the same tool available for native code analysis in Visual Studio Team System.

PREfast scans source code, one function at a time, and looks for coding patterns and incorrect code usage that can indicate a programming error. When PREfast finds an error, it displays a defect warning and provides the line number of the offending source code, as shown in Figure 12-6.

Figure 12-6. Static analysis warnings in Visual Studio using PREfast.

FxCop is an application that analyzes managed code and reports a variety of information such as possible design, localization, performance, and security improvements, as shown in Figure 12-7. In addition to detecting many common coding errors, FxCop detects many violations of programming and design rules (as described in the Microsoft Design Guidelines for Class Library Developers). Anyone creating applications in managed code will find the tool beneficial.

FxCop is available as a stand-alone tool or integrated into Visual Studio. Testers can use the stand-alone tool to examine a managed binary for errors, but most testers and developers use the integrated version of the tool for analysis.

Figure 12-7. FxCop analysis tool.

Note

The Code Analysis Team Blog at http://blogs.msdn.com/fxcop contains a wealth of information on analysis tools in Visual Studio Team System.

FxCop and PREfast are both powerful tools that should be used by every software team. Static analysis tools are great for finding certain classes of errors well before the test team sees the code, but they do not replace testing. It is certainly possible (and probable) to write code that is free of code analysis–detected defects that is still filled with bugs. However, fixing these bugs earlier does give the test team more time—and a much better chance—to find a huge number of more critical bugs throughout the entire product cycle.

Many types of errors are specific to test automation. One such area is environmental sensitivity. This refers to the reliance of tests on noninvariant environmental factors. This issue can reveal itself in a variety of forms, which are best represented by example. Consider a test of a user control called a FileSaveWidget, which accepts an input file path and saves data to the specified file. A simple test of this component is listed in sample code for a FileSaveWidget test here.

1 void TestFileSaveWidget()
2 {
3    // this file contains the expected output of the file save widget
4    string baselineFilePath = @"\test-serverTestDataaseline.txt";
5
6    // the local path for the save file
7    string outputFilePath = @"D:datafile.txt";
8
9    FileSaveWidget widget = new FileSaveWidget();
10   widget.SetDataFile(outputFilePath);
11   widget.Save();
12   try
13   {
14     VerifyDataFile(baselineFilePath, outputFilePath);
15     WriteTestResult("PASS");
16   }
17   catch (Exception e)
18   {
19       String errorMessage = e.Message;
20       if (errorMessage.Contains("File not found"))
21       {
22           WriteTestResult("FAIL");
23       }
24   }
25 }

The code looks straightforward enough, but there are at least three issues in this example. On line 4, notice the use of a remote file share on a computer called test-server. There is no guarantee that this server will be visible when the test runs on a network or user account different from the one the tester originally intended. Line 7 contains a hard-coded reference to drive D. Perhaps all computers that this tester has encountered so far have writeable partitions on D, but this assumption will likely lead to failures later on (D could be a CD-ROM drive or simply nonexistent).

Line 19 hides a more subtle but important issue. The expected error message in the case that the save operation fails might indeed be “File not found”—if the test is running on an English-language operating system or English-language build of the product. However, a Spanish localized build might return “Archivo no encontró” in this case, which would not be caught by this test. Many test teams at Microsoft use scanning tools to find issues that can cause problems with test code. Sample output from a test code–scanning tool is in Table 12-3.

Analysis tools often go beyond analyzing code and look for errors in the test cases or almost anyplace else where testers might make errors. Some examples of the biggest improvements to test code as a result of analysis such as this include the following:

The bigger value added by analysis is that it preempts issues from occurring by flagging them on a daily basis. Every day, testers get feedback on the quality of their tests and test cases, and they use the feedback to improve the quality of their tests and, ultimately, the quality of the product.

These errors might not be important for production code, but they are critical for test code reliability and maintainability. Error-prone test code is one of the biggest concerns I hear from test managers throughout the industry. Even at Microsoft, where many of our SDETs are top-notch programmers, we rely on these tools to help us create tests that can run reliably and report accurate results for a decade or more and through thousands of executions. By now it should be clear that testers at Microsoft commonly use many traditional development tools. What this really means is that at Microsoft, source code is treated the same as product code regardless of whether the code ships or not.

Most of the previously discussed tools are ones that almost every tester uses. Countless others solve big problems for smaller groups of people. Screen recorders, file parsers, add-ons for automation tools, and even the font display tool shown in Chapter 10 are all examples of software programs written to solve specific testing problems.

In addition to tools, testers customarily share libraries (reusable functionality that can be added to any application) in their team so that everyone on the team can solve similar problems using a consistent, well-tested solution. For example, the Office team has shared libraries that are used to help automated tests access the window controls common to all Office applications, and the Windows Mobile team has shared libraries to simulate cellular data. These are both areas where a common solution helps the entire test team do their jobs better.

Microsoft employees love to create tools. Thousands of tools ranging from test helper libraries to Outlook add-ins to productivity enhancers are available in an easy-to-access repository. Engineers love to use software that helps them do their job better, and many employees share their applications with their teams across the entire company.

Note

The Microsoft internal tool repository contains nearly 5,000 different tools written by Microsoft employees.

Employees can search the tools in this repository (the ToolBox), as shown in Figure 12-8, and they can subscribe to RSS feeds to notify them of new tool submissions. Each tool entry lists information about the tool and the name of the tool owner in case more information is needed. Finally, every tool can be rated and comments can be added to aid future toolbox browsers in their tool-choosing decisions.

Figure 12-8. The Microsoft ToolBox.