As the story goes, one day a villager was walking by the river that ran next to his village and saw a man drowning in the river. He swam into the river and brought the man to safety. Before he could get his breath, he saw another man drowning, so he yelled for help and went back into the river for another rescue mission. More and more drowning men appeared in the river, and more and more villagers were called upon to come help in the rescue efforts. In the midst of the chaos, one man began walking away along a trail up the river. One of the villagers called to him and asked, “Where are you going? We need your help.” He said, “I’m going to find out who is throwing all of these people into the river.”

Moving quality upstream is a commonly heard term in software testing circles, yet it’s not done enough in the majority of contexts. Most software used today is too complex, too big, and too expensive to improve quality from testing alone. Almost every computer science book I own shows a graph illustrating the cost of a bug over time, yet the software industry still relies on one of the final stages of software engineering—testing—for a substantial portion of product quality. Table 16-1, taken from Code Complete,^[3] is one such example of cost increase estimates dependent on the phase the bug was introduced.

The data presented by McConnell in this table explains that a bug introduced in the requirements phase that might cost $100 dollars to fix if found immediately will cost 10 times as much to fix if not discovered until the system test phase, or as much as 100 times as much if detected post-release. Bugs fixed close to when they are introduced are generally easier to fix. As bugs age in the system, the cost can increase as the developers have to reacquaint themselves with the code to fix the bug or as dependencies to the code in the area surrounding the bug introduce additional complexity and risk to the fix.

Joseph Juran, who is widely known for his contributions to quality thinking, was recognized for adding a human dimension to quality issues. Juran felt that cultural resistance was the root cause of quality issues.

A culture is shared beliefs, values, attitudes, institutions, and behavior patterns that characterize the members of a community or organization. A quality culture is one in which members of the community share values, attitudes, and beliefs regarding quality and take on their daily work with these attributes as a driving force. If quality is not ingrained into the culture in this way, quality practices will be viewed as pieces of the engineering puzzle that can be done “later.”

Unfortunately, there is no such thing as “just-in-time” quality, and expecting quality to happen at the end of the product cycle is foolish. If quality is not at the forefront of engineering processes, it is impossible to reach acceptable levels of quality in the end. At a recent conference, Watts Humphrey rephrased this long-standing problem as, “If you want to get a high quality product out of test, you have to put a high quality product into test.” The hard problem that we often avoid is, how can we get higher quality products into test?

Machiavelli wrote, “In the beginning of a malady it is easy to cure but difficult to detect, but in the course of time, not having been either detected or treated in the beginning, it becomes easy to detect, but difficult to cure.”^[4] The parallels of this statement in software are obvious: Early detection and prevention are the keys to avoiding problems that might be difficult to “cure” later.

The most common method for improving quality at Microsoft (and in the rest of the software industry as well) is testing. Testing is the means to unearth defects in software before customers can find them. In fact, with some of the best software testers in the industry, and the common practice of involving test early in the product cycle, Microsoft is extremely effective at finding bugs early in the product cycle. As good as that is, the prevailing approach is still to test quality into the product.

The word testing is often synonymous with quality assurance; however, these are two different acts of engineering. Testing is a reactive approach—finding defects that are latent in the product using various detection and investigative techniques, whereas quality assurance is a proactive approach—building an environment where bug prevention and a quality culture are inherent in the development life cycle. Aspects of quality assurance are present in all disciplines but are most prevalent in the test discipline.

The Engineering Excellence group at Microsoft designed and regularly delivers a course for senior testers at Microsoft where a large amount of the material covers quality assurance techniques rather than testing techniques. In many cases, a senior test role at Microsoft is much more like a quality assurance role than a test role, where the primary role is to improve quality practices affecting all engineering disciplines across a group or division.

Many years ago, when I would ask the question, "Who Owns Quality?" the answer would nearly always be, “The test team owns quality.” Today, when I ask this question, the answer is customarily “Everyone owns quality.” Although this might be a better answer to some, W. Mark Manduke of SEI has written: “When quality is declared to be everyone’s responsibility, no one is truly designated to be responsible for it, and quality issues fade into the chaos of the crisis du jour.” He concludes that “when management truly commits to a quality culture, everyone will, indeed, be responsible for quality.”^[5]

A system where everyone truly owns quality requires a culture of quality. Without such a culture, all teams will make sacrifices against quality. Development teams might skip code reviews to save time, program management might cut corners on a specification or fudge a definition of “done,” and test teams might change their goals on test pass or coverage rates deep in the product cycle. Despite many efforts to put quality assurance processes in place, it is a common practice among engineering teams to make exceptions in quality practices to meet deadlines or other goals. Although it’s certainly important to be flexible to meet ship dates or other deadlines, quality often suffers because of a lack of a true quality owner.

Entire test teams might own facets of quality assurance, but they are rarely in the best position to champion or influence the adoption of a quality culture. Senior managers could be the quality champion, but their focus is justly on the business of managing the team, shipping the product, and running a successful business. Although they might have quality goals in mind, they are rarely the champion for a culture of quality. Management leadership teams (typically the organization leaders of Development, Test, and Program Management) bear the weight of quality ownership for most teams. These leaders own and drive the engineering processes for the team and are in the prime organizational position for evaluating, assessing, and implementing quality-based engineering practices. Unfortunately, it seems that quality software and quality software engineering practices are rarely their chief concerns throughout any product engineering cycle.

Senior management support for a quality culture isn’t entirely enough. In a quality culture, every employee can have an impact on quality. Many of the most important quality improvements in manufacturing have come from suggestions by the workers. In the auto industry, for example, the average Japanese autoworker provides 28 suggestions per year, and 80 percent of those suggestions are implemented.^[6]

Ideally, Microsoft engineers from all disciplines are making suggestions for improvement. Where a team does not have a culture of quality, the suggestions are few and precious few of those suggestions are implemented. Cultural apathy for quality will then lead to other challenges with passion and commitment among team members.

Cost of quality is a term that is widely used but just as widely misunderstood. The cost of quality isn’t the price of creating a quality product or service. It’s the cost of not creating a quality product or service.

Every time work must be redone, the cost of quality increases. Obvious examples include the following:

In other words, any cost that would not have happened if quality were perfect contributes to the cost of quality. This is also known as cost of poor quality (COPQ).

In Quality Is Free, Phillip Crosby divides quality costs into three categories: appraisal, preventative, and failure. Appraisal costs include salaries (including the cost of paying testers) and costs to release. Preventative costs are expenditures associated with implementing and maintaining preventative techniques. Failure costs are the cost of rework (or the COPQ as stated previously). We rarely measure preventative costs, and we even more rarely reward them. Instead, we prefer to concentrate on heroics—working all night to fix the last few bugs, or investigating a workaround for a serious issue at the last minute.

Consider this quote from The Persistence of Firefighting in Product Development^[7]

This quote might seem familiar to many readers. We don’t need heroics; we need to prevent the need for them.

Some might ask if quality becomes deeply integrated in the design, and developers create far fewer bugs, what will the test team do? The truth is that finding bugs today might be too easy. In an organization where quality is inherent in the habits of everyone, the emphasis on the role of test moves away from finding massive numbers of defects to concentrating on emulating customer scenarios, peer review, and validation. There will still be bugs to find, but finding them will be a much bigger challenge than it is today. Testers will have the time to investigate complex integration scenarios and, unblocked by basic functionality issues, find a great number of the bugs that our customers find after release, removing much of the tremendous cost of hotfixes, updates, and service packs.

Others from the test role (as well as from other disciplines) can begin to function in a quality assurance role. These individuals will analyze and implement process improvement, defect prevention techniques, and infrastructure, and be the ambassadors for quality thinking and a quality culture in their organization. Growth and development of this role could be a key part of building and evangelizing a quality culture that spans the company.

More ideas on bug prevention, including root cause analysis and the need for a quality culture, are found in The Practical Guide to Defect Prevention (Microsoft Press, 2007), a book written by some of our testing colleagues at Microsoft.

Leadership teams exist for every engineering discipline at Microsoft, and the Microsoft Test Leadership Team (MSTLT) is one of the most active and successful of these leadership teams. The principal goal of this team is to support sharing of testing knowledge and practices between all testers across the company. This goal is reflected in the MSTLT’s mission (see the following sidebar) and in their accomplishments to date.

The MSTLT is made up of about 25 of the most senior test managers, directors, general managers, and VPs representing every product line at Microsoft. The selection process for representation for leadership team membership is critical to ensure its success. Membership selection is based on level of seniority, as well as nomination or approval from the TLT chair and the product line vice president. (See Chapter 2.) Without the proper makeup, leadership teams cannot speak for their community at large in a balanced and representative way. Additionally, they need the support of their organizational leadership to represent their organizations adequately and fairly.

Grant George was (and remains) the first chair of the test leadership team. He took on the role of chair of the MSTLT in August of 2003 and fostered growth of the team through its initial period. In 2004, the Engineering Excellence organization began to support these virtual teams, and as such, the MSTLT chair became a close partner for engineering excellence. This virtual merger also allowed the MSTLT to have a very active role in reviewing and proving feedback on tester training, engineering excellence awards, and the career development road map for testers (developed jointly by the MSTLT and HR).

Every year, the Microsoft Test Leadership Team (MSTLT), in conjunction with the Engineering Excellence team, selects three to six areas where significant research or investigation related to the growth or improvement of testing at Microsoft is necessary. Past examples have included work on career paths for senior testers, automation sharing, lab management, and career stage profiles for testers.

The MSTLT meets monthly. The meeting agendas vary from month to month but focus on a few main areas. These include the following:

The Test Architect Group (TAG) is composed of senior nonmanagement leaders from the test discipline. When Soma Somasegar, then a Director of Test in the Windows division, created the Test Architect title, he encouraged the initial Test Architects to meet regularly and discuss the challenges and successes they were having on their respective teams. The group started with only six Test Architects, mostly from the Windows division, but has since grown to more than 40 Test Architects spanning every major division at Microsoft. Originally, only testers with the Test Architect title were members of the group, but as Microsoft has moved more toward standard titles (for example, SDET, Senior SDET, and Principal SDET, as discussed in Chapter 2), the group now includes all senior testers who are functioning in a test architect role regardless of their address book title.

Nearly 10 years after the establishment of the group, the TAG members still meet nearly every week. Test architects are primarily dedicated to addressing technical issues on their own teams, but this group has found numerous benefits in meeting regularly to share ideas and best practices on testing and quality issues at Microsoft. In fact, many of the meeting agendas center on the original mission of sharing current challenges and success stories.

The value of having Microsoft’s most senior testers regularly review, brainstorm, and dissect solutions for complex test problems is immeasurable. In recent years, TAG has become something of a sounding board for new thinking, new methods, or new tools in testing. Presentations and demonstrations of ideas and implementations from test groups spanning every Microsoft division fill many of the meeting agendas. The value and depth of the feedback that the TAG provides is respected and sought after. A few meetings a year are reserved for “TAG business,” which includes discussions about company-wide initiatives driven by TAG and other projects where TAG is a significant contributor (such as the MSDN Tester Center).

Perhaps the largest benefit of the regular meetings is in the value of networking. The extensive peer discussions and the view into the variety of work done across the company that is presented give TAG members much of the knowledge and information they need to make strategic decisions that affect the entire company.

Another way that the Test Architect Group provides value to Microsoft is by being open and available for informal discussions on any topic with anyone. E-mail discussions and ad hoc meeting requests are a start, but one of the most successful initiatives in this area is the "Have lunch with a Test Architect" program. Every month, Test Architects across the company have dozens of lunch chats with testers of all levels of seniority and in a variety of product groups.

Note

Microsoft Research has an entire group that studies software testing and verification (along with program analysis and other software measurement). Descriptions of their projects can be found at http://research.microsoft.com/srr.

Sharing aligns with the EE goal of sharing best practices. In the case of Test Excellence, sharing incorporates practices, tools, and experiences.

Note

More than 20 Test Talks (presentations on effective test practices) are held every year. Test Talks are open to all testers at Microsoft (including live Webcasts and recordings for testers outside of Redmond).

The Test Excellence team functions as facilitators and experts for the overall test community. They use these roles to establish and maintain connections with testers. In fact, the tagline for the Test Excellence team is We connect the dots... to quality, and this brand statement drives much of their work. Some of the ways this team helps testers are facilitation, answers, and connections:

Another key value of Test Excellence is simply in communicating what they know and discover. There is tremendous value in sharing relevant information in a large community. Some of the communication coming from the Test Excellence team includes the following:

Working together

The Engineering Excellence group recently remodeled their office space to include team rooms for each of the discipline-focused “Excellence” groups. Microsoft’s policy of supplying individual offices to engineers is well known, and Peopleware,^[8] DeMarco and Lister’s seminal text on productivity among software engineers, explains how and why private offices for engineers increase productivity.

Why would Microsoft change their ways and make a “cube farm”? In the case of Engineering Excellence, the decision was almost purely experimental. Agile software development practitioners talk about the necessity for collaboration that a team room provides, but I’ve also met many people who would rather work in a treehouse than in a shared office. The work that our team performs does have some collaborative aspects, but certainly can’t be considered as collaborative as a software product. Nevertheless, in May of 2006, just a few days before the remodel was to begin, we packed and moved to a temporary office; then, moved back into our new team room a few months later.

I don’t know if any of us knew completely what to expect, but overall, I think we were all pleasantly surprised. I think the biggest benefit is that we have nearly 1,000 square feet of space dispersed between six of us. Given the nature of our jobs, it seems even bigger because, outside of team meetings, it’s extremely rare that we are all in the office at the same time. We also have flexibility—if we were to add one more person to the team, we could make some small modifications and find room for that person easily. We have couches, chairs, a ceiling-mounted projector, an Xbox, and an espresso machine. Almost every wall is usable as a whiteboard, and we’ve hung whiteboards on most of those that aren’t. Most of the workspaces sit along a wall of windows. Figure 16-1 shows the current configuration of our room.

Figure 16-1. Test Excellence team room.

The room can get “busy” sometimes, and we all have had to resort to headphones occasionally to cut down on the distractions, but most of the time we are all happy with the room. We have our own spaces, but we can talk face to face when we need to. Our arrangement is a bit of a compromise between a “cube farm” and an individual office.

We have six in the room now, and it would probably work with a few more, but I think the team room concept would fail as team size approached double digits. The biggest benefit of the shared space is that we are all more aware than ever of the work that our teammates are doing—and we have a fun, modern place to work.

Perhaps the most prevalent indicator of the success of our new room was when I took over the leadership of the Test Excellence team. I chose to keep my desk in the team room. I also have an office that we all use for 1:1 (pronounced one-on-one) meetings, interviews, and other private meetings, but I enjoy spending most of my time working in what we have come to call our “test lounge.”

Perhaps the most critical role of the Test Excellence team is anticipating the future needs of testers at Microsoft and being proactive in defining the future vision of software testing. This vision is the basis for yearly planning and Test Excellence initiatives. The purpose of the vision, like most good visions, is to give direction on the discipline and to guide the work to be done.

One of the most unique roles at Microsoft is that of the Director of Test Excellence. In some ways, the role is simply to be the manager in charge of the Test Excellence team, but it also has a tremendous amount of scope and opportunity to influence testers across the entire company.

Not surprisingly, it’s a role that all three authors of this book have held. Currently, the Director of Test role is tied to the Director of Test Excellence role, but originally it was just a senior test position responsible for advancement of the testing profession at Microsoft.

The following people have all held the Director of Test position:

Because this is a full-time position managing a small strategy team the role also functions as the connection between the TLT and the TAG with an emphasis on working closely with the chairs of the two organizations.

The Microsoft Test Leadership Team, Test Architect Group, and Test Excellence organizations fill a vital role in the development and maintenance of the test culture at Microsoft, as shown in Figure 16-2. In each of these organizations is a leadership position chartered to work for the discipline and across the teams. In the case of the TLT chair and the TAG chair, these are virtual teams and the chairmanship is an extra duty for the individual in that role. For the Director of Test Excellence it is a full-time position.

Figure 16-2. Three key leadership positions in the three test support organizations.