Chapter 1. The History and Value of Agile Software Development

This chapter provides the background of the Agile movement and compares and contrasts Agile to the more traditional Waterfall methodology. It describes the Agile Manifesto in detail and explores the key tenets of Agile, such as the following:

• Close collaboration between the programming team and business experts

• Face-to-face communication

• Frequent delivery of new, deployable business value

• Tight, self-organizing teams

• Reduced impact of changes in requirements

The 12 additional supporting principles of the Agile Manifesto are also introduced.

The first, and still the most popular, software development process is referred to as the Waterfall model. The Waterfall model advocates sequential phases of development in which each stage is completed before the next begins, with a focus on structure. For example, all software designs are completed before the coding phase begins. This methodology was introduced in the 1950s at a conference on software development methodology for SAGE, and it has continued to be the predominant software development methodology (Benington 1956; Larman and Basili 2003). Despite the popularity of Waterfall development, it has continued to be criticized in the field for being process heavy and unresponsive to the inevitable changes that arise during software development projects (McConnell 2004).

What is Agile development? What comprises Agile development methodology, and why is it getting the reputation as the superior approach to the Waterfall methodology? Agile is an overarching term that includes iterative approaches to software development that embrace the values of the Manifesto for Agile Software Development, which is discussed in more detail later in this chapter. Agile development methodologies are designed to flex and support the needs of the project and organization. Agile does not describe a specific approach, but instead offers a collection of tools and best practices that help development organizations focus on efficiency, collaboration, quality, and the creation of customer value. Agile development includes methodologies such as Extreme Programming, Scrum, and the Crystal Family; all of these versions of Agile development have their own nuances, but they share the goal of avoiding “a single pass sequential, document-driven, gated-step approach” (Larman and Basili 2003, p. 47). Table 1.1 compares Waterfall and Agile software development approaches.

Including usability specifications (the study of end user behavior and how these users will interact with the software) into iterative development processes was introduced in 2002, when iterative development was starting to gain popularity (Carroll and Rosson 2002). Carroll and Rosson (2002) state that usability designs should not be a static recommendation or design, but rather should evolve and be refined throughout the development process. Larman and Basili (2003) claim that although iterative development has been the new buzz in the IT industry in recent years, it is not a new concept. Researchers at IBM’s TJ Watson Research Center published the first research document describing a process that had the most similarities to today’s Agile or iterative development process (Zurcher and Randell 1968). This process was recommended to IBM executives in 1969 as “A model becomes the system” (Lehman and Belady 1985), but IBM did not start adopting the contemporary model of Agile development until the turn of the present century.

After much debate, the term “Agile” gained consensus from the group, and they proceeded to create the “Manifesto for Agile Software Development.” According to Williams and Cockburn (2003) and the practitioners who authored the Manifesto for Agile Software Development, Agile is a process for developing software that values

individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, responding to change over following a plan. (p. 39)

The practitioners were careful to use the term “over” in an effort to be less prescriptive and instead to make the manifesto more of a guide. They added the phrase “That is, while there is value in the items on the right, we value the items on the left more” to emphasize that there will be times when lighter approaches are not always realistic or possible, but they should always be the goal.

What does each of these values mean, and how do they influence organizations that are using Agile methodologies? Table 1.2 defines each value and gives an example of how each one can be used in an Agile organization.

After developing the list of initial values for Agile software development, the authors decided to add a list of principles that would help teams understand the priorities for Agile practices (see Figure 1.2). They again avoided getting too prescriptive on how to be Agile and emphasized the principles that should guide an Agile team.

As a result of this meeting, the Agile Alliance was formed, and the proponents of lightweight alternatives to Waterfall had a common platform to discuss their ideas. The Agile Alliance (http://www.agilealliance.org/) is a global nonprofit organization whose mission is to advance Agile development principles and practices. The Agile Alliance offers many important benefits to Agile practitioners, including participation in user groups, conference discounts, and volunteer opportunities.

Each year, the Agile Alliance holds a major conference that covers a wide range of Agile topics. Attendance and participation in the conference have grown extensively each year as a result of the widespread adoption of Agile.

Cayman Design’s management team decided to start using more Agile methodologies when they entered the mobile application business. Their competitors were using Agile, and they were able to deliver sites to the market faster, with increased customer satisfaction. Cayman Design knew that using Agile techniques with a global team would be a challenge, but they were committed to making it work.

Workers at Cayman Design are a mix of new hires and highly experienced professionals. The management team started down the Agile path by hiring a consultant to train the employees and provide recommendations on transitioning to Agile methodologies. They chose to implement Scrum, Kanban boards, and Extreme Programming. Communication with global team members is done through video conferences, e-mail, and instant messaging. Cayman Design averages six to eight Scrum teams across the e-commerce and mobile applications teams. You will get to know more about Cayman Design’s adoption of Agile approaches throughout the subsequent chapters in this book.

• Software development has traditionally been performed using the Waterfall methodology, which focuses on completing each phase of development before the next phase starts.

• Agile methods started gaining popularity for software development in the 1990s, and teams began to experiment with better ways to develop software.

• Waterfall development methods are sequential, prescriptive, and documentation intensive, but Agile methods promote iterative development, flexibility, and intuitive design.

• The Agile Manifesto is a declaration that was developed in 2001 to establish agreement on the values shared by Agile approaches and to advocate for a new way to think about developing software.

• Shortly after writing the Agile Manifesto, the authors created 12 additional principles to explain the basis for the values in the Agile Manifesto.

Mr. Martin has published dozens of articles in various trade journals, and is a regular speaker at international conferences and trade shows. He has authored and edited many books, including the following:

• Designing Object-Oriented C++ Applications Using the Booch Method

• Pattern Languages of Program Design 3

• More C++ Gems

• Extreme Programming in Practice

• Agile Software Development: Principles, Patterns, and Practices

• UML for Java Programmers

• Clean Code

• The Clean Coder

A leader in the industry of software development, Mr. Martin served three years as the editor-in-chief of the C++ Report, and was the first chairman of the Agile Alliance.

Kristin and Sondra: Alistair Cockburn states in his book Agile Software Development: The Cooperative Game that you called the meeting for the Agile Manifesto. What gave you the idea to call the meeting, and how did you decide who should be invited?

Bob: In the summer of 2000, Kent Beck called a meeting of people in the Extreme Programming (XP) community. I think he called it XP Leadership or something. The attendees included myself, Ward Cunningham, Martin Fowler, Ken Auer, and many others from the Design Patterns community. The goal was to set the future direction of XP. At that meeting, I stated that I thought a manifesto and an organization would be a good idea as a way to amplify our message. Martin Fowler agreed, and he and I decided to meet in Chicago to discuss the event and the invitation list.

That meeting took place in the fall of 2000. He and I made a list of the people we thought would be important to invite. Specifically, we invited folks who had contributed to the idea of lightweight processes. Indeed, as I recall, we called the event the Lightweight Process Summit.

One of the invitees was Alistair Cockburn. Apparently he had a similar idea, and our invitation came just a few days before he was going to send his. At least that’s how I remember it. He volunteered to do the legwork on finding a venue. He also merged his invitation list with ours, which was a great boon.

Looking back on it, I consider it quite remarkable that so many of the invitees actually came to the event. Apparently the stars were in alignment.

I kicked the meeting off by suggesting we write a manifesto. After that, I was just one of the minor contributors.

Kristin and Sondra: Do you think the Agile Manifesto will need to be updated or changed as Agile development continues to gain greater adoption?

Bob: No, I think it is a statement that points in a direction; and I don’t think that direction has changed, or will change. I don’t believe the Agile Manifesto is, or should be, a living document. I think it’s a milestone on a path, not the path itself.

There have been other manifestos since, which have amplified or extended the direction that the Agile Manifesto began. The software craftsmanship manifesto [http://manifesto.softwarecraftsmanship.org/] comes to mind.

Kristin and Sondra: Do you think Agile methodologies are better than the traditional Waterfall methodology in all cases, or are there situations where Waterfall is more beneficial?

Bob: Agile methods are better in every case, because Agile methods are human methods. We are taught Agile methods when we are in third grade and our teacher tells us to write a story. She helps us with an outline, then a first draft, and a second draft, etc. We learn, from a very early age, that anything worth doing well must be done iteratively through a process of successive refinement.

Kristin and Sondra: Was there a specific event or series of events that led you to recognize the importance of writing clean code and refactoring?

Bob: There was no specific event. Instead, it was a long process of learning and watching and thinking. A critical moment in that process was when I learned TDD—Test-Driven Development [which is described in detail in Chapter 7, “Testing, Quality, and Integration”]—from Kent Beck. As I practiced this critical discipline, it became clear to me that if you have a suite of tests that you trust, then code becomes simple and easy to change. Cleaning code loses the risk that is usually associated with it. Refactoring becomes something you do naturally every day, every hour.

Once you understand that, then the importance of refactoring and cleaning code becomes obvious. When you are afraid to clean, you accept the mess as necessary. When you understand that cleaning can be easy and safe, then the mess becomes unacceptable.

Kristin and Sondra: In your opinion, what is the single most important contribution that Agile development has made to the field of software engineering?

Bob: Agile broke the Waterfall. That, in and of itself, is hugely important because it makes software development into a human activity.

Now I’m waiting for the next revolution, the craftsmanship revolution. We’ve learned that we must iterate and build our systems through successive refinement. The next thing we have to learn is that we don’t really have to make a mess in order to go fast. Our next lesson is “The only way to go fast is to go well.”

Beck, K., Beedle, M., et al. (2001). Manifesto for Agile software development. http://Agilemanifesto.org/.

Benington, H. D. (1956). Production of large computer programs. Symposium on Advanced Programming Methods for Digital Computers. Washington, DC: Office of Naval Research, Dept. of the Navy.

Bittner, K. (2004). Driving iterative development with use cases. IBM developerWorks, http://www.ibm.com/developerworks/rational/library/4029.html.

Carroll, J. M., and Rosson, M. B. (2002). Usability engineering. San Diego, CA: Academic Press.

Doshi, C., Doshi, D. (2009). A peek into an Agile infected culture. Presentation at the Agile 2009 conference, Chicago.

Jacko, J., Düchting, M., et al. (2007). Incorporating user centered requirement engineering into Agile software development. Human-Computer Interaction. Interaction Design and Usability, 4550: 58–67.

Larman, C. (2004). Agile and iterative development: A manager’s guide. Boston: Addison-Wesley.

Larman, C., and Basili, V. R. (2003). Iterative and incremental development: A brief history. Computer, 36(6): 47–56.

Lehman, M. M., and Belady, L. A. (1985). Internal IBM report, 1969. Program evolution: Processes of software change. San Diego, CA: Academic Press Professional.

McConnell, S. (2004). Code complete: A practical handbook of software construction. Redmond, WA: Microsoft Press.

Nerur, S., Mahapatra, R., et al. (2005). Challenges of migrating to Agile methodologies. Communications of the ACM, 48(5): 72–78.

Sanders, E. (2002). From user-centered to participatory design approaches. In J. Frascara, ed., Design and the social sciences: Making connections. New York: Taylor and Francis.

Whiteside, J., Bennett, J., et al. (1988). Usability engineering: Our experience and evolution. In M. Helander, ed., Handbook of human-computer interaction. Amsterdam: North Holland, 791–817.

Williams, L., and Cockburn, A. (2003). Guest editors’ introduction: Agile software development: It’s about feedback and change. Computer 36(6): 39–43.

Woodward, E., Surdeck, S., et al. (2010). A practical guide to distributed Scrum. Boston: Addison-Wesley.

Zurcher, F. W., and Randell, B. (1968). Iterative multi-level modeling: A metholodology for computer system design. Proceedings from the 1968 IFIP congress, IEEE CS Press.

2. How does Waterfall development differ from Agile development?

3. How did the term “Agile” emerge?

5. Why is the term “over” used in the values?

6. What are the key themes in the Agile Manifesto principles?

7. What organization developed as a result of the Agile Manifesto?