Code Smells and Antipatterns (or, How to Tell If You’re Being Too Clever)

Simplicity starts with knowing how not to do too much.   

Here’s an example of how complexity—a lack of simplicity—can affect a product. When one of the authors of this book was in college, he was given a gift of a kitchen appliance that was a combination of a blender, mixer, and food processor. It didn’t do any of those jobs particularly well: it was a pretty good blender, a mediocre mixer, and an utterly unusable food processor. It also had additional problems that you wouldn’t typically get with any of those appliances. For example, it had odd-shaped parts that were difficult to store and almost impossible to clean.

Nevertheless, for more than 10 years after college, he didn’t buy a new mixer or food processor, because it seemed redundant. But because the hodgepodge appliance’s food processor did such a lousy job, he avoided making any recipes that required a food processor. When he finally put the thing on a high shelf (he still couldn’t bring himself to throw out a “perfectly good” electrical appliance) and bought the cheapest food processor at the local chain drug store, suddenly he could make a whole range of recipes that he hadn’t been able to previously. And he wondered why it took over 10 years to get around to doing this.

This appliance was worse than useless. Instead of allowing its owner to make recipes that required a food processor, it basically prevented him from making them. But at the same time, its mere existence prevented him from buying another food processor, so for years he simply avoided any recipes that required a food processor. It’s not that he didn’t have the ability to make these recipes. It just somehow never seemed worth it to make the recipe because it was so much extra work to use the appliance. But for years it never even occurred to him to buy a new food processor, because he already had a “perfectly good” one. He definitely would have bought one had he not already had this appliance.

The reason that this appliance was so bad was because it was complex. Doing three jobs in one required extra parts, and required technical compromises that led to a design that was unintuitive and difficult to use (for example, it was very hard to get the mixer attachment on, because it wouldn’t work unless it was seated perfectly). Replacing the hodgepodge with three simple components—say, a cheap blender, a cheap mixer, and a cheap food processor—would have cost less to begin with, taken up less storage space, and allowed for much more flexibility in the kitchen.

Problems like this are why simplicity is one of the XP values that we learned about in Chapter 6. XP teams recognize that this exact sort of complexity problem can affect project planning—just like we saw earlier in the book when our Scrum teams ran into trouble.

It’s not hard, for example, to get most people to agree that for a team to be productive and get work done on a project, it’s not useful for the project to have people who are sitting around idle with nothing to do. However, some command-and-control project managers take this idea to an extreme: they’ll come up with a project plan that has every resource 100% allocated,⁴⁸ and require each developer to log every minute of his or her time against an approved task to make sure that all of the time is accounted for. Doing all of this extra tracking to achieve 100% allocation and verify that everyone is working all the time creates a lot of project management overhead: people need to keep track of their time and enter it (which doesn’t sound like a lot of work in theory, but in practice can be time consuming and mentally taxing); managers need to create, update, and review the plans; and there are inevitably many meetings to “fix” problems because a developer only seems to be working 95% of the time.

This is a complex project management system—but more importantly, that extra complexity doesn’t actually add value to the project. The team now spends approximately 20% of their time keeping track of and talking about what they do with the other 80%, which makes the whole project take longer. It seems like the team spends more and more of their time in status meetings, answering questions like, “What percent complete are you?” and, “Can you estimate how many more minutes you’ll be spending on this task?”

An XP team uses iteration to make project planning decisions at the last responsible moment, just like Scrum teams. These practices are a much less complex way of planning projects. XP teams do not typically monitor everyone’s time constantly, the way that the project manager in our example does, because monitoring everyone’s time all the time doesn’t actually help the project get done. Despite lofty intentions, the information that’s collected from all of that monitoring is rarely used to make projections during the project, and is almost never revisited after the project is done. Worse, we already know from earlier in the book that projects change over time. Planning for 100% allocation requires the entire team to make all of their project-related decisions at the beginning of the project; when any one of those decisions turns out to be wrong (and some always will—that’s a normal part of running a project), it requires the entire team to revisit the plans, tasks, and allocation levels.

There’s also a larger lesson in this example that can help you understand XP, and it has to do with recognizing patterns and using them to improve how you work.

If you’ve ever been on a team where a strict command-and-control project manager requires everyone to have 100% allocation, calls endless meetings to check everyone’s status and ensure that allocation, and requires everyone to constantly update their tasks to meet that artificial allocation, then reading the last few paragraphs probably made you anxious and angry, and possibly gave you flashbacks to endless status meetings. This is because you recognized an antipattern: a pattern of behavior that a team exhibits that causes problems with the project. Recognizing the antipattern in project management gives you a way to call out the problem, and hopefully find solutions that will help simplify the way your project is managed—and let the team get back to building the product.

XP Teams Look for Code Smells and Fix Them

Code also has antipatterns, and just like with our project management antipattern, recognizing code antipatterns is the first step in stripping the complexity away from your codebase. When an antipattern specifically relates to the structure or design of code, it’s called a code smell. XP teams are always on the lookout for these code smells, and develop the habit of fixing them when they find them, rather than letting them fester. 

We’re going to spend some time talking about a few of the most common code smells, so that you can better understand the mindset of an effective XP developer. This particular list isn’t arbitrary; it comes from the experiences of many, many developers who discussed problems they’ve found in their own projects over the years. The terms “code smell” and “antipattern” first became popular with developers in the mid- to late 1990s, in large part thanks to the WikiWikiWeb, the very first wiki, created by Ward Cunningham, one of the authors of the Agile Manifesto. The WikiWikiWeb was (and in some circles, continues to be) one of the most popular places for developers  to discuss software design. The more people got together to discuss design and architecture problems they found in their projects, the more they found that there were common symptoms that they shared.⁴⁹

Many software engineers—including us—had an almost visceral response when first encountering these problems. It’s a strange feeling to have a frustrating issue that you can’t quite put your finger on perfectly described by a stranger. And that’s actually a very important thing to keep in mind throughout this discussion: that these problems are created by people, recognized by people, and can be prevented or fixed by people. The symptoms might be technical in nature, but the root cause is a problem that affects the people on the team, so the solution will have to be both technical and team-oriented.

When we’re holding training sessions for developers, the code smell that gets more people nodding their heads in recognition and agreement is shotgun surgery. This is what happens when you attempt to make a small change to one part of the code, but find that it requires changes to, say, two or three more seemingly unrelated or barely related parts of the codebase. You try to make those changes, only to discover that one or two of them require additional, also apparently unrelated changes. If the codebase is particularly smelly, it’s not uncommon for a programmer to attempt a change that should be simple, make a dozen or more hops throughout the code, and eventually give up because his brain hurts from keeping track of the expanding trail of cascading changes.

When a developer digs down into the code that caused him to perform shotgun surgery, he’ll often find code that exhibits another code smell: half-baked code (or half-baked objects, because this is not uncommon in object-oriented programming). Code smells half-baked when a programmer, in order to use one object (or module, unit, etc.), must also initialize other objects in a specific, prescribed way. For example, after initializing a particular library, a programmer always has to set certain variables to their default values, and must also initialize a separate library that it’s dependent on. He only knows to do this because of documentation or sample code; failing to initialize it correctly will cause it to crash—or worse, exhibit unpredictable behavior.

Figure 7-1. Effective XP developers are in the habit of finding and fixing code smells like very large classes and duplicated code. These antipatterns have to do with the structure of individual units. We’ll use gears as a visual metaphor for code in the figures in this chapter.

Some code smells have to do with the way the code itself is written. When your code has very large classes (or, for non-object-oriented code, very large methods, or functions, or modules, etc.), it can be difficult to read and maintain. But more importantly, it’s often an indication that it’s trying to do too many things, and can be separated out into smaller units that are easier to understand. On the flip side, duplicated code is an identical (or almost identical) block of code that shows up in more than one place. This is a very likely source of bugs, especially if a programmer makes a change to three of the duplicates, but forgets to make that change to the fourth.

Figure 7-2. Some code smells have to do with the larger design of the system, and how the units interact with each other.

Other code smells concern the overall design of the code—how the individual units of code interact with each other. Spaghetti code, or code that has a complex and tangled structure, is one of the oldest code smells known to software engineering, dating back to at least the 1960s. You can often identify spaghetti code because it’s decorated with scary or apologetic comments from other developers who previously tried and failed to detangle it. But its smelly cousin, lasagna code, might be a more insidious problem. Modern software design typically divides code into layers that each have a specific purpose or behavior that contributes to the whole design. However, when there are far too many layers, and those layers don’t necessarily have a coherent pattern to them, it gets difficult to understand exactly what any one layer is supposed to do. This can be compounded by leaks between layers, where code, types, or ideas that should be contained within one layer find their way into neighboring ones.

Hooks, Edge Cases, and Code That Does Too Much

The code smells we just described are antipatterns that have to do with the structure of the code. There can also be problems with what code does. And effective XP developers understand that they need to be concerned with code behavior as much as code structure.

Here’s a quick example. A team will anticipate a way that a unit of code—like a Java class—will be used in the future, and add a hook, or a placeholder to handle that future case. The hook seems “free” (because there’s so little to it), but it really does have a cost: tying the team to a decision made today that could be put off until later. When it finally comes time to use the hook, the team has learned more about the problem that has to be solved, and the hook needs to change. It’s frustrating for a team to discover that they made assumptions about how the hook would be used, only to realize they designed another part of the software around that hook. So now the hook is harder to change—despite the fact that it’s empty!—because it’s already used elsewhere in the code. A programmer who’s in the habit of adding too many hooks forgets where those hooks are. She’ll routinely run into trouble where she tries to use code she wrote a few weeks ago, only to discover that she never actually wrote the code, and instead will find a comment that says “TODO” where code should be. This is an antipattern: adding so many hooks that it’s difficult to keep track of exactly what the code actually does.

Another antipattern that can make code difficult to understand is obsessing over edge cases. An edge case is a situation that happens rarely, and under a specific set of circumstances.⁵⁰ For example, a program that loads data from a file needs to handle the case where that file is not found. That program may also need to handle the case where the folder that contains the file cannot be found, possibly following different behavior than in the first case. There’s also the case where the file exists but cannot be read from, or the case where the file is deleted partway through the read, or the case where the data in the file is in the wrong character set. A good programmer will be able to think of many, many edge cases just for the simple situation of loading data from a file. Where do you draw the line?

Many developers get bogged down in handling too many edge cases. There’s a genuine need to handle edge cases, and many code situations require handling for several distinct ones. But every time additional code is added to handle each new edge case, the project has diminishing returns. There’s a trap that some programmers are especially prone to where they spend as much time “bullet-proofing” their code against edge cases as they do writing the rest of the code. This is the point where edge cases become similar to overplanning, and start diverting effort. By its nature, code to handle edge cases is rarely run. If the code looks like a tangle of edge case handling, that makes it more difficult to understand, which makes it hard to change.

Good developers tend to be the sort of people who obsess about details—and edge cases are exactly that sort of detail. And cleverness can turn a good developer into an edge-case-obsessed extremist. When this happens, team meetings are hijacked by a passionate but endless discussion of edge cases for a specific situation, where every programmer in the room suddenly develops a strong opinion that his or her favorite obscure edge case absolutely must be handled.

When developers overplan for edge cases or add too many hooks, they’re being too clever. Instead of thinking about building code that’s simple and easy to change, they’re thinking about building code that’s too complex, too abstract, or solves tomorrow’s problems and not just today’s.

This brings us to what we call the framework trap, an antipattern that stems from extreme developer cleverness.  The framework trap is what we call it when a developer has to solve a single problem or perform a task, but instead of just writing code to do that one thing, he writes a larger framework that can be used in the future to solve similar problems or perform similar tasks.

Ron Jeffries, who worked with Kent Beck to co-found XP, described a simple way to avoid the framework trap: “Always implement things when you actually need them, never when you just foresee that you need them.”⁵¹ Some XP teams like to use the acronym YAGNI, or “You Ain’t Gonna Need It,” when they talk about this. When you try to foresee a need, and then you build extra code to meet that need, you’re very cleverly falling into the framework trap.

Figure 7-3. Brilliant web comic xkcd shows what happens when developers fall into the framework trap.

Instead of solving that single, specific problem that needs to be solved today, clever developers like to aim bigger. They’ll have thoughts along the lines of, “I can write this simple solution, but it will be even better if I build something that automates much of the work I just did so that nobody ever has to solve a problem like this one again.” A developer who started out building a web page ends up with a framework that generates web pages. An attempt to solve a specific performance problem turns into a large, general-purpose cache. A simple program to download files on a schedule somehow acquires a scripting engine.

It’s not hard to understand why developers go this route. Say that someone is thinking the following: “Had someone else built a framework for the job I’m working on right now, then I could just use it. So why not take a little extra time, put in the thought to abstract the problem, and come up with a more general solution?” That’s actually a pretty noble sentiment, and it helps explain why so many teams fall into the framework trap.

What’s the difference between a library and a framework?

You know you’re using a library if you can bring it into your project as a single independent part, without having to bring many other parts along with it. The difference is that a library is about separating code into small, reusable components. If the goal is to build simple code, then every component should do exactly one thing; a component that does many things should be separated into smaller, more independent units. This kind of separation is called separation of concerns, and effective XP developers recognize that it is an important element of code simplicity.

A framework, on the other hand, is about combining many small, reusable pieces into one large system. Learning C#, for example, usually means learning to use the .NET Framework as well as the syntax of the C# language. It includes libraries for accessing files, doing mathematical calculations, connecting to networks, and many, many other things (yes, including buttons). When you start building your program with C# and .NET, you get all of these things for “free” because they’re part of the .NET Framework. It’s generally not possible to separate them out. And it wouldn’t really make sense to try to use one piece of the .NET Framework in a program that isn’t written with .NET. Like most frameworks, many pieces have been combined into a large, all-or-nothing system.

Frameworks are much more complex than libraries, and a team that tries to build a framework when a library is more appropriate will often end up with overly complex design. This mistake is common when programmers see that other frameworks save them a lot of time, and want their code to do the same thing: “I didn’t have to include any libraries to use this button in my .NET program, I just got it for ‘free.’ If I build a framework for my whole system, look at all of the things I can give other developers for ‘free,’ too!”

The downside of approaching a problem this way is that if you want to get all of those things for “free,” you must use the entire framework. And this sometimes causes you to think about the problem differently. A .NET developer will naturally think about a problem somewhat differently from a Java developer (but not always too differently, because the frameworks for Java and .NET share many concepts), and very differently from a C++, Python, Perl, or Lisp developer. This is because when you have a set of tools to solve a problem, you think about the solution in terms of those tools. These are examples of languages and their frameworks (you typically don’t think about building software in Java, Python, or Perl without also thinking about their standard libraries), so they don’t place many restrictions on the way you think about the problem you’re solving.

Figure 7-4. Building complex frameworks to solve simple problems sometimes feels like the right thing to do at the time (source: xkcd).

Here’s a common example of framework-oriented thinking. Teams use build scripts to automatically build their software. (In Chapter 6, we learned that on XP teams, these scripts run in under 10 minutes.) There are many tools for building scripts like this, and many of them include scripting languages. But for a framework-minded developer who works on several projects, it’s not enough to write a separate build script for each project. It seems like there is far too much duplicated code between those scripts—and often, there is—so he’ll write an entire build framework for all of the projects. (“If you use my framework, you don’t ever have to write a build script. You’ll get builds for ‘free’!”) So instead of having a small script for each project, there’s a large framework that has to be maintained separately. This developer made a tradeoff: he prevented a few dozen lines of build script from being repeated in several projects, but replaced it with a framework that consists of hundreds or even thousands of lines of code. When something goes wrong with the build (which, based on Murphy’s law, usually happens at midnight just before a release), instead of troubleshooting a small build script, the developer who’s trying to get the build to run has to track down a bug in a large framework. This is not simple.

Combining a lot of functionality into a large framework to solve a specific problem may work really well to solve that particular problem. When the problem itself changes, the framework makes it more difficult to respond to change.

In other words, by combining a lot of different pieces into a large framework, team members think that they’re saving time, but what they’re really doing is limiting their ability to respond to new information and needs. We call this the framework trap, and we can use it as an important idea for learning about simplicity and how it affects your project. The reason we like it as an example is that it captures an important aspect of a mindset that isn’t always compatible with XP: a developer who prefers to combine behavior together into one large unit, rather than separate it into many small units.

Figure 7-5. A framework that seems like a good idea today can turn into a burden tomorrow.

Code Smells Increase Complexity

We’ve just described a bunch of code smells that XP teams habitually avoid. One thing that these code smells all have in common is that they increase the complexity of their projects. And the converse is true: if you find yourself having to remember how to initialize an object, or straining to remember a cascade of changes that you need to make, or giving up trying to pick your way through some painful spaghetti code, those are all indications that your projects’ codebase almost certainly has a lot of unnecessary complexity. This is why XP teams strive to find and fix code smells as early as possible.

These aren’t the only code smells, and they may not even be the most common ones on your projects. However, we chose these because after talking to many developers over the years, we’ve found that almost everyone recognizes at least one of these problems in their own code. These problems don’t happen because of randomness, poorly skilled developers, or deliberate decisions caused by laziness or maliciousness. They’re the result of bad habits that developers fall into, especially when they’re pressed for time, or when they’re forced to work around constraints (especially ones that they put into the code earlier in the project).

So what do you do when you find code smells in your project? Can they be prevented?

Fix Technical Debt by Refactoring Mercilessly

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid.

Ward Cunningham, Agile Manifesto author

Poor software design and development adds up over time. Even great developers write code that can stand to be improved. The longer that design and code problems linger in the codebase, the more those problems can compound, which eventually leads to headaches like shotgun surgery. Teams refer to these lingering design and code problems as technical debt. An effective XP team leaves time in each cycle to “pay down” debt. This is a good way to use slack, the primary XP practice we covered in Chapter 6 in which the team adds stories and tasks to each weekly cycle as a buffer to absorb unexpected work.

Any good financial advisor will tell you that the best way to avoid money problems is never to get into debt in the first place. The same goes for technical debt. This is why XP teams refactor mercilessly, constantly smelling their code and looking for ways to simplify it through refactoring. As the programmers refactor, they learn more and more about how their code is used, and how that actual use differs from the way they’d expected it to be used. Through constant revision, each unit in the codebase gets increasingly better suited to the way it will actually be used. That iterative nature of constant coding and revision is something that a lot of teams don’t necessarily plan for at the beginning of the project. But even though it takes extra time, refactoring mercilessly saves far more time than it costs, because a simple codebase is much easier to work with than a complex one.

When everyone on a team is in the habit of constantly refactoring, they build a codebase that is much easier to change. When the team members discover that they need to implement a new story, or more commonly, they discover that they misunderstood one of the stories and have to change the code, it’s much easier to make the change to the code. They can embrace change—a primary goal of every XP team—because they’re not fighting against the code to make the changes.

Use Continuous Integration to Find Design Problems

We learned about the continuous integration practice in Chapter 6, and it’s one of those mature tools that teams have at their disposal today. One reason that continuous integration improves design and prevents these design problems is that it allows the team to fail early when they introduce integration problems.

Figure 7-8. Continuous integration uncovers issues early.

A system that’s designed to immediately report a failure as soon as it happens is called a fail-fast system. You want your system to fail quickly (hence the name, “fail-fast”) if it needs to be fault-tolerant, and when it’s important to discover the root cause of any problem as early as possible. Early failure provides a feedback loop that lets you apply that knowledge back to the project. This idea can be applied to the way that a team builds software, as much as it applies to the software that they build.

Failure is one of the principles of XP that we discussed in Chapter 6, and using continuous integration enables the project to fail fast when two team members add incompatible or conflicting code at the same time.Team members with a mindset compatible with XP will see integration failures as positive because failures help identify and fix problems early, when those problems are easiest to repair.

Figure 7-9. When design problems are caught early in the project, they’re easier to fix, which prevents messy, kludgey hacks later on. That’s one way continuous integration leads to better overall design.

Test-first programming helps the team naturally build small, independent pieces that are easier to integrate, so continuous integration is less of a burden for an XP team. The team constantly tests each unit to make sure the pieces work. When everyone on the team continuously integrates their code with the rest of the codebase, it prevents an individual developer from going off on his own and building a piece that doesn’t quite fit.

Avoid Monolithic Design

Let’s say that you’re working on a software project, and you learn more about the problem that the software meant to solve partway through. That’s a pretty familiar scenario for most software teams, and your team needs to be capable of making decisions at the last responsible moment in order to solve that problem.

Figure 7-10. When developers don’t have good habits that help them produce simple, decoupled code, they end up with code that has a monolithic design.

Traditional waterfall teams often have trouble with this scenario. Figuring out requirements up front, reviewing them with a large audience, and then building the code all at once naturally leads to code that’s difficult to change. The team doesn’t have a lot of incentive to design for change, because their changes are well controlled. The developers never develop habits (like constantly refactoring or spotting code smells) that cause them to build code that’s easy to change.

This often naturally leads to monolithic design, or a software design made up of large, interconnected units that have many interdependencies and are difficult to separate.

Earlier in the book, we talked about how software is different than construction. In construction, the most damaging thing that you can do is take a sledgehammer and knock down walls. In software, deleting code doesn’t cause much damage—you can just recover it from your version control repository. The most damaging thing you can do is write bad code, and then write more code that depends on it. Adding a dependency like that is sometimes called “coupling,” because once those two pieces of code are tied together, it’s difficult to change one without at least thinking about how the change will impact the other one.

Monolithic design often features tightly coupled code, where the typical unit has many connections to many other parts of the system. Trying to change highly coupled code is very frustrating—it’s often the coupling that leads to shotgun surgery and many other antipatterns. It’s those (often undocumented) connections between units that cause rework that produces bugs.

Just as code can be coupled, it can also be decoupled, by breaking the connections between units (or, even better, by never adding those connections in the first place).

Code smells and refactoring can help you to create decoupled code. Flip back to the refactoring example with the beehive code, where we extracted two methods from a block of C# code to move bees between a field and a hive. We saw that those two methods could be reused by another part of the codebase that also needs to move bees in the same way. But what if those methods are part of a larger unit that needs extensive initialization? If a programmer has good habits, when he tries to reuse those simple methods, he’ll smell the half-baked code and refactor it to remove the extra initialization. Now it can be used in two different parts of the codebase, and it doesn’t need to “know” how it’s being called, because it’s decoupled from the parts of the code that call it. And if it needs to be called from a third part of the codebase, it will be decoupled from that part as well.

A system is easiest to maintain when it’s built with small, independent units: every part of the code is decoupled from every other part as much as is practically possible (which can be a surprising amount), so that there are very few dependencies between them. This is a major key to building a system that can be changed without a lot of rework. If your design is highly decoupled, then you’ll rarely find yourself performing shotgun surgery or making changes to many parts of the system for a single change. And when you do, if you’ve developed the good habits to refactor today (not tomorrow) to remove those couplings, then you’ll leave your codebase in a better, more decoupled state than you found it: with each unit performing only one task, and decoupled from unrelated units.

Teams Work Best When They Feel Like They Have Time to Think

Software development is a game of insight, and insight comes to the prepared, rested, relaxed mind.

Kent Beck Extreme Programming Explained: Embrace Change, 2nd Edition

Energized work means establishing an environment where every team member is given enough time and freedom to do the job. This keeps them in the mental space where they can develop and use good habits that lead to better, more naturally changeable code. This, in turn, lets them produce much more code, and deliver more valuable features (and features that are more valuable) to the users and customers, in a shorter period of time.

Software development is almost entirely a mental exercise.⁵⁵ Every good developer has spent hours staring at a problem and gotten nowhere, then had an epiphany later about exactly how to solve the problem while eating dinner (or taking a shower, or riding a bike, etc.). Every software project relies on a series of small innovations like that, one after another.

It’s widely known that it takes time (usually between 15 and 45 minutes) for a developer to get into a state of “flow,” a state of high concentration in which she’s highly productive.⁵⁶ Interruptions and distractions can pull a developer right out of this state. But when she is put under pressure to deliver code as fast as possible yesterday, it’s almost impossible to get into this state. She’s so worried about pushing the code out the door that she feels like she doesn’t have time to think.

Disrespect, unrealistic artificial deadlines, and other bad behavior from the boss can lead to an unenergized environment, or a working environment where people don’t feel like they have the ability to make decisions, and don’t have the mental space to make good decisions or innovate.

When managers create a climate where development work is always behind, and the team is held to arbitrary, unrealistic deadlines, they’ll cut corners when building the code. This usually means throwing out many of the great practices that XP brings. But more importantly, it means that the developers rarely get to the mental state—“flow”—that’s necessary for bringing simplicity to the project.

XP teams strive to create the opposite kind of workspace: an energized environment. Giving the team an energized environment means helping every person on the team to feel like he or she has autonomy. Everyone can decide how their work should be done, and feels like they have the freedom to make changes—not just to the code, but also to the whole plan for the project, and the way that the project is run. The way XP teams do that is with some of their planning-oriented practices: by using weekly cycles so that they don’t arbitrarily make decisions up front that can be decided later, and by using slack to add lower-priority work that can get moved to the next cycle. These practices increase the team’s feeling of autonomy by giving them more flexibility in the plan. And autonomy can also come from the codebase itself: by avoiding antipatterns and building a codebase that’s not difficult to change, the team opens up choices for themselves later in the project.

In Chapter 3, we learned about the agile principle of sustainable pace: agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Can you see how the XP principle of courage is needed in order to maintain a sustainable pace?

Many XP practitioners use the terms “energized work,” “40-hour work week,” and “sustainable pace” interchangeably, and it’s why XP teams create an environment where they can do energized work by establishing sane working hours. There’s actually a long history behind the 40-hour work week—historically, labor organizers demanded “eight hours labor, eight hours recreation, and eight hours rest.”⁵⁷ By the 1950s, and after countless productivity studies and charts that showed productivity decreasing and quality problems increasing when work weeks expanded past 40 hours, many industries had adopted this principle as well. XP teams know that when they set a sustainable pace, and allow everyone on the team time for a life outside of the office, the quality of the work goes up along with the happiness of the team members.

Team Members Trust Each Other and Make Decisions Together

People need a sense of “team”: We belong. We are in this together. We support each others’ work, growth, and learning.

Kent Beck Extreme Programming Explained: Embrace Change, 2nd Edition

Great teams can accomplish much more working together than the same number of people can accomplish working individually. Why is that? When you bring people with the right skills and perspectives together, and when you give those people an environment that encourages open communication and mutual respect, you help innovation to thrive. People on the team have ideas that spark more ideas in others.

The whole team practice is about helping the individuals on the team come together as a whole. When they encounter obstacles, they work together to overcome them. When there’s an important decision that affects the direction of the project, that decision is made together. Everyone on the team learns to trust the rest of the team members to figure out which decisions can be made individually, and which decisions need to be brought to the rest of the team.

On a whole team, every team member is part of the discussion of what features are valuable to the users, what work the team will take on, and how the software will be built. If every team member is trusted to make coding decisions that will deliver the most value, then there’s little risk of individual programmers spending extra time building extra code that doesn’t accomplish the goal.

The flipside of trust is understanding that everyone makes mistakes. When the “whole team” dynamic is working, a team member isn’t afraid to make those mistakes, because he knows that the rest of the team will understand that mistakes happen, and that the only way to move forward is to make those inevitable mistakes and learn from them together.

The XP Design, Planning, Team, and Holistic Practices Form an Ecosystem That Spurs Innovation

A whole team that has an energized work environment will design better software than a disconnected team in a stifling environment. Innovation may sound like a lofty concept, but it’s a practical, day-to-day reality on an effective XP team. Developers who regularly achieve a state of flow and work in an informative workspace that encourages osmotic communication often find themselves feeding off of each other’s ideas. And designing software incrementally gives each developer the freedom to write new code with few constraints—that’s like giving a blank canvas to an artist. The team’s great habits will help them avoid and fix antipatterns, so that when they grow the codebase incrementally they don’t restrict themselves in the future.

When a team with great habits has a manager who trusts them, even more good things happen. A great manager trusts the team to understand the value that they’re delivering, and gives them a climate where they can focus on creating the best product without having to meet an unrealistic timeline. A team with a manager like this works faster, and is more likely to build a highly changeable product that will do a great job of meeting the users’ needs.

Figure 7-12. All of the practices relate to each other, and reinforce each other.

We waited until Chapter 7 to introduce the holistic practices because they make much more sense when combined with the other primary XP practices, and a team won’t really “get” them without a mindset that’s compatible with the XP values and principles. A team that just looks at individual practices and implements them one by one will probably get better-than-not-doing-it-results, but they won’t see a profound impact on the design of the software.

On a good XP team, where everyone has truly internalized the principles and values, the practices form an ecosystem where they reinforce each other. Testing first helps the team to build small, independent, decoupled units, which make any code problems and antipatterns easier to refactor and eliminate. The team members sit together and work in pairs, which helps them work as a whole team, and causes them to have a more focused, energized working environment, which gives them time to think about the design. They integrate continuously, so when one pair injects a defect that affects a different part of the software it causes unit tests to fail fast. This lets the team fix code problems quickly and before those problems are buried under additional layers of code, which leads to a cleaner codebase. Continuous integration is made easier because they have a 10-minute build, which also makes it easier for the team to focus and have fewer interruptions.

To a developer with the right mindset for XP, this all feels very simple. In fact, it seems like you’re just building software the way you should, and all of these things are happening automatically.

A team that doesn’t have that mindset, on the other hand, will run into barriers with XP adoption—they’ll get buried in implementing the individual practices, and feel like they’re done with their XP implementation once everyone is technically writing tests, coding in pairs, using weekly cycles, or using a continuous integration server. Those are straightforward, tangible things that can be checked off of a list. But then there’s this gap between implementing the practices and getting this “ecosystem” effect that an XP team may read about, but never really achieve—just like Scrum teams that might implement all of the practices, but never see those astonishing results or hyperproductivity.

How does a team get into the right mindset? How do they make the jump between implementing practices, and fundamentally changing the way they design and build software?

Incremental Design Versus Designing for Reuse

Andrew: So, designing for reuse isn’t always a great thing?

Auke: It’s not really a very suitable approach for software development. What you typically do in open source is that you start by building something for one use first. Then, somebody else takes it and starts modifying to their needs. And he can reuse it. And once you’ve done that—use and reuse—you know what the commonality is, and you can refactor out the common functionality.

That pattern you see quite a lot. Design for reuse is too static. It doesn’t work that way and is a very poor fit. It may be something to strive for, but is not feasible in the foreseeable future. Use-use-reuse is a much better fit.

Interview with Auke Jilderda, Beautful Teams (Chapter 8)

Throughout this book, we’ve talked about how a team can start to get themselves into the right mindset for a methodology by adopting its practices. At first, they might just get better-than-not-doing-it results. But as they get used to the practices and start to understand how they fit together, the way that everyone on the team thinks about building software starts to shift. This was true for Scrum, and it’s also true about XP. And a key to getting into the XP mindset lies with incremental design.

The Unix toolset is a classic example of incremental design, and that’s what allowed thousands of developers to contribute small and large pieces to it over the course of many years. The Unix tools consist of many small pieces⁵⁸ that were developed independently, mostly by programmers who were trying to solve a specific problem (and not trying to build a large, monolithic operating system). We can use the Unix toolset to learn about how incremental design can let many different people—many of whom have never met—contribute to a single large, robust, high-quality system that has continued to grow incrementally for decades.

Here’s an example of how the Unix tools work. Let’s say that you have a bunch of large text files with a lot of data—maybe they’re addresses for a mailing list, or configuration files for an operating system, or numbers that need to be crunched. You need to quickly come up with a way to transform them (for example, to pull out just the name and phone number, or alter certain sections of the configuration in a specific way, or find data values that match a pattern). How would you do this?

You could write a program that reads the data from the file, processes the lines, sections, strings, etc., and generates the output. But if you’re familiar with the Unix tools (like cat, ls, cut, awk, sed, etc.), you already know that they’re built to handle problems like this. You’d write a script—or even just a command that you’d execute on the command line—that reads the data from the file, does the transformations, and writes the output to another file.

Say you have a large, comma-delimited address book file of addresses called addr.txt. It has eight columns: name, title, email address, phone number, mailing address, city, state, and US zip code. You want to find the name, title, and phone number of every person in Minnesota. If the full name and title are in the first two columns, the phone number is in the fourth column, and state and zip code are in the last two columns, then this Unix command will produce the right output and write it to output.txt:⁵⁹

egrep ",MN,[0-9]{5}([-][0-9]{4})?$" addr.txt | cut -d, -f1,2,4 > output.txt

So were Unix tools built by people who wanted to make it easy to process address books? Obviously not. The Unix tools were based on a philosophy of simplicity: each tool produces output that every other tool can use as input, and every tool does one straightforward, specific job. The cut tool reads each line of the input, cuts out certain fields or columns, and pastes them into the output. The grep tool checks each line in the input and only prints that line to the output if it matches a pattern. But these two tools—especially when combined with the other Unix tools—can do an almost unlimited number of tasks. For example, a system administrator might combine them with tools like find (which finds files in the filesystem) or last (which tells you what users were on the current machine recently) to perform many complex system administration tasks.

These tools are a fundamental part of what made Unix the most popular type of operating system for Internet services and business systems. And over the years, a general design for system tools emerged, along with a system administrator culture among people who use those tools every day. New tools and new uses for those tools have evolved over time. For example, file compression became more widespread in the 1980s and 1990s. When the gzip tool was added to the Unix toolset in 1992, there was a straightforward pattern for adding it. Before the first line was written, it was clear how it would get its data in and out (pipes), how it would be executed (from the command line), even where documentation would go (man pages). The toolset could easily expand, and because all of the tools are decoupled and independent of each other, the ability to compress and decompress files could be added without requiring any changes elsewhere in the system.

Over the last four decades, the Unix toolset has continued to expand in this way. Thousands of developers contributed individual tools, or made improvements to the existing tools. The Unix system grew in a natural, almost organic way. And along with it, a culture and knowledge base grew. This allowed people who use Unix to perform more and more complex tasks, which in turn helped them find new ways that the system needed to grow.

When Units Interact in a Simple Way, the System Can Grow Incrementally

The Unix tools all follow a very strict pattern about input and output. The | character in the command line above is a pipe—it’s used to take output from one tool and feed it as input into another. The < character pipes input from a file, and the > character pipes output to a file. The standard Unix tools all produce output that works with these pipes—that’s part of a contract that each of those units adheres to. The contract between Unix tools is very simple, which makes it easy to extend the whole Unix system. As long as each additional tool adheres to the same contract, it will fit in with the others.

An important tool for helping a team build a system using incremental design is to have a very simple contract between the units of that system. How do those units communicate with each other? Do they pass messages? Call functions or methods? Access services on the network? The more simple and consistent the unit-to-unit communication mechanism is, the easier it is for the team to add units.

So how do you make a contract simple? The same way you make units simple: make decisions about how units communicate at the last responsible moment. And we already have a tool to help with this: test-first programming. When a developer builds unit tests first, it helps her strip out complexity by forcing her to use that unit before it’s written. She can use the same unit testing tool or framework to write simple integration tests that test how multiple units communicate with each other—and she’ll write those tests before she adds the code to communicate. For example, if there’s an interaction between units where the output of one unit is passed as input into another, whose output is then fed into a third, she’ll write a test that simulates that interaction—how they integrate together.

The reason this helps keep the contract between units simple is that it becomes clear very quickly if that contract is complex. With a simple contract, this sort of integration test is very easy to write. But when the contract is complex, the integration test is a burden: the developer will find herself having to initialize many different, seemingly unrelated objects, clumsily translating data from one format to another, and having to jump through hoops to get the units to communicate. And just like with writing unit tests, writing these integration tests first will help avoid those problems before they get built into the software.

Great Design Emerges from Simple Interactions

We use the term “emergence” when complex behavior arises from simple systems. Emergence is common in nature: individual ants have simple behavior, but a whole ant colony has much more complex behavior that arises out of the interactions between individual ants; the whole of the ant colony is greater than the sum of the ants. 

When a system is designed so that its behavior seems to emerge from the interactions between the individual units working together, in a way that doesn’t seem to originate from one single unit, it’s called emergent design. Systems built using emergent design almost always consist of small, independent, decoupled units (like Unix tools, or ants). Those pieces are combined to perform complex tasks, and the behavior of the system comes as much from the interactions between those units as it does from the individual units themselves.

There’s rarely a deep hierarchy of units calling other units; instead, the system will use messages, queues, or other ways for units to communicate without requiring a central control system. Again, the Unix tools are a good example of this.⁶⁰ Input is piped out of one tool and into another, which makes them easy to chain together. This leads to a very linear usage: the first tool is called, then the second, then the third. It’s possible to create a more complex interaction between the tools, but they lend themselves to a simple, shallow call structure (as opposed to a deep, nested call hierarchy with layers of units that call each other).

When a system is designed like this—with simple units that interact in a simple way—something interesting can happen. It can start to feel to the team like the behavior emerges from the system as a whole, not just the individual units. Complex behavior often doesn’t seem to have a single source at all. This is a lot like the way that an ant colony’s behavior doesn’t seem to originate with any individual ants. The team has this real, palpable feeling that each part of the system is simple, and that the behavior of the whole system emerges from the interactions between those units.

To people who have spent years building the Unix toolset, this feeling of emergent behavior is familiar. You can start to see a very basic kind of emergent behavior from the combination of parts in the simple address book example: cut only knows about extracting characters from a line, and grep only knows how to match patterns, but somehow they combine together to process addresses without a specific command to tell the system what an address is, or how to process it.

XP teams that are firing on all cylinders find themselves naturally building software and systems using emergent design. This starts with simplicity: each unit is designed for a specific use. Test-first programming ensures that the unit stays simple and only has that one use. The programmer writes the tests to make sure that it performs that one function, and then writes the code—and when the tests pass, he stops writing it, so that no extra behavior is added, and there’s no code that’s there for no reason. All of the code in the unit is necessary.

The team avoids deep call stacks, where one unit calls another, which calls a third, and so on. Instead, the program’s call structure looks flat and wide, which reduces dependencies between units. These interactions are kept as simple as possible: when one unit needs data, it gets it from one other unit—or, even better, from a message on a queue, so it doesn’t even need to know where that input came from. To keep the system simple, the team avoids multi-stage, complex interactions between many units (just like Unix tools only pipe data in from one tool and out to another).

New design emerges when the people on the team discover changes that need to be made. To add new behavior to the system or change the way it works, they need to change the individual units, or change the way those units interact. But because each unit is decoupled, and the interactions between them are simple, these changes tend not to cascade throughout the system. If the code smells bad—say, the programmer finds himself doing shotgun surgery, or runs across spaghetti code or half-baked objects—that’s a giant warning sign that unnecessary complexity has worked its way in, and he knows (and, more importantly, he feels) that it’s worth taking the time to refactor it out into simpler units.

A system built using emergent design can grow for years, while staying maintainable and easy to change. Teams that are good at XP have the tools to keep the codebase simple, and they work in a way that encourages them to continually monitor the code for complexity and refactor it out. This in turn lets them comfortably make changes, which allows more design to emerge. It’s a virtuous cycle that leads to simple, stable, very high quality code—and, as a result, teams find that it’s much faster to build and maintain software this way. When the system needs to be maintained (to fix a bug, or to make a change), there’s rarely a large, overarching modification that touches many different parts of the code.

And because the team members know how flexible the system is, and what they can and can’t extend, they get better at making decisions at the last responsible moment. They find that as a team, they have an increasingly good sense of when a specific design decision can be made tomorrow—which in turn helps keep the code simple today. The team is more relaxed, and team members work better together. They produce high-quality code very quickly, and, when they need to change the code, it can be done easily, and without creating bugs.

In other words, teams keep their code simple, which allows them to embrace change.

And that’s the point of XP.