Chapter 5. Delivery Is Still an Experiment
This chapter is about all the ways that Agile delivery enables business agility. We explore how DevOps and Continuous Delivery (CD) enables endless change, making it possible to run continuous experimentation without dramatic consequences. Then, we look at how delivery teams embrace the unknown at scale by using practices like evolutionary architectures and microservices. We’ll see similar patterns of thinking as explored so far, but this time applied to how we build things instead of what things to build.
Until now, we’ve focused on customer value, identified opportunities, tried many options, embraced a learning mindset, and refined our vision and strategy to get there. You’d be forgiven for assuming that this means building working solutions at scale is merely an engineering challenge. We have validated what to build, now we just need to build it.
Wrong.
Software is still an experiment—just a really expensive one. Having delivery teams writing code and deploying working software in no way suggests that things won’t change. But it does mean we’re probably spending a lot more money to learn what is and isn’t working.
So, we need ways of building things that are dynamic and can adapt to change. This isn’t just about pivoting. It’s also about scaling and evolving solutions over time. If we accept that today’s solution might be different than tomorrow’s, we want to create it in a way that:
is fast and economical to meet immediate needs; and,
doesn’t constrain our ability to meet future needs.
A good predictive project will go according to plan, a good Agile project will build something different and better than the original plan foresaw.
Martin Fowler
Another complication is that people generally aren’t very good at communicating what it is that they require or desire until it’s real. “I’ll know it when I see it,” says the stakeholder. For software, when I see it often means when I experience it. Prototypes can help by giving a tangible demonstration of concepts and ideas, but often it’s not until a working solution has been completed and deployed that people can articulate their thoughts about how it needs to be different.
Because it’s intangible, people assume that software is changeable. “We’re not building houses,” they say. That’s true. The problem comes when people conflate possible to change with easy and cheap to change. Although that can be true, it doesn’t happen by accident. Being able to respond rapidly, gracefully, and economically usually requires a change in how work happens, along with investment in the infrastructure needed to work that way.
The goal of great software teams is to handle this kind of situation better. Let’s look at ways that we can do this.
DevOps and CD
DevOps is a portmanteau of the words development and operations. And, like its name suggest, it blends the activity and responsibility of development (building software systems) and operations (running software systems). Why did this become a “thing”? Because doing them separately has a negative impact on an organization’s ability to react. When a separate team builds software, and a different team operates it, bad things happen.
Things take longer because there needs to be a hand-over of responsibility, which, in a siloed, risk-averse culture, is likely to include forensic scrutiny and quality checks for the purposes of ass-covering, before an operations team will accept the work. That’s understandable, because it’s the operations team that is bound to Service-Level Agreements that might require a server to be restarted at 3 a.m. on a Saturday morning.
Things cost more to change, because development teams, often under pressure to meet a launch deadline, might be focused more on completing a working product, than building something that’s scalable and easy to work on after launch. It’s easy to scald teams for such behavior, but that criticism is usually misguided. This happens not because of bad people, rather it’s a symptom of how organizations are structured,1 combined with unrealistic demands, like top-down deadlines that have no relationship with reality.
When things fail, it’s more difficult to recover. That’s because the ops team that is running the system didn’t build the system. The intricacies of its inner workings, the idiosyncrasies of its configuration, and the built-in fragility caused by the initial race to hit a deadline makes for a tough puzzle. This has the potential to cost organizations dearly. Like the four-and-a-half hour outage that grounded more than 400 flights for American Airlines while it restored its reservation system after an IT glitch. IT systems are often more expensive to run than they are to build, with Total Cost of Ownership dwarfing the initial project cost. Yet IT organizations are often optimized for delivering software systems, not running them.
Enter DevOps.
The DevOps movement says “you build it, you run it,” and “you break it, you fix it.” It’s all about merging development and operations together. Collaboration, automation, and continuity are key themes for DevOps. Teams automate things, specifically the build pipeline. Automated build pipelines solve the dev-to-ops handover problem. There is no handover. Instead, all changes are integrated continuously, in keeping with an agreed protocol of automated steps, tests, and decision points. Working in this way means that deployments—to any environment, including live production—can be made at any time. Continuous Integration (CI) and CD are Agile development practices that give businesses the ability to release software continuously. It makes a lot of sense when you think about it:
Small chunks of value can be released often
Releasing software to customers is now primarily a business decision, not a technology decision
Making It Small and Often
Being able to release small batches of work, frequently or continuously, is game changing. Think of the poor team in the example at the beginning of the chapter. It took many months and much more time and money to get its solution to market. It was a big bang release. And, the team learned too late that customers didn’t want it. That’s an expensive lesson learned.
Small chunks mean a shorter cycle-time from concept to cash. It means organizations have more flexibility to decide what to invest in. It means they learn faster and reduce risk in their investments. In short, continuous delivery of software is an enabler of the Lean management style described in the previous chapters. And, from a product development point of view, it means that design can be customer-led, starting with an early release of a usable product, and iterating toward the best solution based on real feedback, as demonstrated in Figure 5-1.
Figure 5-1. Release value early and iterate toward the best solution
Freedom to Decide When and How Much to Release
When changes are continuously integrated and quality assurance tests are happening all the time, there’s nothing left to do but press the big red button, as depicted in Figure 5-2, when the organization decides that it’s time to go live.
Figure 5-2. Deploy!
This is awesome in a few ways. It’s far less risky. What CD does is “bring the pain forward,” to make difficult things easy, by doing them more frequently. When teams do this, deployment complexity and risk is reduced because the wrinkles have already been ironed out before deployment time, making things go more smoothly than a big-bang release, as illustrated in Figure 5-3.
Figure 5-3. Do difficult things more often to make them easier
Controlling when and how much to release is a powerful choice. One indicator of high-performing organizations is the frequency at which deployments are made:
High-performing organizations decisively outperform their lower-performing peers. They deploy 200 times more frequently, with 2,555 times faster lead times, recover 24 times faster, and have three times lower change failure rates.
Puppet, State of DevOps Report 2016
But what’s most important is choice. Every change need not be released instantly. Sometimes, product teams want to release certain elements together. Like a group of new product features that correlate with a point-in-time marketing campaign. This scenario requires a degree of control and coordination.
On the other hand, many changes—even product features—are best deployed incrementally, frequently, and in small pieces. It’s just less risky and easier to adjust when things go awry. It’s better for everyone. Humans tend to resist change, even when the change equals improvement. So, breaking change into smaller bite-sized chunks makes it easier to swallow:
Most redesign launches result in an initial dip in conversion [...] but Marks & Spencer’s online sales plunged by 8.1% in the first quarter following the launch of its new website.
Forester Research, Make This Website Redesign Your Last
DevOps and CD allows choice. No longer must organizations batch all changes in monstrous three-monthly releases, and then hold on tight, hoping for the best when it’s time to go-live.
Evolutionary Architecture and Emergent Design
These approaches help us to efficiently build today’s technology solution without constraining or complicating future work to extend or change it.
Evolutionary architecture is about understanding the principal characteristics of a system—sometimes called fitness functions—and defining the target architecture in a way that accommodates those needs, yet without locking everything down. The goal is to not make decisions that might restrict options later, unless it’s absolutely necessary.
The initial target architecture gives just enough guidance for teams designing technology solutions. Emergent design describes how a solution comes to bear, based on team-level choices like Are we using AngularJS or ReactJS? By working within the boundaries of the current architecture, choosing preferred options, and just getting started, the design emerges as we explore what works and what doesn’t.
Both evolutionary architecture and emergent design (see Figure 5-4) shift the responsibility of decision making to the teams closest to the work. And, decisions are delayed until the last responsible moment, allowing flexibility to respond to new information as we learn it.
Sound familiar? This is the same mental model implied by mission command and autonomous teams that were introduced in Chapter 4, only this time we’re talking about how it’s built, not what is built.
Figure 5-4. Evolutionary architecture and emergent design
Small, Modular, and Service-Oriented
Microservices architectures is a broad topic. Instead of diving into the details (for that, I recommend that you read Martin Fowler and James Lewis’ paper, or Sam Newman’s book Building Microservices2), let’s instead highlight the benefits of microservices, and how it fits with everything else we’ve talked about so far.
Using microservices is a modular approach to building evolutionary architecture. It’s how we break down enormous, all-encompassing blobs of server-side computational logic, into smaller, more independent, and easier-to-handle components. Doing this is beneficial in lots of ways. From an organizational agility point of view, here are just some of the benefits:
- Each service is highly decoupled from other services
- They’re self-contained, thus technical changes in one service (for example, an internal data structure change) cannot affect another service, making it easier to make changes in isolation without risking catastrophe of unintended consequences, or difficult coordination with other teams in order to make a change.
- Infrastructure is not shared
- Developers have freedom to make the most appropriate technology choices for the task at hand. This means that teams can choose simple tools for simple problems, and complex tools for complex problems. This is more economical and flexible.
- Services are oriented around business functions, not job functions
- This means that it’s easier for one cross-functional team to manage a given service, instead of distributing and coordinating effort across many teams to get work done.
So, microservices is all about choice, flexibility, and responding well to the needs of the organization, by decoupling services, democratizing infrastructure choices, and orienting teams to business functions.
Conclusion
These ideas and approaches coalesce into a competitive advantage for organizations that embrace them. Microservices and evolutionary design enable organizations to delay critical technology decisions until the last responsible moment. Making decisions later means that teams can explore what makes sense and what doesn’t while pursuing the desired outcome. Over time, our threshold of knowledge becomes higher because we’ve taken some action, not just theorized, and we obtain more information to make a better decision.
Because services are isolated from infrastructure (modular) and small in size (micro), they’re more nimble and flexible. This enables real-time experimentation. It’s plausible—even trivial, given the right conditions—for a team to build two complete solutions in parallel, and test them over time, with live data to determine which product approach wins out.
DevOps and CD is how we build more robust software in a way that’s less risky to deploy, faster to recover from failure, and cheaper to change. This helps organizations become more resilient to changing conditions. It also gives greater choice about when and how much to deploy. It’s the mechanics of software delivery that makes experimental product development a reality.
It’s how technology enables the art of the possible.
Closing Thoughts on Design Thinking, Lean, and Agile
The mindsets of Design Thinking, Lean, and Agile have different origins, helping us to think in different ways and to solve different problems. We’ve explored how these come together, complement, and overlap one another. Design Thinking brings discipline to how we frame problems, consider the customer, and explore creative solutions. Lean thinking brings discipline to how we learn, make decisions, and coordinate efforts on our quest to achieve our goals. Agile is how we build technology solutions that evolve and adapt as we learn and respond to changing needs that emerge from taking action.
There is no one correct way, nor is one single mindset enough. But all together, elements of each mindset help us to find our way forward.
The four steps of actionable strategy describe how these mindsets come together in practice. It’s the scaffolding that frames the work we do. The glue that brings many ways of thinking together as one.
Although we’ve included practical methods and tools, along with some models to guide how we think and act, making it work is about people. Instead of focusing on process, we ought to challenge how we think, try new ways of doing, adopt the things that work, and learn from the things that don’t. This will be different for everyone. Success is about how we develop new ability, learn by doing, and adapt to what is learned.
If there’s one thing to do after you finish reading this, let it be something new. The best way to build new ability is to try something, see what happens, and adapt as you learn.