In This Context

Humans are too slow and inconsistent to be a blocking factor in the pipeline for deployment to production.

Any human handover or task performed by a human will significantly reduce the number of changes a team can deliver and increase the time required to deliver them.

• Humans can never be as fast as computers.

• Poor testing quality undermines trust in the delivery process, and testing is a waste of time.

• People tend to delay and combine changes if each delivery is risky.

• When tests are slow, people tend to avoid running them as frequently as they should.

Therefore

Automate all the testing required to take any product change to production.

Most functionality should be tested using fast and local unit tests, integration tests can ensure that components are working well together, and only a small portion of the test coverage needs to be on the system UI levels. All long-running and manual tests should be gradually refactored and automated, and they should not block consistent flow of changes to production.

• Use testing pyramid.

• Long tests should not block release.

• Manual and long-running processes happen only in background.

• Continuously add and change tests.

• Consider test-driven development.

• Add advanced in-product testing like A/B, canary, blue/green, etc.

Consequently

The team can trust that the delivery process will catch most issues and that changes will flow to production quickly.

• + The team is ready to deliver changes and take the risks.

• + Developers write tests, which gives deeper insight into the code.

• − If there is a team in charge of manual testing, they may need to be retrained for new responsibilities.

In This Context

When a team of developers works on a set of features that integrates only when all features are finished, the integration process tends to be very complex. The codebase change is large, and in the meantime other devs have integrated separate large changes that can further complicate the integration. To increase productivity, devs often delay interim integration—which leads to a single “big bang” integration just prior to release. A minor bug or conflict that could have been easily caught in an interim integration can now end up delaying the entire release.

• Memory of the change you made fades with time, so delayed integration can increase difficulties.

• Chance of conflicts is smaller when the change is small.

• Frequent execution of the same task creates incentives for automation.

• It is very easy to lose trust in the system if reports are not available.

Therefore

All developers integrate their changes at least once per day.

Integration of all changes is done on a main codebase for each microservice. Code differences are small, less than one day of work, which leads to simpler integration. The main codebase is continually rebuilt and tested to ensure that every developer has functioning and up-to-date code to work with, which minimizes unexpected conflicts with any other newly integrated code.

• Introduce test automation and unit tests.

• Build each change and test it immediately.

• Immediately fix any broken build.

• Commit to the same mainline on the codebase.

• Must have good reporting.

• Use feature toggling.

Consequently

Integration is a nonevent. Products are always in a releasable state.

• + Code is always good-quality, tested, and functional.

• + Collaboration is easier.

• + Minor bugs and conflicts are caught before they cause major problems.

• − There is some overhead.

In This Context

Shared environments and databases are difficult to keep in good shape and create dependencies that lead to delays.

When developers can’t create their own test environments, they may avoid running proper tests before submitting the code or run them on shared environments that may affect the work of their teammates. This affects other developers by making interpretation more difficult.

Differences between development environments and the eventual production environment may lead to the introduction of bugs that happen only in production and are related to those differences.

In all of these scenarios, product quality and developer productivity suffer.

• Local testing reduces interpretation problems.

• If setup for the developer environment is too slow, devs will reuse the same environments.

• If not refreshed, local environments tend to undergo configuration drift.

• Shared environments create dependencies that lead to delays.

• Developers tend to create many test environments if it is easy and fast.

• Developers often have multiple changes that require testing at the same time.

• CI and CD are much more difficult to achieve without being able to test each change thoroughly.

Therefore

Establish a fully automated and fast process to create development environments where devs can test-run their apps. Each developer should be able to have their own environment, or multiple environments, that resemble the eventual production environment.

• Provide the same (or at least close to the same) tooling to deploy apps as in production.

• It is possible to do this on the cloud but will require cost management.

In This Context

Delivery of large monolithic applications developed by large teams require long and complex coordination and extensive testing, leading to longer TTM (Time to Market). Hardware use by such applications is inefficient, which leads to wasted resources.

• People tend to delay painful moments; since integration and delivery are typically painful, their frequency tends to decrease as system longevity increases.

• Larger monolithic systems are increasingly difficult to understand as they grow in size and complexity.

• Monoliths are easier to work with than modular applications as long as they are small enough to be understood by each developer.

• Tiny monoliths (not big ones) are often the quickest, simplest solution to relatively easy problems.

• Conway’s law: architecture tends to resemble the organizational structure.

Therefore

Split applications into smaller, loosely coupled microservices that can be built, tested, deployed, and run independently from other components.

• Small and independent teams work on separate modules and deliver them with only limited coordination across the teams.

• Independent components allow different teams to progress at their own pace.

Consequently

New systems are created from many small and independently built components with a complex web of connections.

• + Faster-moving teams are not held back by slower ones.

• + Teams can choose the most appropriate tools for delivering their particular service.

• − Independence and freedom of choice are achieved, but with the tradeoffs of reduced standardization and certain types of reusability.

In This Context

If APIs among microservices are not well-defined and fully segregated, they will require tighter coupling in development and/or delivery. This in turn introduces dependency, both service-to-service and among teams on an organizational level. This process essentially undoes the move to decouple the monolithic app in the first place, as it leads to coordinated development for the delivery of multiple services and requires very tight collaboration across teams.

This reduces the speed and agility of the organization and effectively re-creates the original monolithic architecture and organizational structure.

• Tight coupling begins with a simple decision to share data directly.

• Conway’s law: a software application’s architecture will evolve to mirror the organizational structure of the company producing it.

• A single team working on multiple microservices may take shortcuts and introduce tight coupling.

Therefore

Microservices should communicate with one another only through the network, using simple, consistent, and stable APIs.

• Build stable APIs with backward compatibility.

• Place most of the service logic within the service itself, keeping the API simple and easily maintainable.

• Smart endpoints, dumb pipes (most of the business logic is in the microservices themselves and not in the APIs).

• Ensure each microservice has no direct access to data of other microservices.

• Make sure there is version control and version management for APIs.

Consequently

Microservices are kept decoupled and independent.

• + Old microservices can easily be replaced when needed as long as APIs are preserved.

• − Communication through network can be slower and is more complex to architect.

In This Context

When moving to cloud native, teams have no experience and no clear reference on the right ways to architect cloud native systems. When a transformation proceeds without a proper plan for standardizing the platform, each team will likely choose very different architecture for the particular piece it is in charge of building.

This makes interpretation of components difficult and maintenance more complex, and will make it more difficult to move developers across teams due to steep learning curves among the different platforms. Furthermore, in the absence of both knowledge and easy solutions, teams may revert to well-known ways (biases) and significantly diminish the value of the transformation.

• It’s easier to reuse existing architecture.

• Some teams would never extend the original version.

• Given full freedom, teams will come up with many different architectures.

• It’s difficult to consider the whole system from within one team.

Therefore

Document the architectural principles to be used in the company, educate the teams on the standardized way to build software in this system, and create an architecture review process to ensure consistency among teams.

Standardizing the architecture early on paves the way for more rapid adoption while preventing chaos. This is an extension of the Vision First pattern: rather than making a bunch of random decisions to move the initiative forward, people with an understanding of how to build distributed systems software have helped make proper technical decisions.

Providing good reference points coupled with clear architectural guidelines right from the start helps the teams to bootstrap the projects in better ways and may avoid costly mistakes.

• Make the architecture sufficiently high-level to allow flexibility for the teams to choose tools.

• Use demo apps as example implementations.

• Create a procedure to uniformly educate everyone who will be using the system.

• Review and help teams to improve their microservices to run optimally on the standardized platform.

• Include recommended languages and tools.

• Just because we are being creative and doing experiments doesn’t mean we should not be doing architecture.

Consequently

Components are consistent across all teams and projects. There is a clear understanding regarding the platform and agreement over preferred application architecture styles. The current state of the platform is known, and it is open for improvements.

• + Easier to improve and maintain.

• + Easier to onboard new devs.

• − May limit freedom of choice.

In This Context

When architecture is very complex and difficult to replicate—or, if as sometimes happens, a piece is entirely missing—a team can struggle to have a quick and effective conversation about technical solutions. Describing complex technical discussions using words can be time-consuming and confusing, resulting in many misunderstandings later on. This may lead to deviations in implementation, which can harm product quality and make maintenance more difficult.

• Different people grasp information in different ways.

• Visualization helps people think more creatively.

• Common language (visual or verbal) helps save time and increase the volume of information during technical discussions.

Therefore

Draw the high-level architecture on a whiteboard and teach all team members how to repeat the drawing.

Use the drawing consistently in team discussions and in all internal documents. Create similar drawings for each subcomponent recursively.

• Create simple elements that are easy to draw.

• Limit the drawing to 20 elements or so.

• Create a digital version that resembles the drawings.

• Use the same graphical language for subcomponents.

• Keep a version of the drawings centrally available for easy reference, and keep it updated.

Consequently

Everyone in the team can draw parts of the architecture in seconds. There is a common visual language as a basis for improved collaboration.

• + Consistent visuals are used throughout the project.

• + Internalized understanding of the architecture’s components and how they relate to each other.

• − Standardized representation of the architecture can crimp creative thinking in the team and lead to conformity in early stages of the product development.

In This Context

Teams onboarding to cloud native don’t know the tools or technologies and typically receive poor onboarding materials. At best, this leads to wasted time, and at worst, teams are forced to create their own cloud native practices, which are uninformed, not designed for this platform, and not uniform.

• There are limited publicly available materials.

• People will use known techniques if they are not provided with clear guidance to new ones.

• If teams are onboarded with insufficient training, they will overload the support team with requests for help.

• People tend to accept default choices, so giving them good defaults increases overall quality.

Therefore

Provide developers onboarding to cloud native everything they need to start working immediately.

Optimally, new developers should be able to commit their first change and deploy it to the test environment on the first day following the onboarding.

• This cloud native “starter kit” of materials should include tool configurations, version control repository, CI/CD pipelines, demo applications for practice, target platform description, trainings, and more.

• All of this needs to be prepared before the next step of onboarding.

Consequently

Cloud native practices are adopted as the Core Team has planned, and there is consistency.

• + Less work and fewer problems for the Core Team after onboarding, because the newly onboarded developers have the tools and confidence to solve their own problems.

• − Less freedom for learning by doing for the dev teams.

In This Context

Teams tend to delay setting up security until relatively late in a project. Cloud native, however, requires many new tools and security techniques, and teams are typically not proficient in working with distributed systems. Waiting to implement security features just before the platform is ready to go live leads to either poor security in production or significant delays while good security measures are finally taken.

• Security in distributed systems cannot be ensured by perimeter security.

• It is more difficult to add security after the fact.

• The cloud native world requires many new and unfamiliar tools and methods.

Therefore

Build the MVP as a highly secure system from Day One.

Embed security practices in the Startup Pack and the Demo Application and run security tests as an integral part of the testing suite during the CI/CD process. This will allow the needed time to create security practices and provide examples for the teams onboarding to the platform.

• Provide good-quality and ongoing security training.

• Ensure that automated security testing is in place.

• Review the security of every tool in your cloud native system.

• Use best practices to secure containers, clusters, APIs, access rights, etc.

Consequently

Security is a high priority from the start and baked in throughout the platform. The Build-Run teams and Platform Team have clear guiding principles for creating secure Distributed Systems.

• + There is no extra cost to add security from the start.

• + The team is proficient in distributed security.

In This Context

Re-architecting a large monolith, built over many years or even decades, is a massive project that can take years. Some companies try to do it all at once, but rewriting a large monolithic application from scratch also carries great risk. You cannot start using the new system until it is developed and functioning as expected, but your company has little cloud native experience or knowledge to get this done. Building a new system from scratch will take a year or (likely) longer. While it is under construction there will be minimal enhancements or new features delivered on the current platform and so the business risks losing market share.

There is also a large risk of doing it all wrong in your first attempt. If the first project covers the entire application, then it will be very difficult to step back and start over due to sunk-cost fallacy—even if doing so is the best solution.

• Teams don’t yet know how to split the monolith into microservices.

• The first time you do something you are going to make mistakes; it is a learning experience rather than an execution.

• Monoliths hide unexpected problems inside their huge size.

• A well-scoped migration can handle problems as they emerge, but if you are trying to do everything all at one time, they will cripple the initiative.

• 20/80 principle: it takes 20% of the time to get 80% finished, and then 80% of the time to finish the last 20% (and the last 1% will take as much time as the first 99%, so keep that 1% on the mainframe—see Lift and Shift At the End).

Therefore

Once the cloud native platform is ready, take small pieces of the monolithic application and, one at a time, re-architect them and then move them to the new platform.

The business value of new functionality is achieved much more quickly, and the cloud native architecture of loosely coupled services means future refactoring work will be simple. This is the cloud native version of Martin Fowler’s classic strangler pattern.

• Going piece by piece and over an extended period of time is key.

• First have the final functional platform in place.

• Give priority to pieces that change frequently and those that are easy to extract.

• Create demo apps.

• Document a simple way for migrating pieces to the platform to make the process consistent, replicable, and as quick and effortless as possible.

• Leave the things that are running but not changing at all behind on the old system, and move them at the very end.

Consequently

There is a mixed environment of old and new applications working together. The team is getting better at re-architecting the pieces of the monolith.

• + A plan is in place for moving pieces over time.

• − Some teams are still working in the old environment—the entire company is not all moving to Kubernetes on Day One.

• − Two different operational models are in place, which can create its own set of problems.

In This Context

Automation is essential for success in cloud native, but people tend to try to create a full and automated solution in the beginning before real pain points are uncovered (taking an academic approach rather than experimental). This leads to automation of the wrong thing when the problem is not fully understood. Or, paraphrasing Bill Gates, who describes this conundrum as “Crap in, crap out, only faster.”

• Universities teach to solve the problem in the “right” way.

• Engineers prefer automation versus manual work.

Therefore

Understand the problem well, create a solution, make it work, and only then automate, scale, optimize, and improve.

Before automating anything, first solve the problem manually. The team needs to see the solution by doing it manually for a bit to experience and identify the pain points. Focus first on low-hanging fruit of automation (i.e., tasks that demand a lot of human time and are easy to automate).

• Run the process manually a few times.

• Create a blueprint (a document with steps).

• Do crude automation first (experiments, then an MVP version).

• Optimize and scale.

• Continually improve.

Consequently

Only the right things get automated. All the important and time-consuming tasks get automated eventually.

• + Scaled work becomes a well-understood process.

• − Process is manual for a while.

In This Context

Many development teams tend to create their own tools/solutions even when reasonable alternatives are available. Custom solutions are expensive and slow to build, difficult to maintain, and quick to become outdated. They don’t take advantage of developments in the industry, and the eventual cost is high.

• Internal devs are most content to build core products.

• Tools that are not core business rarely get full attention.

• Everything that is not business logic or user interaction is not your core business.

• Every off-the-shelf product is a core business for the company or the community that makes it.

• Cloud native ecosystem is growing very fast.

• Many engineers think “they know better.”

• Open source attracts many devs.

Consequently

The team can focus on core business.

• + New functionality is constantly introduced with third-party product releases.

• + Quality of off-the-shelf products is typically higher.

• + There is external product/user support.

• + Easier to hire people when using common tools.

• − Some problems are too specific for any off-the-shelf solution to address.

• − Third-party products are often expensive.

• − Less control over functionality.

In This Context

There is no practical way to predict how customers will respond to changes. In the absence of actual customer usage data, design and implementation decisions must be based on guesswork and intuition. Since our intuition is not perfect and full of biases, we may not get the best possible results.

• People don’t always know what they need/want.

• Delivering fast without adjusting the product based on measurement and feedback will change nothing.

• It’s impossible to make logical decisions in an unknown environment.

• There are unlimited variations and combinations of possible solutions.

Therefore

Prepare multiple versions of a solution to a challenge/problem and present them to randomized small portions of the client base. Measure the customer response in terms of value for them, and based on that choose the solution.

• Famous Google example of testing 41 shades of blue for its toolbar¹ to find which inspired the most consumer clicks—because, for Google, clicks equal revenue.

• The Obama campaign raised more money by using A/B testing to choose the most effective messaging.

• Need to provide businesspeople with the opportunity to run the A/B test experiments themselves, and an accessible way for them to do so.

Consequently

You now have an easy way to test assumptions live in a real-world environment.

Instead of making guesses or assumptions regarding which of two ideas, implementation strategies, etc., is better, a team can quickly put together a simple prototype with two or more versions and release them to small subsets of real customers. Based on customer response, the team can choose the more appropriate/preferred solution. This way many options could be tested while costs are ultimately saved because only the best option is ever fully implemented.

• + Customers see response to their needs.

• + If something doesn’t work, you can easily roll back to previous version.

• − Human insight might be sidelined when user response data is followed blindly.

• − Some innovative solutions take time to gain customer acceptance; A/B testing brings a risk of prematurely canceling such products.