Chapter 7. Use Continuous Integration
Fall seven times, stand up eight times
Japanese proverb
Best Practice:
-
Achieve Continuous Integration by setting up a CI server after you have version control, build automation, and automated tests in place.
-
Keep track of the build time and test time.
-
This improves the development process because it relieves developers of building the system and it shortens the feedback loop of issues.
The term continuous in Continuous Integration (CI) says it all: continuously integrate (merge) code. CI puts into practice the idea that code integration and building should be done as often as possible. The integration happens at each commit to a CI server that merges the commit into a central repository. Fortunately, most CI servers do more: after each commit, they perform a clean checkout of the branch that was changed, build it, and perform the supplied tests (unit, integration, and regression tests). Running automated tests is not “required” in order to be called CI, but why would you not? Using a CI server is a great opportunity to have your tests run automatically. You can even automate further and run additional scripts after successful builds, to achieve automated deployment. The latter is a topic for Chapter 8.
Motivation
Continuous Integration speeds up development because it is fast, has short feedback cycles, and is more reliable than manual integration. Thereby it allows for further automation steps such as “continuous delivery.”
CI Is Efficient
With test automation as part of the CI server, developers are relieved from tedious manual integration efforts and integration testing. Because automated tests are fast and effortless, you can frequently test your system. This controls the consequences of human error: instead of fearing that integration goes wrong, you merge and test code as often as possible.
Of course, when merge conflicts happen during a commit, the version control system asks the developer which change should take precedence over the other.
CI Gives Feedback Quickly and Thereby Speeds Up Bug Fixing
CI gives developers quick feedback on whether integration of code succeeds. Quick feedback means identifying problems early, when they are relatively easy to fix. Imagine that your team integrates all code at the end of the week, and it turns out that application builds fail because of integration errors. This can mean a lot of work and code reversions in order to get things to work again.
CI Is More Reliable Than Manual Merging
With CI, the building, merging, and testing process is consistent. As with all automated tasks, this consistency avoids a fair level of human error. A CI server should start processing a commit with setting up a clean checkout. That controls the risk that cached files or a corrupted database will influence the build process.
CI Facilitates Further Automation
Using a CI server paves the way for other kinds of “continuous” development such as “continuous delivery.” Continuous delivery shares the goals of automating the development pipeline as much as possible and achieving short iterations of deploying new code. Continuous delivery implies that you can deliver each code modification into production. This demands a high level of certainty on whether the system is working as intended. Therefore, it is a horizon for developers as to what can be achieved by automating tests and deployment.
How to Apply the Best Practice
Before you can achieve CI, you need three things: version control, automated builds, and (obviously) a CI server. In particular, to get the most out of CI you need automated tests: CI without testing will do nothing more than tell you whether the code can be merged and compiled.
Requirement: Version Control
To integrate code automatically after each commit, you need a version control system. This is also necessary for the CI server to initiate a build on the central code repository. See Chapter 4 for elaboration on the use of version control.
Requirement: Automated Builds
The CI server will need build scripts and configurations (i.e., configuration of code structure and conventions) in order to execute builds. It is important that you are able to synchronize configuration files from developers, not just for Integrated Development Environments (IDEs) but also for databases and any other elements that may be customized for their development environment. In this way, newly introduced dependencies are taken into account when the CI server executes new builds.
Requirement: CI Server
A CI server is a dedicated server that checks out code from version control, uses the build scripts and configurations to execute builds, and performs tests after building. For basic setups and mainstream technologies, most CI servers already have default build tools, so this can save you time. For less common technologies, you may need to adapt your CI server configuration to accommodate the desired build tools.
Requirement: Automated Tests
Strictly speaking, when you have set up the previous parts, you already have Continuous Integration in place. But really you should add automated testing to the mix. A CI server is the ideal tool to perform all necessary unit, integration, and regression tests immediately and report the results to developers.
Additions to the CI Server
Most CI servers allow the execution of additional scripts, depending on the outcome of a build. So you can add deployment scripts on successful builds when you have automated your deployment process (this is the subject of Chapter 8). When the build fails, the CI server notifies you and points you to the failing tests. Typically you will have a quick overview of which unit tests have failed on which lines, and whether this was due to, for example, unexpected values or a compile/syntax error.
Important Best Practices for CI
Continuous Integration works by virtue of regular commits and quick builds.
If commits are not done on a regular basis—say, at least daily—you can still end up with integration errors that are hard to fix. After all, after a day a lot of time has been spent writing code that may break the build. The person fixing it (not necessarily the one who “broke it”) has to go through a lot of code. So commits should be done regularly, as often as possible. This assumes that tasks can be broken down into pieces of work that take less than a day, but that is normally not a problem.
With branching, separate builds and tests become a greater concern. If developers use a new branch for every feature they implement, they should be built and tested through the CI server before they are merged into the master branch. This requirement can be configured in the CI server. In this way, you ensure that developers must have a successful build before they merge their new feature into the master branch. This is notably helpful when developers are making feature branches that need extensive testing (which may be the reason to use feature branches in the first place). This sounds trivial, but may be of particular concern when development teams are scattered geographically and/or have different release cycles.
When an application grows really large, it could be that build and test times increase to a point where it becomes impractical to always build after every commit. This should not happen too easily and it is a cue to revisit the design of the system to be able to build smaller parts. However, most CI servers can also cope with this technically. They allow parallel build jobs and dependency caching to speed up your builds. Of course, when changes are layered upon each other and build/test jobs take a long time, there is still a risk that after a certain change, all other changes need revisiting as well.
Controlling Continuous Integration
With the advantages of CI in speed, reliability, and flexibility, you would expect that your development process is speeding up with fewer issues. Fundamentally you would like to know whether and how productivity is improving. To have more confidence in a successful implementation of CI for your team, consider the following GQM model and metrics.
For the following metrics, the value is again mostly in large deviations, since these signal a problem. In agreement with the team you should determine a norm for which test and build durations are “short” enough.
-
Goal A: Understanding how CI influences development productivity.
-
Question 1: Do our developers receive timely feedback on integration errors?
-
Metric 1: Average feedback time after a commit. Compared to a situation without CI, the feedback time will be lower, as CI deals with building and testing automatically. Over time, expect the trend line to go up: as the codebase grows, more tests are written and builds take more time. Because codebase volume of systems in maintenance is typically a few percent per year, those times should not raise alarms. The interest of this metric is mostly in large deviations, which warrant an investigation: is new development somehow too complex? Are tests being skipped?
-
-
Question 2: Can we integrate/merge fast enough?
-
Metric 2a: Average build time (comparing builds of the master branch). Expect a downward trend of this metric if you are comparing with a situation without CI. Expect a slight upward trend line over time as the size of the codebase grows.
-
Metric 2b: Average test run time per type of test. Expect this metric to be fairly stable, assuming that the different test types (e.g., unit tests, sanity checks, end-to-end tests) are internally consistent (i.e., for each test type, their tests are similar in size, complexity, etc.).
-
-
In Question 2, what is “fast” is typically a comparison with the baseline. Primarily, the feedback time depends on server processing time. So if the processing times are out of control, it will influence all of the above metrics. You will then need to speed up the CI process first before gaining benefits of quick feedback. Beware that speeding up build time at the expense of creating a clean environment may lower reliability of test results.
An easy way to see the progress of your CI server is to show the average build and test times per week. We show a simplified example in the following figure, where results are aggregated per week. Figure 7-1 shows an example chart that tracks the build and test time on a weekly basis.
Figure 7-1. Trend line of build and test times
Common Objections to Continuous Integration Metrics
Typical objections to using CI metrics concern whether its processing time can be controlled and how one should deal with expectations of who will fix the build. They are discussed in this section.
Objection: We Cannot Get Control Over Our Build Time
“My CI server reinstalls all dependencies when a new build is available, so we cannot keep our build time low enough.”
We do not wish to be puritans: if reinstalls keep you from doing regular builds, do not reinstall everything. For instance, a database does not have to be reinstalled every time you do a build as long as you clean and refill the database with data that is consistent with other environments. Of course, the ideal is that the test data realistically represents data in the production environment. Or you could opt for a special nightly build that carries out reinstallation of your dependencies. For added certainty it is more important to be sure that builds in the acceptance environment reinstall everything.
Objection: My Colleague Broke the Build, Not Me
“After my commit, the new build was broken, but I am sure my code is perfect. It is only because the code of my colleague in the other room does not align with mine that the build failed.”
A broken build is a broken system, so this is an urgent matter. These are the situations that can be solved by having good agreements about what the leading principles of build success are. If you have agreed as a team that the tests determine build success, then your colleague committed code that has passed the test, so your code should be revised. If for some reason the (unit) tests are no longer adequate, then they should have been revised in the latest commit. Who will do the fix is a matter of agreement. Effective teams share responsibilities: if the task seems small, any developer with available time should feel responsible to take it up and fix it.
Metrics Overview
As a recap, Table 7-1 shows an overview of the metrics discussed in this chapter, with their corresponding goals.
| Metric # in text | Metric description | Corresponding goal |
|---|---|---|
CI 1 |
Commit feedback time |
Team productivity |
CI 2a |
Build time |
Team productivity |
CI 2b |
Test run time |
Team productivity |
When you have CI in place in a stable manner, you have already made great progress toward continuous delivery and automation. In the following chapter, we will look at automated deployment: automating manual configuration of environments and their deployment.