Automating releasing and testing is a common task. Both tend to have a lot of steps, so creating a consistent place and way to run tests and releases can improve overall reliability. Without this, a build or test could happen on a differently configured machine each time and introduce unknown variables. Incomplete configuration, steps done out of order, or incorrectly typed commands have caused many an outage, so automating this process is incredibly valuable.
Automation of testing and releasing usually requires automating three things:
Automation exists because people are forgetful. All three of these things can be automated using similar tools. Almost all of these tools are essentially job runners that run on an event queue. That queue can be filled by actions, such as code committed to a version control repository or a button click. The queue can also be filled with events based on time, like an event every hour to run integration tests. There are many tools that can do this and many are hosted services. The top three that I have used and enjoyed are:
- Travis CI
- Circle CI
- Jenkins
Each has different features and is changing constantly, but all of them operate on the same methodology—reading events and running jobs. Travis and Circle are hosted services that are configured with YAML (a whitespace sensitive markup language). Jenkins is an open-source piece of software that people can run and maintain themselves. It's a pain to configure and run, but it's very powerful if a hosted service doesn't meet your needs.
How you design your automation is up to you and your developer workflow. Sometimes you will have to build twice—once with options for testing and once with production optimizations. Sometimes you will need to build a binary and then build a package around it to distribute it. You can also cache your builds, so maybe your test runner just needs to go and get the most recently built binary to do the test. Most services even support defining workflows, so you can do many different things in parallel or in sequence.
The point is, just automate your tests and releases. You are going to have an emergency and need to push out a quick hot fix, or someone new is going to need to push out a deploy and they aren't going to know what to do. By automating these steps, you can make it so all the person has to do is trigger an event and they can just sit back and watch things happen.
You can also add automation to other places in the stack. One common trick is to have things like lint checks run and automatically comment on code reviews. You can also move a lot of your automation into chat. This allows for things like sending group chat messages that tell the service to deploy, thus informing everyone that a deploy is happening and also making the deploy happen at the same time.
One last subject I would like to talk about for testing and releasing is the continuous category. There are a lot of words that people like to put after continuous, and often they cause arguments that are incredibly pedantic. This is usually just describing the three steps I described above in the Automation section. Some call the whole thing continuous integration, which is what most of these job-runner service providers name themselves. I have also read many think pieces, each trying to define very subtle differences, none of which matter. Terms I have heard that all seem to be very similar are:
- Continuous integration
- Continuous release
- Continuous deployment
- Continuous delivery
- Deploy on green
The overarching point is that things are tested, built, and delivered all with automation. Some systems add human checks into pipelines. Some allow for automated rollbacks if issues arise. Some have buttons that need to be clicked or test conditions that need to be passed. I highly recommend staying out of pedantic arguments about naming your integration and instead focusing on finding the workflow that works best for your organization and the levels of risk your coworkers are comfortable with.
That being said, each of these ideas and their subtle differences can be useful. Continuous integration is the idea that on every commit a set of tests should be run to determine whether code has bugs or not. Continuous deployment is the idea that every commit should be deployed to production automatically. Continuous delivery is similar but instead focuses on the idea that once you decide to deploy code, the entire deployment happens automatically. Deploy on green is the idea that continuous deployment should happen whenever the tests pass.
All of these different definitions (and the many more I didn't touch on) each have long books and blog posts written on them. More often than not, especially for smaller organizations, knowing the difference doesn't matter. Instead, focus on consistency. As you discover problems with your existing processes, use the ideas in those long writings to figure out ways you can improve your process. It's rarely worth starting from scratch, instead just keep iterating as you would with any piece of software.