Once we introduce a tool like Puppet in our infrastructure, everything changes and we should be well aware of this.
There's a wonderful term that describes what configuration management software like Puppet or Chef involve: Infrastructure as code; we define our IT infrastructure with formal code, the configurations of our servers, the procedures to set them up, whatever is needed to turn a piece of bare metal or a blank VM in to a system that provides services for our purposes.
When we can use a programming language to configure our systems, a lot of powerful collateral effects take place:
- Code can be versioned with an SCM: The history of our commits reflects the history of our infrastructure: we can know who made a change, when and why. There's a huge intrinsic value in this; the power of contextual documentation and communication. Puppet code is inherently the documentation of the infrastructure, and the commits log reflects how this code has been deployed and evolved. It quickly communicates in a single place the rationale and reasons behind some changes much better than sparse e-mail, a wiki, phone calls, or direct voice requests. Also for this reason it is quite important to be disciplined and exhaustive when doing our commits. Each commit on the Puppet code base should reflect a single change and explain as much as possible about it, possibly referring to relevant ticket numbers.
- Code can be run as many times as wanted: We have already underlined this concept in Chapter 1, Puppet Essentials of this book, but it's worth further attention. Once we express with code how to setup our systems we can be confident that what works once works always: the setup procedure can be repeated and it always delivers the same result (given we have correctly managed resources dependencies). There's a wonderful feeling we gain once we have Puppet on our systems: we are confident that the setup is coherent and is done how we expect it. A new server is installed and all the configurations we expect from it are delivered in a quick and automated way: there is no real need to double check if some settings are correct, if all the common configurations are applied as expected: in a mature and well tested Puppet infrastructure they are. There's more, though, a lot more: ideally we just need our Puppet codebase and a backup of our application data to rebuild from scratch, in reasonable times, our whole infrastructure. In a perfect world we can create a disaster recovery site from scratch in a few minutes or hours, given that we have quick and automated deployment procedures (in the cloud this is not difficult), 100% Puppet coverage of all needed configurations, and automated or semi-automated application deployments and restore facilities of our data. A completely automated disaster recovery site setup from scratch is probably not even needed. (It's much more probable that during a crisis we have all our sys admin working actively on an existing disaster recovery site). But whatever can be automated, can save time and human errors during the most critical hours.
- Code can be tested: We will review in the next chapter the testing methodologies we have at our disposal when writing Puppet code. However, here it is important to stress that the possibility to test our code allows us to preventively verify how our changes may affect the systems where we want to apply them and make them much more controlled and predictable. This is crucial for tools like Puppet that directly affect how systems operate and can introduce changes (and failures) at scale. We will soon realize that automating the testing of our Puppet changes is possible and recommended and can help us in being more confident on how to deliver our code to production.
- Code can be refactored and refined: The more we develop Puppet code the more we understand Puppet and how to make it fit better to our infrastructure. This is a learning process that inevitably has to be done by ourselves and on our infrastructure. We will definitely consider what we wrote a year or just few months ago inaccurate and inappropriate for new the technical requirements that have emerged or are just badly written. The good news is that, as with any type of code, this can be refactored and made better. How and what to fix in our code base should depend on how it affects our work: bad looking code that works and fulfills its duties has less refactoring priority than code that creates problems, is difficult to maintain or doesn't fit new requirements.
It should be clear by now that such a radical change of how systems are managed involves also a change in how the system administrator works.
The primary and most important point is that the sys admin has more and more code to cope with code. This involves that somehow he has to adopt a developer's approach to his work: using a SCM, testing and refactoring code might be activities that are not familiar to a sys admin but have to be tackled and embraced because they are simply needed.
The second radical change involves how systems are managed. Ideally, in a Puppet managed infrastructure we don't even need to SSH to a system and issue commands from the shell: any manual change on the system should be definitely avoided.
This might be frustrating and irritating, since we can quickly do things manually that with Puppet requires a lot more time (write the code, commit, test, apply. A trivial change you can make in five seconds by hand may take you minutes).
The point is that what is done on Puppet remains. It can be applied when the system is reinstalled and is inherently documented: a quick manual fix is easily and quickly forgotten and may make us lose a lot of time when there's the need to replicate it and there's no trace of how, where, and why it was done.
Many of the most annoying and dangerous issues people experience when Puppet is introduced are actually due to the lack of real embracement of the mind set and the techniques that the tools require: people make manual changes and don't implement them in manifests. Then, at the first Puppet run, they are reverted and for any problems that take place Puppet gets the blame. In other cases, there's the temptation to simply disable it and leave the system modified manually. This may make sense in exceptional emergency cases, but after the manual fix the change has to be integrated and Puppet should be re-enabled.
If this doesn't happen we will soon fill our infrastructure with servers where Puppet is disabled and the drift for the configured baselines gets wider and wider, making it more and more difficult, risky and time consuming to re-enable the service.
We should have realized by now that there are things that Puppet does well and things that are not properly in its chords. Puppet is great at managing files, packages and services, and is less effective in executing plain commands especially if they happen occasionally.
Once we start to introduce Puppet in our infrastructures we should start, where possible, to make things in a way that favors its implementation.
Some time ago in a conference open space I remember a discussion with a developer about how to make a Java application more manageable with Puppet. From his point of view a good API could be a solid approach to make it more configurable. All the systems administrators, and Puppet users, replied that actually that was far from being a nice thing to have: it is much better to have good old plain configuration files.
If we can express it with a file, we can easily manage it with Puppet.
Whatever requires the execution of commands, even if doable, is always considered with skepticism. Installation of software should definitely be done via the OS native packages. Even if most of us have done our definitions that fetch a tar ball, unpack it and compile it, nobody really likes it: that involves a procedural and not a declarative approach, it makes idempotence more difficult and is definitely worse to express with Puppet DSL.
Services should be managed following the OS native approach, be that init, systemd, upstart, or whatever. Managing them with the execution of a command in a custom startup script is definitely not Puppet friendly.
Configuration files from common stacks of applications should be leveraged. For example when we have to manage the Java settings on an application server it makes sense to manage them always in the same place; standardizing them is a matter of overall coherence and easier management.
Once we gain affinity and expertise with Puppet we quickly understand what are the things that can be done quickly and safely and what are the things that make our life harder, requiring more resources or quick and dirty patches.
It is our responsibility to discuss this with the developers and the stakeholders of an application to find a collaborative approach to its setup that makes it easier to be managed with Puppet while preserving the requested implementation needs.
During this chapter we have seen some characteristics that make working with Puppet a much better experience. Let's remark on some of these characteristics with insights about how its deficiencies can make our code more complex or less likely to behave in a deterministic way. Some of these characteristics depend on how the software is packaged, but others may depend on the software itself. Let's look at them:
- Use the standard packaging system of the OS used in your infrastructure. This way the
packageresource type can be used in a homogeneous way. If the software to be installed is not officially packaged for the needed packaging system it's important to evaluate the possibility of packaging it. This will allow you to simplify the implementation of its Puppet module. The alternative could probably be to use some custom scripts possibly wrapped around defines to call them in their use cases, something that adds complexity to the deployment. - Use composable configuration files. As seen in the previous section, should prefer good old plain text configuration files are preferred over other more complex systems based on APIs or databases. Puppet has very good support for deploying files based on static sources or templates, which makes it easy to deploy configuration. Sometimes the configuration of an application or service is too complex to be managed by only a template or static file. In these cases, we should expect this software to support composable configuration files by reading them from a
conf.ddirectory. Then our Puppet modules will contain the logic to decide what files to use in each role while keeping the files as simple as possible. - Install services disabled by default. When installing packages from official distributions services can be enabled and started by default, this can lead to unexpected behaviors. For example the installation can fail if the service starts by default in an already used port, or a package that installs multiple services could start one that we don't want running on our system. These errors will make puppet execution fail, leaving the node in an unknown state. It can also hide future problems as the code may correctly apply in a working system, but it will fail in a new node. Sometimes these problems can be worked around by deploying the configuration before installing the package, but then problems with users or permissions can arise. The ideal workflow for installing services with Puppet would be to install with
package, configure withfiles,and enable and/or start withservice. - Something that is sometimes complex to implement, but is very desirable, is online configuration reload. As we are seeing throughout this book, Puppet is in charge of keeping our infrastructure in an expected state. This implies that sooner or later we'll want to change the configuration of our services. One of the most used approaches is to modify a
fileand subscribe theserviceto it that will provoke a restart of the service. Sometimes this is not acceptable, as the service will drop connections or lose data. There are a couple of options to manage this scenario. One is to coordinate the restarts, but Puppet alone is not very good at doing this kind of coordination. The other option is to check if the software is able to receive signals requesting configuration reloads and use thehasreloadparameter of service resources. This last option is preferable when working with Puppet because it fits perfectly in the usual workflows, but the software has to support it. - Another desirable characteristic of Puppet-friendly software is client-side registration. When a service needs to know a collection of clients (or workers, backend), it uses to be easier to manage if the clients are the ones taking the initiative to join the cluster. Puppet also has support for the opposite scenario, but it uses to imply the use of exported resources and the need to reload the configuration in the service, which is not so straightforward.
In most of the cases we cannot take the decision to use software just because it's more Puppet friendly. But if we are managing our infrastructure with Puppet it's worth considering these characteristics. If we can choose between different options, we can add these arguments to the balance of the software that implements them. If we cannot choose, check if the software to be used can be used in a Puppet-friendly way even if it's not its default behavior.