We're probably already working with Puppet or we are planning to use it in the near future. In any case, we have to face the daunting task of automating the configuration and reproducing the state of a variable number of different servers.
Using Puppet for server infrastructure involves analysis, planning, and decisions.
In this chapter, we will review the different steps to be considered in our journey towards infrastructure automation:
- The possible scenarios and approaches we might face: new setups, migrations and updates of existing systems, and Puppet code refactoring
- Patterns for Puppet deployments; a step-by-step approach divided in reiterated phases: information gathering, priorities definition, evaluation on how to proceed, coding, testing, applying the changes, and verifying the reports
- The cultural and procedural changes that configuration management involves
Puppet is a tool that offers a world of order, control, and automation; we have learnt it, we know at least the basics of how it works and we want to install it on our servers.
We might have just a few dozens or several thousand systems to manage, they might be mostly similar or very different from one another, have very few things to be configured or a large number of resources to deal with.
We might manage them by hand or, maybe, already have some automation scripts or tools, Puppet itself included, that help us in installing and maintaining them but do that in a suboptimal way that we want to fix.
We might have a brand new project to build and this time we want to do the right thing and automate it from the start.
The number of variables here are many, every snowflake is unique, but there's a first basic and crucial distinction to make; it defines where our work will be done:
- On a brand new servers infrastructure
- On existing systems, either managed manually or already (semi-automated)
- On an infrastructure already managed by Puppet, to be heavily refactored
This distinction is quite important and can affect heavily how quickly and safely we can work with Puppet.
If we work on a startup or a new project, we have the opportunity to create our infrastructure from scratch, which is often a preferred, easier, more comfortable, and safer option for various reasons:
- More freedom on architectural choices: Starting from scratch we are not bound to existing solutions, we can concentrate on the tools and languages that better fit our current and future needs without caring about backwards compatibility and existing legacies. This freedom does not strictly relate to just our Puppet activities and it allows us to evaluate technologies also based on how easily they can be automated.
- Brand new operating system and application stacks, possibly homogeneous: We can choose a recent version of our preferred OS and, even better, we can start from a standardized setup where all servers have the same OS and version which definitely makes our life with Puppet easier and our infrastructure more coherent and easier to manage.
- Clean setups: We don't have to be concerned about, reproduce, reverse engineer, and maintain existing systems, so we know exactly the configurations needed on them. There is always the risk of messing with them in the future, but introducing a configuration management system at the beginning definitely helps in keeping things tidy and homogeneous for a long time.
- Sound design from the foundations can be defined: We can setup our servers using all our expertise on how to do things in a good way and we can generally benefit from the fact that newer versions of applications generally make their setup easier and easier, for example with updated packages, options that make configuration easier or allow splitting files in
conf.ddirectories. - No mess with current production: Being a new project, we don't have to cope with existing running systems and we don't risk introducing disrupting changes while applying or testing our Puppet configurations.
- Faster and smoother deployment times: These are the natural consequences of being free from working on existing setups in production. We don't have to worry about maintenance windows, safe runs, double checks, remediation times, and all the burdens that can greatly reduce the pace of our work.
When we need to automate existing systems things are quite different, as we have to cope with stratifications of years of many people's work with different skills under variable conditions. They may be the result of a more or less frantic and inorganic growth and evolution of an IT infrastructure, whose design patterns may have been changed or evolved with time. This might not be the case for everybody, but is quite common with infrastructures where a configuration management tool has not been steadily introduced.
We will probably have to manage:
- A heterogeneous mess of different operating systems and versions, deployed over many years. It is indeed very difficult to have OS homogeneity on a mature infrastructure. If we are lucky, we have different versions of the same OS. This is because they were installed at different times, but more likely we have to manage a forest of variegated systems, from Windows to Unix to almost any conceivable Linux distribution, installed according to the project software needs, commercial choices, or personal preferences of the sys admins in charge at the time.
- Heterogeneous hardware, from multiple vendors and of different ages, is likely to be present in our infrastructure. We have probably already made great progress in recent years, introducing virtualization and maybe a large effort of consolidation of the previous hardware mess has already been done. Still, if we haven't moved everything on the cloud, it's very likely that our hardware is based on different models possibly by different vendors.
- Incoherent setups, with systems configured by different people who may have used different logic, settings, and approaches also for the same kind of functionality. Sys admins might be complex and arcane dudes, who like to make great things in very unexpected ways. Order and coherency does not always find a place on their crazy keyboards.
- Uncertain setup procedures whose documentation may be incomplete, obsolete, or just non-existent. Sometimes we have to reverse engineer them, sometimes the offered functionality is easy to understand and replicate, other times we may have an arcane black box, configured at the dawn of time by people who don't work anymore in the company, which nobody wants to touch because it's too critical, fragile or simply indecipherable.
- Production systems, which are always delicate and difficult to operate. We might be able to work only on limited maintenance windows, with extra care of the changes introduced by Puppet and their evaluation on operations. We have to make choices and decisions for each change, as the homogeneity that Puppet introduces will probably change and leverage some configurations.
- Longer Puppet rollout times are the obvious side effects when we work on existing systems. These have to be taken into account , as they might influence our strategy for introducing Puppet.
We have two possible approaches to follow when dealing with existing systems. They are not mutually exclusive, we can adopt both of them in parallel, evaluating for each node which one is the most fitting:
A migration approach involves the installation and configuration, via Puppet, of new servers as replacements for existing ones: we build a system from scratch, we configure it so that it reproduces the current functionalities and, when ready, we promote it to production and decommission the old machine.
The main advantages of such an approach, compared to a local update, are as follows:
- We can operate on new systems that are not (yet) in production, so we can work faster and with less risk. There is no need to worry about breaking existing configurations or introducing downtimes to production.
- The setup procedure is done from scratch, so we're validating whether it's fully working and whether we have full Puppet coverage of the required configurations. Once a system has been successfully migrated and fully automated, we can be confident that we can consistently repeat its setup any time.
- We can test our systems before decommissioning the old ones and a rollback is generally possible if something goes wrong when we make the switch.
- We can use such an occasion to rejuvenate our servers, install newer and patched operating systems and generally have a more recent application stack (if our migrated applications can run on it), although it's always dangerous to do multiple changes at the same time.
There is a critical moment to consider when we follow this approach: when we shift from the old systems to the new ones.
Generally this can be done with a change of IP on the involved systems (with relevant ARP cache cleanups on peer devices on the same network), or of DNS entries or in configurations of load balancers or firewalls; all these approaches are reversible being mostly done at network level and rollback procedures can be automated for a quick recovery in case of problems.
Whatever the approach used we'll have the new systems doing what was previously done by old and different ones, so we might have untested and unexpected side effects. Maybe without the commonly used features of an application, or scheduled jobs which might manifest themselves at later and unexpected times.
So it makes sense to keep the migrated systems under observation for some days and double check if, besides the correct behavior of the mains services they provide, other functionalities and interactions with external systems are preserved.
Generally the shift is easier if no data is involved; much more attention is definitely needed when migrating systems that persist data, such as databases.
Here the scenarios might be quite different and the strategy to follow for data migration depends on the software used, the kind and size of data, if it's stored locally or via a network, and other factors that definitely are out of the scope of this book.
When we prefer or have to install Puppet on existing running systems, the scenario changes drastically. Different variables come into play and the approaches to follow are definitely alternative.
In these cases, we will have:
- Probably different OS to manage, some of them might be rather ancient and here we might have problems in installing a recent version of Puppet, so it makes sense to verify if the version of the client is compatible with the one of the Puppet Master.
- Undetermined setup and configuration procedures that we might have to gradually cover with Puppet with the hope of managing whatever is needed.
- Puppet deployments on running production systems so we have to manage the risk of configuration changes delivered by Puppet. This makes our lives harder and more dangerous, especially if we don't follow some precautions (more on this later).
- Manual configurations accumulated over time will probably determine systems which are not configured in the same way and we'll realize this when Puppet is deployed. What's likely to happen is that many unexpected configurations will be discovered and, probably, wrong or suboptimal settings will be fixed.
What has been described in the previous paragraphs might have been a common situation some years ago, now times have definitely changed and people have been using configuration management tools and cloud computing for years.
Actually, it is likely that we might already have a Puppet setup and we are looking for ways to reorganize and refactor it.
This is becoming a quite common situation: we learn Puppet while we work on it and during the years of usage patterns evolve, the product offers new and better ways to do things, and we understand better what has worked and what hasn't.
So we might end up in scenarios where our Puppet architecture and code is in different shapes:
- We have some old code that needs fixing and refactoring in a somewhat limited and non-invasive way: no revolutions, just evolutions
- Our setup works well but is based on older Puppet versions and the migration needs some more or less wide refactoring (a typical case here is to do major version upgrades with backwards incompatibilities)
- Our setup more or less does its job but we want to introduce new technologies (for example Hiera) or reorganize radically our code (for example, introducing well separated and reusable application modules and/or the roles and profiles pattern)
- Our Puppet code is a complete mess, it produces more problems than the ones it solves and we definitely need a complete rewrite
Different cases may need different solutions, they can follow some of the patterns we describe for the setup of new infrastructures, or the migration or update of existing ones, but here the work is mostly done on the Puppet Master rather than the clients, so we might follow different approaches:
- Refactoring of the code and the tools keeping the same Puppet Master. If there are few things to be changed, this can be based on simple evolutionary fixes on the existing code base, if changes are more radical we can use Puppet's environments to manage the old and new codebase in parallel and have clients switching to the new environment in a gradual and controlled way.
- Creation of a new Puppet Master can be an alternative when changes are more radical, either for the versions and tools used or for the complete overhaul of the codebase. In these cases we might prefer to keep the same certificate authority and Puppet Master's certificate or create a new one, with the extra effort of having to re-sign our clients' certificates. Such an approach actually may involve both the creation of new systems that point directly to the new Puppet Master, or the reconfiguration of existing ones. In both cases we can follow the same gradual approach described for other scenarios. When we need to upgrade the Puppet Master a few rules have to be considered:
- The Puppet Master version should always be newer than the version on the clients. The client-server compatibility between single major releases is generally guaranteed until Puppet 4. 2.7 servers are backwards compatible with 2.6 clients and 3.x servers are compatible with 2.7 clients. In most cases backwards compatibility extends for more versions, but that's not guaranteed. Starting on Puppet 4 compatibility is only guaranteed for packages in the same package collection, it will make it easier to be sure that even different versions will work well together if they belong to the same collection, but it breaks compatibility with older versions.
- It's recommended to upgrade from one major version to the next one, avoiding wider jumps. Starting from version 3 a semantic versioning scheme has been introduced (earlier major versions can be considered 2.7, 2.6, 0.25, 0.24) and a major version change, now expressed by the first digit, can have backward incompatibilities regarding the language syntax and other components that are not related the client-server communication. Puppet Labs, however, has the good habit of logging deprecation notices for features that are going to be removed or managed in different ways, so we can have a look on the Puppet Master's logs for any similar notice and fix our code before making the next upgrade step.