Puppet doesn't organize things in modules exclusively. There is a higher-level unit called environment that groups and contains the modules. An environment mainly consists of:
- One or more site manifest files
- A
modulesdirectory - An optional
environment.confconfiguration file
When the master compiles the manifest for a node, it uses exactly one environment for this task. As described in Chapter 2, The Master and Its Agents, it always starts in manifests/*.pp, which form the environment's site manifest. Before we take a look at how this works in practice, let's see an example environment directory:
/opt/puppetlabs/code/environments
production
environment.conf
manifests
site.pp
nodes.pp
modules
my_app
ntpThe environment.conf file can customize the environment. Normally, Puppet uses site.pp and the other files in the manifests directory. To make Puppet read all the pp files in another directory, set the manifest option in environment.conf:
# /opt/puppetlabs/code/environments/production/environment.conf manifest = puppet_manifests
In most circumstances, the manifest option need not be changed.
The site.pp file will include classes and instantiate defines from the modules. Puppet looks for modules in the modules subdirectory of the active environment. You can define additional subdirectories that hold the modules by setting the modulepath option in environment.conf:
# /opt/puppetlabs/code/environments/production/environment.conf modulepath = modules:site-modules
The directory structure can be made more distinctive:
/opt/puppetlabs/code/environments/
production
manifests
modules
ntp
site-modules
my_appPuppet uses the production environment by default. This and the other environments are expected to be located in /opt/puppetlabs/code/environments. You can override this default by setting the environmentpath option in puppet.conf:
[main] environmentpath = /etc/local/puppet/environments
With Puppet 3, you are required to set this option, as it is not yet the default. It is available from version 3.5. The earlier versions needed to configure the environments right in puppet.conf with their respective manifest and modulepath settings. These work just like the settings from environment.conf:
# in puppet.conf (obsolete since 4.0) [testing] manifest = /etc/puppet/environments/testing/manifests modulepath = /etc/puppet/environments/testing/modules
For the special production environment, these Puppet setups use the manifest and modulepath settings from the [main] or [master] section. The old default configuration had the production environment look for the manifests and the modules right in /etc/puppet:
# Obsolete! Works only with Puppet 3.x! [main] manifest = /etc/puppet/site.pp modulepath = /etc/puppet/modules
The sites that operate like this today should be migrated to the aforementioned directory environments in /opt/puppetlabs/code/environments or similar locations.
Downloading existing modules is very common. Puppet Labs hosts a dedicated site for sharing and obtaining the modules - the Puppet Forge. It works just like RubyGems or CPAN and makes it simple for the user to retrieve a given module through a command-line interface. In the Forge, the modules are fully named by prefixing the actual module name with the author's name, such as puppetlabs-stdlib or ffrank-constraints.
The puppet module install command installs a module in the active environment:
root@puppetmaster# puppet module install puppetlabs-stdlib
The current release of the stdlib module (authored by the user puppetlabs) is downloaded from the Forge and installed in the standard modules' location. This is the first location in the current environment's modulepath, which is usually the modules subdirectory. Specifically, the modules will most likely end up in the environments/production/modules directory.
Tip
The stdlib module in particular should be considered mandatory; it adds a large number of useful functions to the Puppet language. Examples include the keys, values, and has_key functions, which are essential for implementing the proper handling of hash structures, to name only a few. The functions are available to your manifests as soon as the module is installed - there is no need to include any class or other explicit loading. If you write your own modules that add functions, these are loaded automatically in the same way.
With all the current versions of Puppet, you should make it a habit to put all the manifest code into modules, with only the following few exceptions:
- The
nodeblocks - The
includestatements for very select classes that should be omnipresent (the most common design pattern does this in the so-called base role, however; see Chapter 8, Separating Data from Code Using Hiera, for the Roles and Profiles pattern) - Declarations of helpful variables that should have the same availability as the Facter facts in your manifests
This section provides details on how to organize your manifests accordingly. It also advises some design practices and strategies in order to test the changes to the modules.
You might find some manifests in very old installations that gather lots of manifest files in one or more directories and use the import statements in the site.pp file, such as:
import '/etc/puppet/manifests/custom/*.pp'
All classes and defined types in these files are then available globally.
It is far more efficient to give meaningful names to the classes and defined types so that Puppet can look them up in the collection of modules. The scheme has been discussed in an earlier section already, so let's just look at another example where the Puppet compiler encounters a class name, such as:
include ntp::server::component::watchdog
Puppet will go ahead and locate the ntp module in all the configured module locations of the active environment (path names in the modulepath setting). It will then try and read the ntp/manifests/server/component/watchdog.pp file in order to find the class definition. Failing this, it will try ntp/manifests/init.pp.
This makes compilation very efficient. Puppet dynamically identifies the required manifest files and includes only those for parsing. It also aids code inspection and development, as it is abundantly clear where you should look for specific definitions.
Each module should ideally serve a specific purpose. On a site that relies on Puppet to manage a diverse server infrastructure, there are likely modules for each respective service, such as apache, ssh, nagios, nginx, and so forth. There can also be site-specific modules such as users or shell_settings if the operations require this kind of fine-grained control. Such customized modules are sometimes just named after the group or the company that owns them.
The ideal granularity depends on the individual requirements of your setup. What you generally want to avoid are modules with names such as utilities or helpers, which serve as a melting pot for ideas that don't fit in any of the existing modules. Such a lack of organization can be detrimental to discipline and can lead to chaotic modules that include definitions that should have become their own respective modules instead.
Adding more modules is cheap. A module generally incurs no cost for the Puppet master operation, and your user experience will usually become more efficient with more modules, not less so. Of course, this balance can tip if your site imposes special documentation or other handling prerequisites on each module. Such rulings must then be weighed into the decisions about module organization.
Depending on the size of your agent network, some or many of your modules can be used by a large variety of nodes. Despite these commonalities, these nodes can be quite different from one another. A change to a central module, such as ssh or ntp, which are likely used by a large number of agents, can have quite extensive consequences.
The first and the most important tool for testing your work is the --noop option for Puppet. It works for puppet agent, as well as puppet apply. If it is given on the command line, Puppet will not perform any necessary sync actions, and merely present the respective line of output to you instead. There is an example of this in Chapter 1, Writing Your First Manifests.
When using a master instead of working locally with puppet apply, a new problem arises, though. The master is queried by all your agents. Unless all the agents are disabled while you are testing a new manifest, it is very likely that one will check in and accidentally run the untested code.
It is very important to guard against such uncontrolled manifest applications. A small mistake can damage a number of agent machines in a short time period. The best way to go about this is to define multiple environments on the master.
Besides the production environment, you should create at least one testing environment. You can call it testing or whatever you like. When using the directory environments, just create its directory in environmentpath.
Such an additional environment is very useful for testing changes. The test environment or environments should be copies of the production data. Prepare all the manifest changes in testing first. You can make your agents test this change before you copy it to production:
root@agent# puppet agent --test --noop --env testing
You can even omit the noop flag on some or all of your agents so that the change is actually deployed. Some subtle mistakes in the manifests cannot be detected from an inspection of the noop output, so it is usually a good idea to run the code at least once before releasing it.
Tip
Environments are even more effective when used in conjunction with source control, especially distributed systems such as git or mercurial. Versioning your Puppet code is a good idea independently of environments and testing—this is one of the greatest advantages that Puppet has to offer you through its Infrastructure as Code paradigm.
Using environments and the noop mode form a pragmatic approach to testing that can serve in most scenarios. The safety against erroneous Puppet behavior is limited, of course. There are more formal ways of testing the modules:
- The
rspec-puppettool allows the module authors to implement unit tests based onrspec. You can find more details at http://rspec-puppet.com/. - Acceptance testing can be performed through
beaker. You can refer to https://github.com/puppetlabs/beaker/wiki/How-To-Beaker for details.
Explaining these tools in detail is beyond the scope of this book.