We created and used modules up to this point when we installed and configured tuned
using the is_virtual
fact. We created a module called virtual
in the process. Modules are nothing more than organizational tools, manifests, and plugin files that are grouped together.
We mentioned pluginsync
in the previous chapter. By default, in Puppet 3.0 and higher, plugins in modules are synchronized from the master to the nodes. Plugins are special directories in modules that contain Ruby code.
Plugins are contained within the /lib
subdirectory of a module, and there can be four possible subdirectories defined: files
, manifests
, templates
, and lib
. The manifests
directory holds our manifests, as we know files
has our files, templates
has the templates, and lib
is where we extend Augeas, Hiera, Facter, and/or Puppet depending on the files we place there.
In this chapter, we will cover how to use the modulename/lib/facter
directory to create custom facts, and in subsequent chapters, we will see how to use the /lib/puppet
directory to create custom types.
The structure of a module is shown in the following diagram:
A module is a directory within the modulepath
setting of Puppet, which is searched when a module is included by name in a node manifest. If the module name is base
and our modulepath
is $codedir/environments/$environment/modules:$codedir/environments/$environment/dist:$codedir/environments/production/modules
, then the search is done as follows (assuming codedir
is /etc/puppetlabs/code
):
/etc/puppetlabs/code/environments/$environment/modules/base/manifests/init.pp /etc/puppetlabs/code/environments/$environment/modules/dist/base/manifests/init.pp /etc/puppetlabs/code/environments/production/modules/base/manifests/init.pp
Each module is expected to have an init.pp
file defined, which has the top-level class definition; in the case of our base example, init.pp
is expected to contain class base { }
.
Now, if we include base::subitem
in our node manifest, then the file that Puppet will search for will be base/manifests/subitem.pp
, and that file should contain class base::subitem { }
.
It is also possible to have subdirectories of the manifests
directory defined to split up the manifests even more. As a rule, a manifest within a module should only contain a single class. If we wish to define base::subitem::subsetting
, then the file will be base/manifests/subitem/subsetting.pp
, and it would contain class base::subitem::subsetting { }
.
Naming your files correctly means that they will be loaded automatically when needed, and you won't have to use the import
function (the import
function is deprecated in version 3 and completely removed in version 4). By creating multiple subclasses, it becomes easy to separate a module into its various components; this is important later when you need to include only parts of the module in another module. As an example, say we have a database system called judy
, and judy
requires the judy-server
package to run. The judy
service requires the users judy
and judyadm
to run. Users judy
and judyadm
require the judygrp
group, and they all require a filesystem to contain the database. We will split up these various tasks into separate manifests. We'll sketch the contents of this fictional module, as follows:
judy/manifests/groups.pp
, we'll have the following code:class judy::groups { group {'judygrp': } }
judy/manifests/users.pp
, we'll have the following code:class judy::users { include judy::groups user {'judy': require => Group['judygrp'] } user {'judyadm': require => Group['judygrp'] } }
judy/manifests/packages.pp
, we'll have the following code:class judy::packages { package {'judy-server': require => User['judy','judyadm'] } }
judy/manifests/filesystem.pp
, we'll have the following code:class judy::filesystem { lvm {'/opt/judy': require => File['/opt/judy'] } file {'/opt/judy': } }
judy/manifests/service.pp
:class judy::service { service {'judy': require => [ Package['judy-server'], File['/opt/judy'], Lvm['/opt/judy'], User['judy','judyadm'] ], } }
Now, we can include each one of these components separately, and our node can contain judy::packages
or judy::service
without using the entire judy
module. We will define our top level module (init.pp
) to include all these components, as shown here:
class judy { include judy::users include judy::group include judy::packages include judy::filesystem include judy::service }
Thus, a node that uses include judy
will receive all of those classes, but if we have a node that only needs the judy
and judyadm
users, then we need to include only judy::users
in the code.
Transferring files with Puppet is something that is best done within modules. When you define a file resource, you can either use content => "something"
or you can push a file from the Puppet master using source
. For example, using our judy
database, we can have judy::config
with the following file definition:
class judy::config { file {'/etc/judy/judy.conf': source => 'puppet:///modules/judy/judy.conf' } }
Now, Puppet will search for this file in the [modulepath]/judy/files
directory. It is also possible to add full paths and have your module mimic the filesystem. Hence, the previous source line will be changed to source => 'puppet:///modules/judy/etc/judy/judy.conf'
, and the file will be found at [modulepath]/judy/files/etc/judy/judy.conf
.
The puppet:///
URI source line mentioned earlier has three backslashes; optionally, the name of a puppetserver
may appear between the second and third backslash. If this field is left blank, the puppetserver
that performs the catalog compilation will be used to retrieve the file. You can alternatively specify the server using source => 'puppet://puppetfile.example.com/modules/judy/judy.conf'
.
Having files that come from specific puppetservers
can make maintenance difficult. If you change the name of your puppetserver
, you have to change all references to that name as well. Puppet is not ideal for transferring large files, if you need to move large files onto your machines, consider using the native packaging system of your client nodes.
Templates are searched in a similar fashion. In this example, to specify the template in judy/templates
, you will use content =>template('judy/template.erb')
to have Puppet look for the template in your modules' templates
directory. For example, another config file for judy
can be defined, as follows:
file {'/etc/judy/judyadm.conf': content => template('judy/judyadm.conf.erb') }
Puppet will look for the 'judy/judyadm.conf.erb'
file at [modulepath]/judy/templates/judyadm.conf.erb
. We haven't covered the Embedded Ruby (ERB) templates up to this point; templates are files that are parsed according to the ERB syntax rules. If you need to distribute a file where you need to change some settings based on variables, then a template can help. The ERB syntax is covered in detail at http://docs.puppetlabs.com/guides/templating.html. Puppet 4 (and Puppet 3 with the future parser enabled) supports EPP templates as well. EPP templates are Embedded Puppet templates that use Puppet language syntax rather than Ruby.
Modules can also include custom facts, as we've already seen in this chapter. Using the lib
subdirectory, it is possible to modify both Facter and Puppet. In the next section, we will discuss module implementations in a large organization before writing custom modules.
Modules must begin with a lowercase letter and only contain lowercase letters, numbers, and the underscore (_) symbol. No other characters should be used. While writing modules that will be shared across the organization, use names that are obvious and won't interfere with other groups' modules or modules from the Forge. A good rule of thumb is to insert your corporation's name at the beginning of the module name and, possibly, your group name.
While designing modules, each module should have a specific purpose and not pull in manifests from other modules and each one of them should be autonomous. Classes should be used within the module to organize functionality. For instance, a module named example_foo
installs a package and configures a service. Now, separating these two functions and their supporting resources into two classes, example_foo::pkg
and example_foo::svc
, will make it easier to find the code you need to work on, when you need to modify these different components. In addition, when you have all the service accounts and groups in another file, it makes it easier to find them, as well.
To start with a simple example, we will use Puppet's module
command to generate empty module files with comments. The module name will be example_phpmyadmin
, and the generate
command expects the generated argument to be [our username]-[module name]
; thus, using our sample developer, samdev
, the argument will be samdev-example_phpmyadmin
, as shown here:
[samdev@stand ~]$ cd control/dist/ [samdev@standdist]$ puppet module generate samdev-example_phpmyadmin We need to create a metadata.json file for this module. Please answer the following questions; if the question is not applicable to this module, feel free to leave it blank. Puppet uses Semantic Versioning (semver.org) to version modules. What version is this module? [0.1.0] --> 0.0.1 Who wrote this module? [samdev] --> What license does this module code fall under? [Apache-2.0] --> How would you describe this module in a single sentence? --> An Example Module to install PHPMyAdmin Where is this module's source code repository? --> https://github.com/uphillian Where can others go to learn more about this module? [https://github.com/uphillian] --> Where can others go to file issues about this module? [https://github.com/uphillian/issues] --> ---------------------------------------- { "name": "samdev-example_phpmyadmin", "version": "0.0.1", "author": "samdev", "summary": "An Example Module to install PHPMyAdmin", "license": "Apache-2.0", "source": "https://github.com/uphillian", "project_page": "https://github.com/uphillian", "issues_url": "https://github.com/uphillian/issues", "dependencies": [ {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"} ] } ---------------------------------------- About to generate this metadata; continue? [n/Y] -->y Notice: Generating module at /home/samdev/control/dist/example_phpmyadmin... Notice: Populating templates... Finished; module generated in example_phpmyadmin. example_phpmyadmin/manifests example_phpmyadmin/manifests/init.pp example_phpmyadmin/spec example_phpmyadmin/spec/classes example_phpmyadmin/spec/classes/init_spec.rb example_phpmyadmin/spec/spec_helper.rb example_phpmyadmin/tests example_phpmyadmin/tests/init.pp example_phpmyadmin/Gemfile example_phpmyadmin/Rakefile example_phpmyadmin/README.md example_phpmyadmin/metadata.json
The previous command generates metadata.json
and README.md
files that can be modified for your use as and when required. The metadata.json
file is where you specify who wrote the module and which license it is released under. If your module depends on any other module, you can specify the modules in the dependencies
section of this file. In addition to the README.md
file, an init.pp
template is created in the manifests
directory.
Our phpmyadmin
package needs to install Apache (httpd
) and configure the httpd
service, so we'll create two new files in the manifests directory, pkg.pp
and svc.pp
.
In init.pp
, we'll include our example_phpmyadmin::pkg
and example_phpmyadmin::svc
classes, as shown in the following code:
class example_phpmyadmin { include example_phpmyadmin::pkg include example_phpmyadmin::svc }
The pkg.pp
file will define example_phpmyadmin::pkg
, as shown in the following code:
class example_phpmyadmin::pkg { package {'httpd': ensure => 'installed', alias => 'apache' } }
The svc.pp
file will define example_phpmyadmin::svc
, as shown in the following code:
class example_phpmyadmin::svc { service {'httpd': ensure => 'running', enable => true } }
Now, we'll define another module called example_phpldapadmin
using the puppet module
command, as shown here:
[samdev@standdist]$ puppet module generate samdev-example_phpldapadmin We need to create a metadata.json file for this module. Please answer the following questions; if the question is not applicable to this module, feel free to leave it blank. … Notice: Generating module at /home/samdev/control/dist/example_phpldapadmin... Notice: Populating templates... Finished; module generated in example_phpldapadmin. example_phpldapadmin/manifests example_phpldapadmin/manifests/init.pp example_phpldapadmin/spec example_phpldapadmin/spec/classes example_phpldapadmin/spec/classes/init_spec.rb example_phpldapadmin/spec/spec_helper.rb example_phpldapadmin/tests example_phpldapadmin/tests/init.pp example_phpldapadmin/Gemfile example_phpldapadmin/Rakefile example_phpldapadmin/README.md example_phpldapadmin/metadata.json
We'll define the init.pp
, pkg.pp
, and svc.pp
files in this new module just as we did in our last module so that our three class files contain the following code:
class example_phpldapadmin { include example_phpldapadmin::pkg include example_phpldapadmin::svc } class example_phpldapadmin::pkg { package {'httpd': ensure => 'installed', alias => 'apache' } } class example_phpldapadmin::svc { service {'httpd': ensure => 'running', enable => true } }
Now we have a problem, phpldapadmin
uses the httpd
package and so does phpmyadmin
, and it's quite likely that these two modules may be included in the same node.
We'll include both of them on our client by editing client.yaml
and then we will run Puppet using the following command:
[root@client ~]# puppet agent -t Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Package[httpd] is already declared in file /etc/puppetlabs/code/environments/production/dist/example_phpmyadmin/manifests/pkg.pp:2; cannot redeclare at /etc/puppetlabs/code/environments/production/dist/example_phpldapadmin/manifests/pkg.pp:2 at /etc/puppetlabs/code/environments/production/dist/example_phpldapadmin/manifests/pkg.pp:2:3 on node client.example.com Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
A resource in Puppet can only be defined once per node. What this means is that if our module defines the httpd
package, no other module can define httpd
. There are several ways to deal with this problem and we will work through two different solutions.
The first solution is the more difficult option—use virtual resources
to define the package and then realize the package in each place you need. Virtual resources are similar to a placeholder for a resource; you define the resource but you don't use it. This means that Puppet master knows about the Puppet definition when you virtualize it, but it doesn't include the resource in the catalog at that point. Resources are included when you realize them later; the idea being that you can virtualize the resources multiple times and not have them interfere with each other. Working through our example, we will use the @
(at) symbol to virtualize our package and service resources. To use this model, it's helpful to create a container for the resources you are going to virtualize. In this case, we'll make modules for example_packages
and example_services
using Puppet module's generate
command again.
The init.pp
file for example_packages
will contain the following:
class example_packages { @package {'httpd': ensure => 'installed', alias => 'apache', } }
The init.pp
file for example_services
will contain the following:
class example_services { @service {'httpd': ensure =>'running', enable => true, require => Package['httpd'], } }
These two classes define the package and service for httpd
as virtual. We then need to include these classes in our example_phpmyadmin
and example_phpldapadmin
classes. The modified example_phpmyadmin::pkg
class will now be, as follows:
class example_phpmyadmin::pkg { include example_packages realize(Package['httpd']) }
And the example_phpmyadmin::svc
class will now be the following:
class example_phpmyadmin::svc { include example_services realize(Service['httpd']) }
We will modify the example_phpldapadmin
class in the same way and then attempt another Puppet run on client (which still has example_phpldapadmin
and example_phpmyadmin
classes), as shown here:
[root@client ~]# puppet agent -t Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for client.example.com Info: Applying configuration version '1443928369' Notice: /Stage[main]/Example_packages/Package[httpd]/ensure: created Notice: /Stage[main]/Example_services/Service[httpd]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Example_services/Service[httpd]: Unscheduling refresh on Service[httpd] Notice: Applied catalog in 11.17 seconds
For this solution to work, you need to migrate the resources that may be used by multiple modules to your top-level resource module and include the resource module wherever you need to realize the resource.
In addition to the realize
function, used previously, a collector exists for virtual resources. A collector
is a kind of glob that can be applied to virtual resources to realize resources based on a tag. A tag in Puppet is just a meta attribute of a resource that can be used for searching later. Tags are only used by collectors (for both virtual and exported resources, the exported resources will be explored in a later chapter) and they do not affect the resource.
To use a collector in the previous example, we will have to define a tag in the virtual resources, for the httpd
package this will be, as follows:
class example_packages { @package {'httpd': ensure => 'installed', alias => 'apache', tag => 'apache', } }
And then to realize the package using the collector, we will use the following code:
class example_phpldapadmin::pkg { include example_packages Package <| tag == 'apache' |> }
The second solution will be to move the resource definitions into their own class and include that class whenever you need to realize the resource. This is considered to be a more appropriate way of solving the problem. Using the virtual resources described previously splits the definition of the package away from its use area.
For the previous example, instead of a class for all package resources, we will create one specifically for Apache and include that wherever we need to use Apache. We'll create the example_apache
module monolithically with a single class for the package and the service, as shown in the following code:
class example_apache { package {'httpd': ensure => 'installed', alias => 'apache' } service {'httpd': ensure => 'running', enable => true, require=> Package['httpd'], } }
Now, in example_phpldapadmin::pkg
and example_phpldapadmin::svc
, we only need to include example_apache
. This is because we can include a class any number of times in a catalog compilation without error. So, both our example_phpldapadmin::pkg
and example_phpldapadmin::svc
classes are going to receive definitions for the package and service of httpd
; however, this doesn't matter, as they only get included once in the catalog, as shown in the following code:
class example_phpldapadmin::pkg { include example_apache }
Both these methods solve the issue of using a resource in multiple packages. The rule is that a resource can only be defined once per catalog, but you should think of that rule as once per organization so that your modules won't interfere with those of another group within your organization.