Turning on the debugging option on your Puppet master isn't such a big deal with a few hundred nodes. However, in an environment with thousands of nodes, it isn't a viable option. Nevertheless, you sometimes need to enable debugging to figure out where catalog compilation is failing. Our proxy configuration comes to the rescue here. The idea is to have one Puppet master dedicated to debugging. The debugging server will have debugging turned on, by changing the puppetserver logging settings in the logback.xml file. The advantage of this method over that of running puppet master –compile, as we showed earlier, is that, while you are debugging your node, you place it in a debugging environment (problem for instance). While the node is in the debugging environment, it will be removed from your reporting infrastructure and not continue to alert you to failures.
To do this, we go back to our proxy.conf file on our Puppet master and define a new balancer named puppetproblem that goes to our debugging worker. We'll use worker2 (192.168.100.102) in the following example:
<Proxy balancer://puppetproblem> BalancerMember http://192.168.100.102:18140 </Proxy>
We now add a new ProxyPassMatch line to our VirtualHost right after the certificate matching line:
ProxyPassMatch^/(problem/.*)$ balancer://puppetproblem/$1
Restart httpd on the master to make the change effective. With this in place, we edit logback.xml on our debugging Puppet master and change the LOGLEVEL to DEBUG.
Restart puppetserver on the debugging Puppet master to make the change effective. Now, when you have a problem with a node, you can send it to worker2 by specifying the environment "problem" when running the agent. The steps to diagnose a problem are, as follows:
- Create the problem branch in Git.
- Work on the issue.
- Set the environment of a test node to the new environment.
- Solve the problem.
- Merge that branch back into the working branch or production.
Using this method, you can also tie the catalog compilation to a specific worker, which makes tracking down bugs much easier. Without this, your catalog might compile on any one of your workers and some large installations have several workers.
When working through a catalog compilation issue, it is sometimes useful to start attacking the problem and changing things on-the-fly. To avoid problems with other nodes, you should work in a new branch (which will create a new environment, just as we configured our Puppet masters to have dynamic environments in Chapter 3, Git and Environments). If you are frequently creating branches, you can create one named after yourself or your username, for instance. In an example in Chapter 3, Git and Environments, we created a thomas branch and worked in the thomas branch by specifying --environment thomas when running puppet agent. Working through problems in a personal branch is a great troubleshooting technique that allows the rest of the nodes to continue working against the main branch or master. If multiple members of your team are working on an issue, it is useful to create a working branch for your team, possibly named either after the issue or more likely after the trouble ticket created by the issue.
When working on a problem branch, you are free to add any number of debugging print or echo statements to your code. In Puppet, these take the form of notice or notify lines. I prefer notify lines, since notify lines will be printed when I run puppet agent -t on a node. Usually, I place all the variables of the affected module in a single notify statement to make sure that the variables are getting set to the values I believe they should. This method is very useful when working with data from Hiera, where you would like to know if the value returned by Hiera is correct, as shown in the following example:
$importantSetting = hiera('importantSetting','defaultValue')
notify {"importantSetting is $importantSetting": }It is not uncommon to have many notify lines throughout a module during the development phase.
Occasionally, you will have naming conflicts with variables or modules when working on a large code base. For variables, using a notify statement can quickly determine if your code is using the variable you believe it should. For modules, it can sometimes be difficult to determine if the module you intended is being included. For example, you have two modules called packages and example::ntp::packages. The packages module contains a single notify statement in packages/manifests/init.pp, as shown in the following code:
class packages {
notify {"this is packages":}
}The example::ntp::packages module has a similar notify statement in example/manifests/ntp/packages.pp, as shown in the following code:
class example::ntp::packages {
notify {"this is example::ntp::packages": }
}Now, in example/manifest/ntp.pp, we use include packages, as shown in the following code:
class example::ntp {
include packages
}You may be surprised by the following result from puppet agent:
# puppet agent -t ... Notice: this is example::ntp::packages Notice: /Stage[main]/Example::Ntp::Packages/Notify[this is example::ntp::packages]/message: defined 'message' as 'this is example::ntp::packages'
We might have expected include packages to use the top-scope packages class, but it actually searched the local scope and used example::ntp::packages instead. When working in a large environment, it is advisable to use very specific names for classes or always specify the scope. We can achieve the result we expected using the following code for the definition of example::ntp:
class example::ntp {
include ::packages
}If we run puppet agent against this version, we see the notification we were expecting, as follows:
# puppet agent -t ... Notice: this is packages Notice: /Stage[main]/Packages/Notify[this is packages]/message: defined 'message' as 'this is packages'
If your Puppet runs are taking a long time to complete, it is useful to see where there are bottlenecks. From the command line, you can pass the --evaltrace --summarize option to puppet agent to tell the agent to keep a track of how long the operations took to complete and display a summary at the end of compilation, as shown in the following screenshot:
puppetserver also has the ability to send profiling information to a graphite server. Information on configuring puppetserver to communicate with a graphite server is available at http://docs.puppetlabs.com/pe/latest/puppet_server_metrics.html.