Lots of things happen on a Linux system while it’s running. Part of your job as a Linux administrator is knowing just what is happening and watching for when things go wrong. The primary tool for accomplishing that task is the logging service. All Linux distributions implement some type of logging service that tracks system events and stores them in log files. This chapter explores the two most popular logging methods used in Linux distributions, rsyslog and systemd-journald. First, the chapter explains Linux logging principles to help give you an idea of what logging is all about. Then the chapter discusses both the rsyslogd
and the systemd-journald
methods of generating logs.
All Linux distributions implement some method of logging. Logging directs short messages that indicate what events happen, and when they happen, to users, files, or even remote hosts for storage. If something goes wrong, the Linux administrator can review the log entries to help determine the cause of the problem.
The following sections discuss the basics of how logging has been implemented in Linux and shows the main logging packages that you’ll most likely run into while working with various Linux distributions.
In the early days of Unix, myriad logging methods were used to track system and application events. Applications used different logging methods, making it difficult for system administrators to troubleshoot issues.
In the mid-1980s, Eric Allman defined a protocol for logging events from his Sendmail mail application, called syslog. The syslog protocol quickly became a standard for logging both system and application events in Unix and made its way to the Linux world.
What made the syslog protocol so popular is that it defines a standard message format that specifies the timestamp, type, severity, and details of an event. That standard can be used by the operating system, applications, and even devices that generate errors.
The type of event is defined as a facility value. The facility defines what is generating the event message, such as a system resource or an application. Table 17.1 lists the facility values defined in the syslog protocol.
Table 17.1 The syslog protocol facility values
Code | Keyword | Description |
0 | kern |
Messages generated by the system kernel |
1 | user |
Messages generated by user events |
2 | mail |
Messages from a mail application |
3 | daemon |
Messages from system applications running in background |
4 | auth |
Security or authentication messages |
5 | syslog |
Messages generated by the logging application itself |
6 | lpr |
Printer messages |
7 | news |
Messages from the news application |
8 | uucp |
Messages from the Unix-to-Unix copy program |
9 | cron |
Messages generated from the cron job scheduler |
10 | authpriv |
Security or authentication messages |
11 | ftp |
File Transfer Protocol application messages |
12 | ntp |
Network Time Protocol application messages |
13 | security |
Log audit messages |
14 | console |
Log alert messages |
15 | solaris-cron |
Another scheduling daemon message type |
16-23 | local0–local7 |
Locally defined messages |
As you can tell from Table 17.1, the syslog protocol covers many different types of events that can appear on a Linux system.
Each event is also marked with a severity. The severity value defines how important the message is to the health of the system. Table 17.2 shows the severity values as defined in the syslog protocol.
Table 17.2 The syslog protocol severity values
Code | Keyword | Description |
0 | emerg |
An event that causes the system to be unusable |
1 | alert |
An event that requires immediate attention |
2 | crit |
An event that is critical but doesn’t require immediate attention |
3 | err |
An error condition that allows the system or application to continue |
4 | warning |
A non-normal warning condition in the system or application |
5 | notice |
A normal but significant condition message |
6 | info |
An informational message from the system |
7 | debug |
Debugging messages for developers |
Combining the facility and severity codes with a short informational message provides enough logging information to troubleshoot almost any problem in Linux.
Over the years there have been many open source logging projects for Linux systems. The following have been the most prominent:
syslogd
program to monitor the system and applications for events and the klogd
program to monitor the Linux kernel for events.The following sections dive into the details of the two most popular logging applications, rsyslog and systemd-journald.
The rsyslog application utilizes all of the features of the original syslog protocol, including the configuration format and logging actions. The following sections walk through how to configure the rsyslog logging application and where to find the common log files it generates.
The rsyslog package uses the rsyslogd
program to monitor events and log them as directed, using the /etc/rsyslog.conf
configuration file to define what events to listen for and how to handle them. Many Linux distributions also use a /etc/rsyslog.d
directory to store individual configuration files that are included as part of the rsyslog.conf
configuration. This allows separate applications to define their own log settings.
The configuration file contains rules that define how the program handles syslog events received from the system, kernel, or applications. The format of an rsyslogd
rule is as follows:
facility.priority action
The facility
entry uses one of the standard syslog protocol facility keywords. The priority
entry uses the severity keyword as defined in the syslog protocol, but with a twist. When you define a severity, syslogd
will log all events with that severity or higher (lower severity code). Thus, the entry
kern.crit
logs all kernel event messages with a severity of critical, alert, or emergency. To log only messages with a specific severity, use an equal sign before the priority keyword:
kern.=crit
You can also use wildcard characters for either the facility or priority. The entry
*.emerg
logs all events with an emergency severity level.
The action
entry defines what rsyslogd
should do with the received syslog message. There are six action options you have available:
Listing 17.1 shows the entries in the configuration file for an Ubuntu 18.04 system.
Listing 17.1: The rsyslogd.conf
configuration entries for Ubuntu 18.04
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
mail.err /var/log/mail.err
*.emerg :omusrmsg:*
The first entry shown in Listing 17.1 defines a rule to handle all auth
and authpriv
facility type messages. This shows that you can specify multiple facility types by separating them with commas. The rule also uses a wildcard character for the priority, so all severity levels will be logged. This rule indicates that all security event messages will be logged to the /var/log/auth.log
file.
The second entry defines a rule to handle all events (*.*) except security events (the .none
priority). The event messages are sent to the /var/log/syslog
file. The minus sign in front of the filename tells rsyslogd
not to sync the file after each write, increasing the performance. The downside to this is if the system crashes before the next normal system sync, you may lose the event message.
The kern.*
entry defines a rule to store all kernel event messages in a separate log file, located in the /var/log/kern.log
file. This has become somewhat of a standard in Linux distributions.
The *.emerg
entry defines a rule to handle all emergency events. The omusrmsg
command indicates to send the event message to a user account on the system. By specifying the wildcard character, this rule sends all emergency event messages to all users on the system.
For comparison, Listing 17.2 shows the entries in the rsyslogd
configuration file for a CentOS 7 system.
Listing 17.2: The rsyslog.conf
configuration file for CentOS 7
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
Notice that Red Hat–based systems use the /var/log/messages
file for informational messages and the /var/log/secure
file for security messages.
As you can guess, for busy Linux systems it doesn’t take long to generate large log files. To help combat that, many Linux distributions install the logrotate
utility. It automatically splits rsyslogd
log files into archive files based on a time or the size of the file. You can usually identify archived log files by the numerical extension added to the log filename.
If you create and run scripts on your Linux system (see Chapter 25), you may want to log your own application events. You can do that with the logger
command-line tool:
logger [-isd] [-f file] [-p priority] [-t tag] [-u socket] [message]
The -i
option specifies the process ID (PID) of the program that created the log entry as part of the event message. The -p
option allows you to specify the event priority. The -t
option allows you to specify a tag to add to the event message to help make finding the message in the log file easier. You can either specify the message as text in the command line or it as a file using the -f
option. The -d
and -u
options are advanced options for sending the event message to the network. The -s
option sends the event message to the standard error output.
An example of using logger
in a script would look like this:
$ logger This is a test message from rich
On an Ubuntu system, you can look at the end of the /var/log/syslog
file to see the log entry:
$ tail /var/log/syslog
...
Feb 8 20:21:02 myhost rich: This is a test message from rich
Notice that rsyslogd
added the timestamp, host, and user account for the message. This is a great troubleshooting tool!
Generally, most Linux distributions create log files in the /var/log
directory. Depending on the security of the Linux system, many log files are readable by everyone, but some may not be.
As seen in Listing 17.1 and Listing 17.2, most Linux distributions create separate log files for different event message types, although they don’t always agree on the log file names.
It’s also common for individual applications to have a separate directory under the /var/log
directory for their own application event messages, such as /var/log/apache2
for the Apache web server.
The easiest way to find the log files for your system is to examine the /etc/rsyslog.conf
configuration file. Just remember to also look for additional configuration files in the /etc/rsyslog.d
directory.
Since rsyslogd
log files are text files, you can use any of the standard text tools available in Linux, such as cat
, head
, tail
, and, of course, vi
to view them. One common trick for administrators is to use the -f
option with the tail
command. That displays the last few lines in the log file but then monitors the file for any new entries and displays those too.
While not part of rsyslogd
, some Linux distributions also include the lastb
tool. It displays the event messages from the /var/log/wtmp
log file, used by many Linux distributions to log user logins.
The Systemd system services package includes the systemd-journald
journal utility for logging. Notice that we called it a journal utility instead of a logging utility. The systemd-journald
program uses its own method of storing event messages, completely different from how the syslog protocol specifies.
The following sections discuss how to use the systemd-journald
program to track event messages on your Linux system.
The systemd-journald
service reads its configuration from the /etc/systemd/journald.conf
configuration file. When you examine this file, you’ll notice that there aren’t any rules defined, only settings that control how the application works:
Storage
setting determines how systemd-journald
stores event messages. When the setting is set to auto
, it will look for the /var/log/journal
directory and store event messages there. If that directory doesn’t exist, it stores the event messages in the temporary /run/log/journal
directory, which is deleted when the system shuts down. You must manually create the /var/log/journal
directory for the event messages to be stored permanently. If you set the Storage
setting to persistent
, systemd-journald
will create the directory automatically. When set to volatile
, it only stores event messages in the temporary directory.Compress
setting determines whether to compress the journal files.ForwardToSyslog
setting determines if systemd-journald
should forward any received messages to a separate syslog program, such as rsyslogd
, running on the system. This provides two layers of logging capabilities.There are quite a few settings that allow you to customize exactly how systemd- journald
works in your system. For a full list and explanation of all the settings, type man journald.conf
at the command prompt.
The systemd-journald
program doesn’t store journal entries in text files. Instead it uses its own binary file format that works like a database. While this makes it a little harder to view journal entries, it does provide for quick searching for specific event entries.
The journalctl program is our interface to the journal files. The basic format for the journalctl
command is as follows:
journalctl [options] [matches]
The options
control data returned by the matches is displayed. Table 17.3 lists the options available.
Table 17.3 The journalctl
command options
Option | Description |
-a |
Displays all data fields |
-e |
Jumps to the end of the journal and uses the pager utility to display the entries |
-l |
Displays all printable data fields |
-n number |
Shows the most recent number journal entries |
-r |
Reverses the order of the journal entries in the output |
The matches
parameter defines what type of journal entries to display. Table 17.4 lists the matches available.
Table 17.4 The journalctl matches
parameter
Match | Description |
Fields |
Matches specific fields in the journal |
Kernel |
Only displays kernel journal entries |
PRIORITY= value |
Matches only entries with the specified priority |
_UID= userid |
Matches only entries made by the specified user ID |
_HOSTNAME= host |
Matches only entries from the specified host |
_TRANSPORT= trans |
Matches only entries received by the specified transport method |
_UDEV_SYSNAME= dev |
Matches only entries received from the specified device |
OBJECT_PID= pid |
Matches only entries made by the specified application process ID |
The journalctl
command is great for when you are looking for specific event entries in the journal; it allows you to filter out events using the matches and determine how to display them using the options, as shown in Listing 17.3.
Listing 17.3: Output from the journalctl
command
[rich@localhost ~]$ journalctl -r _TRANSPORT=kernel
-- Logs begin at Fri 2019-02-08 12:47:04 EST, end at...
Feb 08 12:48:36 localhost.localdomain kernel: TCP: lp registered
Feb 08 12:48:17 localhost.localdomain kernel: fuse init (API version 7.22)
Feb 08 12:47:43 localhost.localdomain kernel: virbr0: port 1(virbr0-nic)
Feb 08 12:47:43 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
Feb 08 12:47:43 localhost.localdomain kernel: virbr0: port 1(virbr0-nic)
Feb 08 12:47:35 localhost.localdomain kernel: e1000: enp0s8 NIC Link is Up
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE)
Feb 08 12:47:32 localhost.localdomain kernel: ip_set: protocol 6
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
...
That makes digging through journal files much easier!
Exercise 17.1 walks through how to monitor the system logs in real time and how to watch as you send a message to the logging facility on your Linux system.
This exercise demonstrates how to create a log or journal entry and view that entry in both rsyslogd
logs and systemd-journald
journals.
sudo tail -f /var/log/syslog
. For Red Hat–based systems such as Fedora and CentOS, use sudo tail -f /var/log/messages
.logger This is a test log entry
.journalctl -r
. You should see your log event message appear in the journal as well. Red Hat–based systems use systemd-journald
, but also forward event messages to the rsyslogd
application, so you can often see events in both systems.Logging events that occur on your Linux system or applications is crucial to being able to troubleshoot problems. The Linux system has adopted the syslog protocol method for handling event messages. The syslog protocol defines an event type, called the facility, and an event severity. This helps you determine which events are important to act on and which ones are just informational.
The rsyslogd
logging application utilizes the syslog protocol to log system and application events. The /etc/rsyslogd.conf
configuration file controls what events are logged and how they are logged. The rsyslogd
application can log events to files, applications, remote hosts, terminals, and even directly to users.
The Linux Systemd services package also includes its own method for logging events called journaling. The systemd-journald
application monitors the system and applications for all events and sends them to a special journal file. The operation of systemd-journald
is controlled by the /etc/systemd/journald
configuration file. You must use the journalctl
application to read events stored in the journal file.
Describe the logging protocol used by most Linux logging applications. The syslog protocol has become the de facto standard for most Linux logging applications. It identifies events using a facility code, which defines the event type, and a severity, which defines how important the event message is. The sysklogd
, syslogd-ng
, and rsyslogd
applications all use the syslog protocol for managing system and application events in Linux.
Describe how the rsyslogd
application directs events to specific locations. The rsyslogd
application uses the /etc/rsyslogd.conf
configuration file to define rules for handling events. Each rule specifies a syslog facility and severity along with an action to take. Events that match the facility and have a priority equal to or higher than the severity defined are sent to the defined action. The action can be sending the event message to a log file, piping the message to an application, or sending the event message to a remote host or to a user on the system.
Explain how the Systemd service uses a different method for logging events. The Systemd service package uses the systemd-journald
application, which doesn’t use the syslog protocol for logging events. Instead, systemd-journald
creates its own binary journal files for storing event messages. The binary journal file is indexed to provide faster searching for events. The journalctl
application provides the interface for sending search queries to the journal files and displaying the search results.
What protocol became a de facto standard in Linux for tracking system event messages?
journalctl
Nancy wants to write a rsyslogd
rule that separates event messages coming from the system job scheduler to a separate log file. Which syslog facility keyword should she use?
cron
user
kern
console
local0
What syslog severity level has the highest priority ranking in rsyslogd
?
crit
alert
emerg
notice
err
What syslog severity level represents normal but significant condition messages?
crit
notice
info
alert
local0
What syslog application is known for its rocket-fast speed?
syslogd
syslog-ng
systemd-journald
klogd
rsyslogd
What configuration file does the rsyslogd
application use by default?
rsyslog.conf
journald.conf
syslogd.conf
rsyslog.d
syslog.d
James needs to log all kernel messages that have a severity level of warning or higher to a separate log file. What facility and priority setting should he use?
kern.=warn
kern.*
*.info
kern.warn
kern.alert
Barbara wants to ensure that the journal log files will be saved after the next reboot of her Linux system. What systemd-journald
configuration setting should she use?
Storage=auto
Storage=persistent
ForwardToSyslog=on
Storage=volatile
ForardToSyslog=off
Katie wants to display the most recent entries in the journal log on her Linux system. What journalctl
option should she use?
-a
-l
-r
-e
-n
Tony is trying to troubleshoot errors produced by an application on his Linux system but has to dig through lots of entries in the journal log file to find them. What journalctl
match option would help him by only displaying journal entries related to the specific application?
OBJECT_PID
Kernel
_TRANSPORT
_UID
_UDEV