Server log files contain informational messages about the kernel, applications, and services running on a system. These files can be very useful for troubleshooting and debugging system-level issues. DBAs often look in the system log files as a first step in diagnosing server issues. Even if you’re working with competent SAs, you can still save time and gain valuable insights into the root cause of a problem by inspecting these log files.

This appendix covers managing Linux and Solaris log files. You’ll learn about the basic information contained in the log files and the tools available to rotate the logs.

Managing Linux Log Files

Most of the system log files are located in the /var/log directory. There is usually a log file for a specific application or service. For example, the cron utility has a log file named cron (no surprise) in the /var/log directory. Depending on your system, you may need root privileges to view certain log files.

The log files will vary somewhat by the version of the OS and the applications running on your system. Table B-1 contains the names of some of the more common log files and their descriptions.

Rotating Log Files

The system log files will continue to grow unless they are somehow moved or removed. Moving and removing log files is known as rotating the log files, which means that the current log file is renamed, and a new log file is created.

Most Linux systems use the logrotate utility to rotate the log files. This tool automates the rotation, compression, removal, and mailing of log files. Typically, you’ll rotate your log files so that they don’t become too large and cluttered with old data. You should delete log files that are older than a certain number of days.

By default, the logrotate utility is automatically run from the cron scheduling tool on most Linux systems. Here’s a typical listing of the contents of the /etc/crontab file:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

Notice that the /etc/crontab uses the run-parts utility to run all scripts located within a specified directory. For example, when run-parts inspects the /etc/cron.daily directory, it finds a file named logrotate that calls the logrotate utility. Listed here are the contents of a typical logrotate script:

#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

The behavior of the logrotate utility is governed by the /etc/logrotate.conf file. Here’s a listing of a typical /etc/logrotate.conf file:

# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp -- we’ll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
    rotate 1
}
# system-specific logs may be also be configured here.

By default, the logs are rotated weekly on most Linux systems, and four weeks’ worth of logs are preserved. These are designated by the lines weekly and rotate 4 in the /etc/ logrotate.conf file. You can change the values within the /etc/logrotate.conf file to suit the rotating requirements of your environment.

If you list the files in the /var/log directory, notice that some log files end with an extension of .1 or .gz. This indicates that the logrotate utility is running on your system.

You can manually run the logrotate utility to rotate the log files. Use the -f option to force a rotation, even if logrotate doesn’t think it is necessary:

# logrotate -f /etc/logrotate.conf

Application–specific logrotate configurations are stored in the /etc/logrotate.d directory. Change directories to the /etc/logrotate.d directory and list some typical application logs on a Linux server:

# cd /etc/logrotate.d
# ls
acpid  cups  mgetty  ppp  psacct  rpm  samba  syslog  up2date  yum

Setting Up a Custom Log Rotation

The logrotate utility is sometimes perceived as a utility only for SAs. However, any user on the system can use logrotate to rotate log files for applications for which they have read/write permissions on the log files. For example, as the oracle user, you can use logrotate to rotate your database alert.log file.

Here are the steps for setting up a job to rotate the alert log file of an Oracle database:

1. Create a configuration file named alert.conf in the directory $HOME/config (create the config directory if it doesn’t already exist):

   /oracle/RMDB1/admin/bdump/*.log {
   daily
   missingok
   rotate 7
   compress
   mail oracle@localhost
   }

2. In the preceding configuration file, the first line specifies the location of the log file. The asterisk (wildcard) tells logrotate to look for any file with the extension of .log in that directory. The daily keyword specifies that the log file should be rotated on a daily basis. The missingok keyword specifies that logrotate should not throw an error if it doesn’t find any log files. The rotate 7 keyword specifies that the log files should be kept for seven days. The compress keyword compresses the rotated log file. Finally, a status e-mail is sent to the local oracle user on the server.
3. Create a cron job to automatically run the job on a daily basis:

   0 9 * * * /usr/sbin/logrotate -f -s /home/oracle/config/alrotate.status
   /home/oracle/config/alert.conf

    Note  The previous two lines of code should be one line in your cron table (the code didn’t fit nicely on this page on one line).

4. The cron job runs the logrotate utility every day at 9 a.m. The -s (status) option directs the status file to the specified directory and file. The configuration file used is /home/oracle/config/alert.conf.
5. Manually test the job to see whether it rotates the alert log correctly. Use the -f switch to force logrotate to do a rotation:

   $ /usr/sbin/logrotate -f -s /home/oracle/config/alrotate.status 
   /home/oracle/config/alert.conf

As shown in the previous steps, you can use the logrotate utility to set up log rotation jobs.

Compare using logrotate instead of writing a custom shell script such as the one described in recipe 10-8.

Monitoring Log Files

Many Linux systems have graphical interfaces for monitoring and managing the log files. As a DBA, you often need to look only at a specific log file when trying to troubleshoot a problem. In these scenarios, it is usually sufficient to manually inspect the log files with a text editor such as vi or a paging utility such as more or less.

You can also monitor the logs with the logwatch utility. You can modify the default behavior of logwatch by modifying the logwatch.conf file. Depending on your Linux system, the logwatch.conf file is usually located in a directory named /etc/log.d. To print the default log message details, use the --print option:

# logwatch --print

Many SAs set up a daily job to be run that automatically e-mails the logwatch report to a specified user. Usually this functionality is implemented as a script located in the /etc/cron.daily directory. The name of the script will vary by Linux system. Typically, these scripts are named something like 0logwatch or 00-logwatch.

Managing Solaris Log Files

The Solaris OS logs can be found under the /var directory. Table B-2 documents the names and purpose of commonly used log files in a Solaris environment.

Viewing System Message Log Files

The syslogd daemon automatically records various system errors, warnings, and faults in message log files. You can use the dmesg command to view the most recently generated system-level messages. For example, run the following as the root user:

# dmesg

Here’s some sample output:

Apr  1 12:27:56 sb-gate su: [ID 810491 auth.crit] ’su root’ failed for mt...
Apr  2 11:14:09 sb-gate sshd[15969]: [ID 800047 auth.crit] monitor fatal: protocol error...

The /var/adm directory contains several log directories and files. The most recent system log entries are in the /var/adm/messages file. Periodically (typically every 10 days), the contents of the messages file are rotated and renamed to messages.N. For example, you should see a messages.0, messages.1, messages.2, and messages.3 file (older files are deleted). Use the following command to view the current messages file:

# more /var/adm/messages

If you want to view all logged messages, enter the following command:

# more /var/adm/messages*

Rotating Solaris Log Files

You can rotate logs in a Solaris environment via the logadm utility, which is a very flexible and powerful tool that you can use to manage your log files. The logadm utility is called from the root user’s cron table. Here’s an example:

10 3 * * * /usr/sbin/logadm

This code shows that the logadm utility is called once per day at 3:10 a.m. The logadm utility will rotate files based on information in the /etc/logadm.conf file. Although you can manually modify this file, the recommended approach to modifying the /etc/logadm.conf file is via the logadm utility.

A short example will help illustrate how to add an entry. This next line of code instructs the logadm utility to add an entry with the -w switch:

# logadm -w /orahome/logs/mylog.log -C 8 -c -p 1d -t ’/orahome/logs/mylog.log.$n’ -z 1

Now if you inspect the contents of the /etc/logadm.conf file, the prior line has been added to the file:

/orahome/logs/mylog.log -C 8 -c -p 1d -t ’/orahome/logs/mylog.log.$n’ -z 1

The preceding line of code instructs logadm to rotate the /orahome/logs/mylog.log file. The -C 8 switch specifies that it should keep eight old versions before deleting the oldest file. The -c switch instructs the file to be copied and truncated (and not moved). The -p 1d switch specifies that the log file should be rotated on a daily basis. The -t switch provides a template for the rotated log file name. The -z 1 switch specifies that the number 1 rotated log should be compressed.

You can validate your entry by running logadm with the -V switch. Here’s an example:

# logadm -V

You can also force an immediate execution of the entry via the -p now switch:

# logadm -p now /orahome/logs/mylog.log

After running the preceding command, you should see that your log has been rotated:

# cd /orahome/logs
# ls -altr
-rw-r--r--   1 root     root           0 Apr  5 16:40 mylog.log.0
-rw-r--r--   1 root     root           0 Apr  5 16:40 mylog.log

To remove an entry from the /etc/logadm.conf file, use the -r switch. Here’s an example:

# logadm -r /orahome/logs/mylog.log

Summary

Server log files are often the first places to look when you experience performance and security issues. These files contain messages that help diagnose and troubleshoot problems. Because log files tend to grow very fast, it is important to understand how to rotate the logs, which ensures that they are archived, compressed, and deleted at regular intervals.

On Linux systems, use the logrotate utility to rotate log files; on Solaris servers, use the logadm utility.