Server log files contain informational messages about the kernel, applications, and services running on the system. These files can be very useful for troubleshooting and debugging system-level issues. DBAs often look in the system log files as a first step in diagnosing server issues. Even if you're working with competent system administrators, you can still save yourself time (and gain valuable insights to the root cause of a problem) by inspecting the log files yourself.
Most of the system log files are located in the /var/log directory. Typically there is a log file for a specific application or service. For example, the cron utility has a log file named cron (no surprise) in the /var/log directory. Depending on your system, you may need root privileges to view certain log files.
The log files will vary somewhat by the version of Linux and the applications running on your system. Table B-1 contains the names of some of the more common log files and their descriptions.
Table B-1. Typical Linux Log Files and Descriptions
Log File Name | Purpose |
|---|---|
| System boot messages |
|
|
| Mail server log file |
| General system messages |
| Authentication log file |
| Login records |
|
|
Note
Some utilities may have their own subdirectory under the /var/log directory.
The system log files will continue to grow unless they are somehow moved or removed. Moving and removing log files is known as rotating the log files. Rotating means that the current log file is renamed and a new log file is created.
Most Linux systems use the logrotate utility to rotate the log files. This tool automates the rotation, compression, removal, and mailing of log files. Typically, you will want to rotate your log files so that they don't become too large and cluttered with old data. You'll also want to delete log files that are older than a certain number of days.
By default, on most Linux systems, the logrotate utility is automatically run from the cron scheduling tool. Here is a typical listing of the contents of the /etc/crontab file:
SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root HOME=/ # run-parts 01 * * * * root run-parts /etc/cron.hourly 02 4 * * * root run-parts /etc/cron.daily 22 4 * * 0 root run-parts /etc/cron.weekly 42 4 1 * * root run-parts /etc/cron.monthly
Notice that the /etc/crontab uses the run-parts utility to run all scripts located within a specified directory. For example, when run-parts inspects /etc/cron.daily, it finds a file named logrotate that calls the logrotate utility. Listed next are the contents of a typical logrotate script:
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0The behavior of the logrotate utility is governed by the /etc/logrotate.conf file. Here's a listing of a typical /etc/logrotate.conf file:
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}
# system-specific logs may be also be configured here.By default, on most Linux systems, the logs are rotated weekly, and four weeks' worth of logs are preserved. These are designated by the lines weekly and rotate 4 in the /etc/logrotate.conf file. You can change the values within the /etc/logrotate.conf file to suit the rotating requirements of your environment.
If you list the files in the /var/log directory, you may notice some log files ending with an extension of .1 or .gz. This indicates that the logrotate utility is running on your system.
You can manually run the logrotate utility to rotate the log files. Use the -f option to force a rotation even if logrotate doesn't think it's necessary:
# logrotate -f /etc/logrotate.conf
Application-specific logrotate configurations are stored in the /etc/logrotate.d directory. Here we change directories to the /etc/logrotate.d directory and list some typical application logs on a Linux server:
# cd /etc/logrotate.d # ls acpid cups mgetty ppp psacct rpm samba syslog up2date yum
The logrotate utility is sometimes perceived as a utility only for system administrators. However, any user on the system can utilize logrotate to rotate log files for applications that they have read/write permissions on the log files. For example, as the oracle user, you can use logrotate to rotate your database alert.log file. Here are the steps for setting up a job to rotate the alert log file of an Oracle database:
Create a configuration file named
alert.confin the directory/home/oracle/config(create the directory if it doesn't already exist):/oracle/RMDB1/admin/bdump/*.log { daily missingok rotate 7 compress mail oracle@localhost }In the previous configuration file, the first line specifies the location of the log file. The asterisk (wildcard) tells
logrotateto look for any file with the extension of.login that directory. Thedailykeyword specifies that the log file should be rotated on a daily basis. Themissingokkeyword specifies thatlogrotateshould not throw an error if it doesn't find any log files. Therotate 7keyword specifies that the log files should be kept for seven days. Thecompresskeyword compresses the rotated log file. Lastly, a status e-mail is sent to the localoracleuser on the server.Create a
cronjob to automatically run the job on a daily basis:0 9 * * * /usr/sbin/logrotate -s /home/oracle/config/alrotate.status /home/oracle/config/alert.conf
The previous two lines of code should be one line in your
crontable (it didn't fit nicely on this page on one line). Thecronjob runs thelogrotateutility every day at 9 a.m. The-s(status) option directs the status file to the specified directory and file. The configuration file used is/home/oracle/config/alert.conf.Manually test the job to see whether it rotates the alert log correctly. Use the
-fswitch to forcelogrotateto do a rotation:$ /usr/sbin/logrotate -f -s /home/oracle/config/alrotate.status /home/oracle/config/alert.conf
As shown in the previous steps, you can use the logrotate utility to set up log rotation jobs. Similar examples of using the logrotate utility are shown in recipe 11-9. Juxtapose using logrotate instead of writing a custom shell script such as the one described in recipe 11-8.
Many Linux systems have graphical interfaces for monitoring and managing the log files. As a DBA, oftentimes you need to look only at a specific log file when trying to troubleshoot a problem. In these scenarios, it's usually sufficient to manually inspect the log files with a text editor such as vi or a paging utility such as more or less.
You also can monitor the logs with the logwatch utility. You can modify the default behavior of logwatch by modifying the logwatch.conf file. Depending on your Linux system, the logwatch.conf file is usually located in a directory named /etc/log.d. To print the default log message details, use the --print option:
# logwatch --print
Many system administrators set up a daily job to be run that automatically e-mails the logwatch report to a specified user. Usually this functionality is implemented as a script located in the /etc/cron.daily directory. The name of the script will vary by Linux system. Typically these scripts are named something like 0logwatch or 00-logwatch.