11
THE LOGGING SYSTEM

For any Linux user, it’s crucial to be knowledgeable in the use of the log files. Log files store information about events that occur when the operating system and applications are run, including any errors and security alerts. Your system will log information automatically based on the series of rules that I will show you how to configure in this chapter.

As a hacker, the log files can be a trail to your target’s activities and identity. But it can also be a trail to your own activities on someone else’s system. A hacker therefore needs to know what information they can gather, as well as what can be gathered about their own actions and methods in order to hide that evidence.

On the other side, anyone securing Linux systems needs to know how to manage the logging functions to determine whether a system has been attacked and then decipher what actually happened and who did it.

This chapter shows you how to examine and configure log files, as well as how to remove evidence of your activity and even disable logging altogether. First, we’ll look at the daemon that does the logging.

The rsyslog Logging Daemon

Linux uses a daemon called syslogd to automatically log events on your computer. Several variations of syslog, including rsyslog and syslog-ng, are used on different distributions of Linux, and even though they operate very similarly, some minor differences exist. Since Kali Linux is built on Debian, and Debian comes with rsyslog by default, we focus on that utility in this chapter. If you want to use other distributions, it’s worth doing a little research on their logging systems.

Let’s take a look at rsyslog on your system. We’ll search for all files related to rsyslog. First, open a terminal in Kali and enter the following:

kali >locate rsyslog
/etc/rsyslog.conf
/etc/rsyslog.d
/etc/default/rsyslog
/etc/init.d/rsyslog
/etc/logcheck/ignore.d.server/rsyslog
/etc/logrotate.d/rsyslog
/etc/rc0.d/K04rsyslog
--snip--

As you can see, numerous files contain the keyword rsyslog—some of which are more useful than others. The one we want to examine is the configuration file rsyslog.conf.

The rsyslog Configuration File

Like nearly every application in Linux, rsyslog is managed and configured by a plaintext configuration file located, as is generally the case in Linux, in the /etc directory. In the case of rsyslog, the configuration file is located at /etc/rsyslog.conf. Open that file with any text editor, and we’ll explore what’s inside (here, I use Leafpad):

kali >leafpad /etc/rsyslog.conf

You should see something like Listing 11-1.

#/etc/rsyslog.conf Configuration file for rsyslog.

# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html

#################
#### MODULES ####
#################


module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

###########################
#### GLOBAL DIRECTIVES ####
###########################
--snip--

Listing 11-1: A snapshot of the rsyslog.conf file

As you can see, the rsyslog.conf file comes well documented with numerous comments explaining its use. Much of this information will not be useful to you at this moment, but if you navigate down to below line 55, you’ll find the Rules section. This is where you can set the rules for what your Linux system will automatically log for you.

The rsyslog Logging Rules

The rsyslog rules determine what kind of information is logged, what programs have their messages logged, and where that log is stored. As a hacker, this allows you to find out what is being logged and where those logs are written so you can delete or obscure them. Scroll to about line 55 and you should see something like Listing 11-2.

###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.*             /var/log/auth.log
*.*;auth,authpriv.none      -/var/log/syslog
#cron.*                     /var/log/cron.log
daemon.*                    -/var/log/daemon.log
kern.*                      -/var/log/kern.log
1pr.*                       -/var/log/lpr.log
mail.*                      -/var/log/mail.log
user.*                      -/var/log/user.log

#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                   -/var/log/mail.info
mail.warn                   -/var/log/mail.warn
mail.err                    /var/log/mail.err

Listing 11-2: Finding the logging rules in rsyslog.conf

Each line is a separate logging rule that says what messages are logged and where they’re logged to. The basic format for these rules is as follows:

facility.priority            action

The facility keyword references the program, such as mail, kernel, or lpr, whose messages are being logged. The priority keyword determines what kind of messages to log for that program. The action keyword, on the far right, references the location where the log will be sent. Let’s look at each section more closely, beginning with the facility keyword, which refers to whatever software is generating the log, whether that’s the kernel, the mail system, or the user.

The following is a list of valid codes that can be used in place of the facility keyword in our configuration file rules:

auth, authpriv   Security/authorization messages

cron   Clock daemons

daemon   Other daemons

kern   Kernel messages

lpr   Printing system

mail   Mail system

user   Generic user-level messages

An asterisk wildcard (*) in place of a word refers to all facilities. You can select more than one facility by listing them separated by a comma.

The priority tells the system what kinds of messages to log. Codes are listed from lowest priority, starting at debug, to highest priority, ending at panic. If the priority is *, messages of all priorities are logged. When you specify a priority, messages of that priority and higher are logged. For instance, if you specify a priority code of alert, the system will log messages classified as alert and higher priority, but it won’t log messages marked as crit or any priority lower than alert.

Here’s the full list of valid codes for priority:

•   debug

•   info

•   notice

•   warning

•   warn

•   error

•   err

•   crit

•   alert

•   emerg

•   panic

The codes warn, error, and panic have all been deprecated and should not be used.

The action is usually a filename and location where the logs should be sent. Note that generally, log files are sent to the /var/log directory with a filename that describes the facility that generated them, such as auth. This means, for example, that logs generated by the auth facility would be sent to /var/log.auth.log.

Let’s look at some examples of log rules:

mail.* /var/log/mail

This example will log mail events of all (*) priorities to /var/log/mail.

kern.crit /var/log/kernel

This example will log kernel events of critical (crit) priority or higher to /var/log/kernel.

*.emerg :omusmsg:*

This last example will log all events of the emergency (emerg) priority to all logged-on users. With these rules, the hacker can determine where the log files are located, change the priorities, or even disable specific logging rules.

Automatically Cleaning Up Logs with logrotate

Log files take up space, so if you don’t delete them periodically, they will eventually fill your entire hard drive. On the other hand, if you delete your log files too frequently, you won’t have logs to investigate at some future point in time. You can use logrotate to determine the balance between these opposing requirements by rotating your logs.

Log rotation is the process of regularly archiving log files by moving them to some other location, leaving you with a fresh log file. That archived location will then get cleaned up after a specified period of time.

Your system is already rotating log files using a cron job that employs the logrotate utility. You can configure the logrotate utility to choose the regularity of your log rotation with the /etc/logrotate.conf text file. Let’s open it with a text editor and take a look:

kali >leafpad /etc/logrotate.conf

You should see something like Listing 11-3.

  # see "man logrotate" for details
  # rotate log files weekly
➊ weekly

  # keep 4 weeks worth of backlogs
➋ rotate 4

➌ # create new (empty) log files after rotating old ones
  create

➍ # uncomment this if you want your log files compressed
  #compress

  # packages drop log rotation information into this directory
  include /etc/logrotate.d

  # system-specific logs may also be configured here

  --snip--

Listing 11-3: The logrotate configuration file

First, you can set the unit of time your rotate numbers refer to ➊. The default here is weekly, meaning any number after the rotate keyword always refers to weeks.

Further down, you can see the setting for how often to rotate logs—the default setting is to rotate logs every four weeks ➋. This default configuration will work for most people, but if you want to keep your logs longer for investigative purposes or shorter to clear them out quicker, this is the setting you should change. For instance, if you check your log files every week and want to save storage space, you could change this setting to rotate 1. If you have plenty of storage for your logs and want to keep a semi-permanent record for forensic analysis later, you could change this setting to rotate 26 to keep your logs for six months or rotate 52 to keep them for one year.

By default, a new empty log file is created when old ones are rotated out ➌. As the comments in the configuration file advise, you can also choose to compress your rotated log files ➍.

At the end of each rotation period, the log files are renamed and pushed toward the end of the chain of logs as a new log file is created, replacing the current log file. For instance, /var/log.auth will become /var/log.auth.1, then /var/log.auth.2, and so on. If you rotate logs every four weeks and keep four set of backups, you will have /var/log.auth.4, but no /var/log.auth.5, meaning that /var/log.auth.4 will be deleted rather than being pushed to /var/log/auth.5. You can see this by using the locate command to find /var/log/auth.log log files with a wildcard, as shown here:

kali >ls /var/log/auth.log*
/var/log/auth.log.1
/var/log/auth.log.2
/var/log/auth.log.3
/var/log/auth.log.4

For more details on the many ways to customize and use the logrotate utility, see the man logrotate page. This is an excellent resource to learn about the functions you can use and the variables you can change to customize how your logs are handled. Once you become more familiar with Linux, you’ll get a better sense of how often you need to log and what options you prefer, so it’s worth revisiting the logrotate.conf file.

Remaining Stealthy

Once you’ve compromised a Linux system, it’s useful to disable logging and remove any evidence of your intrusion in the log files to reduce the chances of detection. There are many ways to do this, and each carries its own risks and level of reliability.

Removing Evidence

First, you’ll want to remove any logs of your activity. You could simply open the log files and precisely remove any logs detailing your activity, line by line, using the file deletion techniques you learned in Chapter 2. However, this could be time-consuming and leave time gaps in the log files, which would look suspicious. Also, deleted files can generally be recovered by a skilled forensic investigator.

A better and more secure solution is to shred the log files. With other file deletion systems, a skilled investigator is still able to recover the deleted files (deleted files are simply made available to be overwritten by the filesystem; they still exist until they are overwritten), but suppose there was a way to delete the file and overwrite it several times, making it much harder to recover. Lucky for us, Linux has a built-in command, appropriately named shred, for just this purpose.

To understand how the shred command works, take a quick look at the help screen by entering the following command:

kali >shred --help
Usage: shred [OPTION]...FILE...
Overwrite the specified FILE(s) repeatedly in order to make it harder
for even very expensive hardware probing to recover data
--snip--

As you can see from the full output on your screen, the shred command has many options. In its most basic form, the syntax is simple:

shred <FILE>

On its own, shred will delete the file and overwrite it several times—by default, shred overwrites four times. Generally, the more times the file is overwritten, the harder it is to recover, but keep in mind that each overwrite takes time, so for very large files, shredding may become time-consuming.

Two useful options to include are the -f option, which changes the permissions on the files to allow overwriting if a permission change is necessary, and the –n option, which lets you choose how many times to overwrite the files. As an example, we’ll shred the log files in /var/log/auth.log 10 times using the following command:

kali >shred -f -n 10 /var/log/auth.log.*

We need the –f option to give us permission to shred auth files, and we follow the –n option with the desired number of times to overwrite. After the path of the file we want to shred, we include the wildcard asterisk so we’re shredding not just the auth.log file, but also any logs that have been created with logrotate, such as auth.log.1, auth.log.2, and so on.

Now try to open a log file:

kali >leafpad /var/log/auth.log.1

Once you’ve shredded a file, you’ll see that the contents are indecipherable gibberish, as shown in Figure 11-1.

Figure 11-1: A shredded log file

Now if the security engineer or forensic investigator examines the log files, they will find nothing of use because none of it is recoverable!

Disabling Logging

Another option for covering your tracks is to simply disable logging. When a hacker takes control of a system, they could immediately disable logging to prevent the system from keeping track of their activities. This, of course, requires root privileges.

To disable all logging, the hacker could simply stop the rsyslog daemon. Stopping any service in Linux uses the same syntax, shown here (you’ll see more on this in Chapter 12):

service servicename start|stop|restart

So, to stop the logging daemon, you could simply enter the following command:

kali >service rsyslog stop

Now Linux will stop generating any log files until the service is restarted, enabling you to operate without leaving behind any evidence in the log files!

Summary

Log files track nearly everything that happens on your Linux system. They can be an invaluable resource in trying to analyze what has occurred, whether it be a malfunction or a hack. For the hacker, log files can be evidence of their activities and identity. However, an astute hacker can remove and shred these files and disable logging entirely, thus leaving no evidence behind.