CHAPTER 10 Shell Scripting

Overview

In previous chapters, we looked at some of the common tasks associated with system administration. Beginning with the introduction to the role of the system administrator in Chapter 2, subsequent chapters have introduced the technical controls used to combat security threats and the efforts required after a security breach has occurred.

In this chapter, we will introduce you to a way to handle the sometimes complex and often repetitive, tasks required for effective system administration. The BASH shell provides a mechanism for creating a script – an application constructed from multiple command line applications – to accomplish complex tasks. At the end of this chapter, you should know:

• how to write a simple BASH shell script
• the use of common programming elements (variables, loops, etc)
• how to handle user interaction
• how to use common UNIX tools to parse and manipulate text files

Introduction

Basic knowledge of shell scripting is necessary for anyone wanting to complete common system administration tasks, audit the security of a system, or implement many of the controls we have discussed in the previous chapters. Shell scripts are used for automating processes throughout a Unix system, from starting network services at boot up to configuring the user's shell environment during login. This chapter will just be an introduction to shell scripting. To begin, we will be creating scripts that are intended as examples of common structures and procedures used in all shell scripts. Later in the chapter, we will combine some of these common elements to demonstrate the automation of processes that would be too time-consuming to do manually or need to be repeated in the future.

What exactly is a script and how does it differ from a program written in other programming languages you may be familiar with such as C# or Java? The most important difference between a script and a program written in a language such as Java is that scripts don't have to be compiled into a binary file to be run. The script is interpreted and converted into the necessary binary code at run-time. Since the compilation process is eliminated, developing applications with a scripting language is generally faster than with compiled languages; however, there can be a performance penalty when the code is executed. There are several popular scripting languages in use, including PHP, Python, and Ruby. Unlike scripts written in these languages, shell scripts do not require an interpreter program to convert the script into binary. Shell scripts are interpreted directly by the shell process; BASH in our case, but any of the other popular shells could be used.

So, how do we write a shell script? At its most basic, a shell script is a list of commands saved in a text file that we can run by calling the BASH program on the command line:

Listing 1: /opt/book/scripting/backup_v1

[alice@sunshine ~]$ cat /opt/book/scripting/backup_v1
mkdir -p /tmp/backups
cp -pr /home/alice/work /tmp/backups
cd /tmp/backups/
zip -qr backup.zip work/
rm -rf /tmp/backups/work

echo “Done Backing up the work directory”
[alice@sunshine ~]$ bash /opt/book/scripting/backup_v1
Done Backing up the work directory
[alice@sunshine ~]$ ls /tmp/backups
backups.zip

This saves you the effort of retyping a list of commands each time you need to complete a task. However, by adding a single line to the top of the script and changing the file permissions to make it executable, we can convert this list of commands into a command of its own:

Listing 2: /opt/book/scripting/backup_v2

#! /bin/bash
# This is a comment.
# Lines starting with the pound sign (#) are ignored in BASH scripts
#
# This script copies and compresses files in /home/alice/work and
# saves them to /tmp/backups/work
mkdir -p /tmp/backups
cp -pr /home/alice/work /tmp/backups
cd /tmp/backups/
zip -qr backup.zip work/
rm -rf /tmp/backups/work
echo “Done Backing up the work directory”

[alice@sunshine ~]$ chmod 500 /opt/book/scripting/backup_v2
[alice@sunshine ~]$ /opt/book/scripting/backup_v2
Done Backing up the work directory

The chmod command sets the executable bit for the owner of the file. All others will not be able to execute the script. The first line of this version of the script (#! /bin/bash) tells the operating system that this file should be sent to the specified program for processing. We have also added a comment to the script. Any line that starts with a pound sign (#) is ignored by the BASH interpreter as a comment. Comments help you document how a script works, especially if you are working with complex logic. You can add a comment explaining what a particular statement should be doing and what the expected output is. Comments also allow you to add important information about the script such as the author and last modification date.

Once this line is added to the top of a text file, you can set the permissions to make the file executable and you have created a brand-new custom application. You can use these steps to create a script for any set of commands you need to repeat on a regular basis. Also, the length of the list of commands doesn't matter – you could have a list of a hundred commands or just one. It's often a good idea to create scripts for a single command if multiple command line options are required to accomplish a task, such as with curl or wget.

Output redirection

We've seen how to save multiple programs into a single script file, but there is another way to integrate multiple command line programs to accomplish a complex task. The output from one command can be used as the input for another one, creating what amounts to a script on a single line. This is possible because of the UNIX architecture's use of streams. “A stream is nothing more than a sequence of bytes that can be read or written using library functions that hide the details of an underlying device from the application. The same program can read from, or write to a terminal, file, or network socket in a device-independent way using streams.”¹ There are three standard I/O streams:

• Standard Input (stdin) provides input from the keyboard.
• Standard Output (stdout) displays output from commands to the screen.
• Standard Error (stderr) displays error messages to the screen.

These I/O streams can be easily redirected in BASH, allowing you to read input from a file instead of the keyboard, send one (or both) output streams to another program as input, or save the output to a file. The pipe (|) operator connects the stdout stream of one program to stdin of another program. As an example, we'll list all of the commands in /usr/bin that contain the word “gnome” in the filename:

[alice@sunshine ~]$ ls -l /usr/bin | grep gnome

When you run this command, you will receive about 50 files as a result. What if you just want the first three results? You can pipe the output from grep to another command:

[alice@sunshine ~]$ ls -l /usr/bin | grep gnome | head -3
-rwxr-xr-x. 1 root root  37070 Mar 20 2012 gnome-about
-rwxr-xr-x. 1 root root  88944 Jun 25 10:29 gnome-about-me
-rwxr-xr-x. 1 root root 233664 Jun 25 10:29 gnome-appearance-properties

The redirect operator (>) is used to send the output to a file instead of displaying it on screen. You can also append data to an existing file using the >> operator:

[alice@sunshine ~]$ ls -l /usr/bin | grep gnome | head -3 > /tmp/exam-
ple.txt
[alice@sunshine ~]$ cat /tmp/example.txt
-rwxr-xr-x. 1 root root    37070 Mar 20 2012 gnome-about
-rwxr-xr-x. 1 root root    88944 Jun 25 10:29 gnome-about-me
-rwxr-xr-x. 1 root root  233664 Jun 25 10:29 gnome-appearance-properties
[alice@sunshine ~]$ ls -l /usr/bin | grep gnome | head -5 >> /tmp/
example.txt
[alice@sunshine ~]$ cat /tmp/example.txt
-rwxr-xr-x. 1 root root   37070 Mar 20 2012 gnome-about
-rwxr-xr-x. 1 root root   88944 Jun 25 10:29 gnome-about-me
-rwxr-xr-x. 1 root root 233664 Jun 25 10:29 gnome-appearance-properties
-rwxr-xr-x. 1 root root   37070 Mar 20 2012 gnome-about
-rwxr-xr-x. 1 root root   88944 Jun 25 10:29 gnome-about-me
-rwxr-xr-x. 1 root root 233664 Jun 25 10:29 gnome-appearance-properties

Using multiple small programs in sequence instead of a single, complex application is central to the Unix design. The original developer of the I/O redirection system in Unix, Doug McIlroy, summed it us this way: “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”²

Text manipulation

Because the use and manipulation of text streams is so important to writing BASH scripts, we are going to spend a little bit of time on command line applications that specialize in manipulating text streams. Together, these commands are the “swiss-army knife” of text manipulation, providing everything from file sorting to case conversion and are used in virtually every BASH script regardless of size.

Cut

You will often find yourself dealing with columnar data that uses some form of separator, such as a tab or comma, to delimit each column in the data set. The cut command allows you to parse each line of the data file and extract only the column that you need. For this example, we'll be using a Comma-Separated Value (CSV) file exported from Excel that has the following fields: first name, last name, username, and email address. To extract the email address for all users:

[alice@sunshine ~]$ head -3 /opt/book/scripting/users.csv
Ian,Cook,ian.cook,[email protected]
Christine,Riggs,christine.riggs,[email protected]
Lindsay,Fishbein,lindsay.fishbein,[email protected]
[alice@sunshine ~]$ cut -d, -f4 /opt/book/scripting/users.csv
[email protected]
[email protected]
[email protected]
…

We can also return multiple columns and filter the output by combining the cut command with grep:

[alice@sunshine ~]$ cut -d, -f1,2,4 /opt/book/scripting/users.csv |
grep john
John,Jayavelu,[email protected]
Jennifer,Johnson,[email protected]
John,Altier,[email protected]
…

As you can see, it returned the first, second, and fourth columns and only returned those records that contained the string “john.”

Sort

The sort command does exactly what its name implies – it sorts the lines of a text file:

[alice@sunshine ~]$ cat /opt/book/scripting/words.txt
eyes
record
explosive
spice
prison
videotape
leg
ice
magnet
printer
[alice@sunshine ~]$ sort /opt/book/scripting/words.txt
explosive
eyes
ice
leg
magnet
printer
prison
record
spice
videotape

Be aware that the default sort order assumes text data, therefore the -n switch must be given if you are sorting numerical data:

[alice@sunshine ~]$ sort /opt/book/scripting/numbers.txt
1
1002
1234567
356
4
8675309
99
[alice@sunshine ~]$ sort -n /opt/book/scripting/numbers.txt
1
4
99
356
1002
1234567
8675309

uniq

Continuing the trend of simple commands that are named after their function, uniq removes duplicate lines from a text file. Uniq only searches adjacent lines to find duplicates, so input must be sorted first:

[alice@sunshine ~]$ cat /opt/book/scripting/duplicates.txt
apple
banana
orange
orange
kiwi
banana
kiwi
apple
[alice@sunshine ~]$ sort /opt/book/scripting/duplicates.txt | uniq
apple
banana
kiwi
orange

tr

The tr command substitutes the specified list of characters with a second set of characters or deletes (-d) them from the input stream. To substitute x, y, and z for all occurrences of a, b, and c in a text file:

[alice@sunshine ~]$ cat /opt/book/scripting/original.txt
The quick brown fox jumps over the lazy dog.
[alice@sunshine ~]$ cat /opt/book/scripting/original.txt | tr “abc”
“xyz”
The quizk yrown fox jumps over the lxzy dog.
[alice@sunshine ~]$ cat /opt/book/scripting/original.txt | tr -d
“abc”
The quik rown fox jumps over the lzy dog.

A more commonly used function of tr is to convert lower case text to uppercase and vise versa:

[alice@sunshine ~]$ cat /opt/book/scripting/original.txt | tr “[:lower:]”
“[:upper:]”
THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG.

[:lower:] and [:upper:] are character sets; they are a quick way to specify all lower and upper case letters, respectively. See the manual page for tr (man tr) for a full list of available character sets.

Variables

A variable is a representation of a piece of data (number, filename, text string, etc.) stored in the computer's memory. To create a new variable, you simply need to supply the variable name and the data it represents: its value

[alice@sunshine ~]$ myVariable=20
[alice@sunshine ~]$ echo $myVariable
20

No spaces are allowed before or after the = when assigning a value. Therefore, the following assignments will all result in an error:

[alice@sunshine ~]$ myVariable = 20
[alice@sunshine ~]$ myVariable =20
[alice@sunshine ~]$ myVariable= 20

You can also assign text or even another variable as a variable's value.

[alice@sunshine ~]$ hello=“Hello World”
[alice@sunshine ~]$ world=$hello
[alice@sunshine ~]$ echo $hello
Hello World
[alice@sunshine ~]$ echo $world
Hello World

Finally, you can assign the output from a command as the value of a variable by enclosing the command in $(), which is known as a command expansion.

[alice@sunshine ~]$ now=$(date)
[alice@sunshine ~]$ echo $now
Wed Dec 19 10:41:40 EST 2012

You can also do basic arithmetic with integers (whole numbers) in BASH by using the $(()) construct, which is referred to as an arithmetic expansion. A list of arithmetic operations that can be performed by BASH are listed in Table 10.1.

[alice@sunshine ~]$ myVariable=20
[alice@sunshine ~]$ myBigVariable=$(( $myVariable * 100 ))
[alice@sunshine ~]$ echo myBigVariable
2000
[alice@sunshine ~]$ echo $(( $myBigVariable + 1 ))
2001

Quoting

Enclosing a variable in double quotes (“ “) does not affect its use. However, single quotes (‘ ‘) will cause the variable name to be used literally instead of substituting the variable's value.

Listing 3: /opt/book/scripting/quoting

#! /bin/bash
name=Alice
echo “My name is $name and the date is $(date +%m-%d-%Y)”
echo ‘My name is $name date is $(date +%m-%d-%Y)’

[alice@sunshine ~]$ /opt/book/scripting/quoting
My name is Alice and the date is 12-19-2012
My name is $name and the date is $(date +%m-%d-%Y)

As you can see, the use of single quotes in the second line printed the literal variable names while the variable substitution took place on the first string. Also, note that the current date was substituted for $(date +%m−%d−%Y) without having to assign a variable name. Commands enclosed in $() are run each time they are encountered in a script and the value is determined dynamically.

Environment variables

Some variables are created automatically when you login or start a new terminal window. These environment variables hold default values and user preferences for the current terminal session. To view the list of environment variables and their values, use the env command:

[alice@sunshine ~]$ env
HOSTNAME=sunshine.edu
SHELL=/bin/bash
USER=alice
PATH=    (/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/
sbin:/usr/sbin:/sbin:/home/alice/bin
…
PWD=/home/alice
TERM=xterm

You will see several screens of data in the results, most of which are application-specific, but there are a few that are worth mentioning (Table 10.2).

These variables can be used in a command line just like regular variables:

[alice@sunshine ~]$ echo “My name is $USER and my current directory is
$PWD”
My name is alice and my current directory is /home/alice

We can also take advantage of these variables in our shell scripts. For example, look at listing 4.

Listing 4: /opt/book/scripting/env_variable_example

#! /bin/bash
echo “Hello $USER”
echo “You are calling this program from $PWD”
echo “Your home directory is $HOME”

Since the environment variables are created automatically by BASH, we can have dynamic output based on the user that is executing the script. Here is the output when the user Alice runs this script:

[alice@sunshine Desktop]$ /opt/book/scripting/env_variable_example
Hello alice
You are calling this program from /home/alice/Desktop
Your home directory is /home/alice

And here is the result when Bob runs it:

[bob@sunshine tmp]$ /opt/book/scripting/env_variable_example
Hello bob
You are calling this program from /tmp
Your home directory is /home/bob

The PATH variable is different from the other environment variables we have looked at. Instead of being used as part of a command, the value in the PATH variable is used directly by the BASH shell itself. When a user inputs a command, such as firefox to start a web browser, BASH looks for that command sequentially in each directory listed in PATH. You can use the which command to see a demonstration of the search in action:

[alice@sunshine ~]$ which firefox
/usr/bin/firefox
[alice@sunshine ~]$ which ThisProgramDoesNotExist
/usr/bin/which: no ThisProgramDoesNotExist in (/usr/lib/qt-
3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/
home/alice/bin)

Built-in variables

In addition to environment variables, BASH also defines several variables with useful values; collectively they are referred to as built-in variables. The built-in variables provide a wide array of small functions, from reporting on the type of hardware the server is running on to returning the status of the last command issued. There are dozens of built-in variables to choose from (see the BASH man page for a full list), but we will be working with a small subset (Table 10.3).

The script in listing 5 is an example of how these can be used.

Listing 5: /opt/book/scripting/builtin_variable_example

#! /bin/bash
echo “This script is executing with process ID: $$”
echo “OS: $OSTYPE Hardware: $MACHTYPE”
echo “This is he current date and time:”
date
echo “The exit value from date was $?”
echo “This command should fail:”
ls -l NoFile
echo “The exit value was $?”
echo “Wait 2 seconds”
sleep 2
echo “Here are 3 random numbers:”
echo $RANDOM
echo $RANDOM

echo $RANDOM
echo “Wait 3 seconds”
sleep 3
echo “This script has run for $SECONDS seconds”

[alice@sunshine ~]$ /opt/book/scripting/builtin_variable_example
This script is executing with process ID: 10380
OS: linux-gnu Hardware: i386-redhat-linux-gnu
This is the current date and time:
Wed Dec 19 11:41:40 EST 2012
The exit value from date was 0
This command should fail:
ls: cannot access NoFile: No such file or directory
The exit value was 2
Wait 2 seconds
Here are 3 random numbers:
10549
319
20535
Wait 3 seconds
This script has run for 5 seconds

Conditionals

In the last section, we introduced the $? variable and that it returns the exit value from the last command run. What if you wanted to take one action if the command succeeds ($? equal to 0) and another action if it fails? Like any other programming language, BASH provides constructs that test a set of given conditions and act based on the result of the test.

If/then

The most basic form of conditional is the if/then construct. The if command checks the exit value of a series of comparison statements. If the exit value equals zero, the commands in the then stanza are executed. The entire construct is terminated by the ficommand.

#! /bin/bash
if [ “$USER” = “alice” ]
then
     echo “Good Morning, Alice!”
fi

If the user running this script has the username alice (“$USER” = “alice”), the echo command is executed. If their username is anything else, the script completes without executing any commands.

The syntax of the if/then construct in BASH is a bit different than that of most other programming languages.³ The most common mistake when writing an if/then statement in BASH is not separating elements with spaces. You must put a space between the if and the square brackets and around the comparison statement inside the brackets. Most other languages are more forgiving in their use of white space. Any of these attempts at an if statement will fail with an error:

if[”$USER” = “alice”]
if [”$USER” = “alice”]
if[ “$USER” = “alice” ]

Another difference in the use of if/then in BASH compared to other languages is that string comparison such as the example above uses a different operator (=) than a numerical comparison (-eq) would. The following chart has a list of the comparison operators with examples of their use.

If/then/else

If you want to take one action if a conditional statement is true and a different action if it is false, you'll use the if/then/else construct. It is identical to the if/then construct except that for the addition commands in the else stanza that will be executed if the conditions were not met. /opt/book/scripting/number_guess_v1 is an example of a basic if/then/else construct. We will be building on this basic example to develop a more complex application over the rest of the chapter.

Listing 6: /opt/book/scripting/number_guess_v1

#! /bin/bash
guess=2
number=$(( ( $RANDOM % 100) + 1 ))
#Is the guess correct?
if [ $guess -eq $number ]
then
     echo “Correct Guess: The number is $number”
else
      # Is the guess high?
      if [ $number -lt $guess ]
then
      echo “Guess lower: The number is less than $guess”
fi
     # Is the guess low?
if [ $number -gt $guess ]
then
     echo “Guess higher: The number is greater than $guess”
fi
fi

Because guess is equal to 2 and number is 5, the first if/then statement will always be false (we'll add user-supplied guesses later in the chapter), and the program should always execute the code in the else stanza. At this point, the script does something we haven't seen before: a nested-if statement.

A nested-if statement is just a normal if/then statement inside another if or else statement.

if [ condition1 ]
then
   echo “condition1 is true”
else
   #Nested-if statement
if [ condition2 ]
   then
        echo “condition2 is true”
    else
        echo “Neither condition is true”
    fi
fi

If condition1 is true, the else section is skipped, so the nested-if statement is not executed at all and no tests are run on condition2. However, if condition1 is not true, the nested-if is executed and condition2 is checked. In the number guessing script, since guess is less than 5, the first of the two nested-if statements will return true and it should print Guess higher: The number is greater than 2. We can run the script to test the output:

[alice@sunshine ~]$ /opt/book/scripting/number_guess_v1
Guess higher: The number is greater than 2

If/then/elif

The final construct we'll be looking at is the if/then/elif construct. elif is a contraction of “else if” and is an alternative to nested-if structures. The example from above can be written using elif as follows:

if [ condition1 ]
then
   echo “condition1 is true”
elif [ condition2 ]
then
     echo “condition2 is true”
else
     echo “neither condition is true”
fi

Multiple elif stanzas can be added to an if statement if there are more than two conditions to be checked. As an example, we'll update the number guessing script as /opt/book/scripting/number_guess_v2

Listing 7: /opt/book/scripting/number_guess_v2

#! /bin/bash
guess=2
number=$(( ( $RANDOM % 100 ) + 1 ))
#Is the guess correct?
if [ $guess -eq $number ]
then
    echo “Correct guess: The number is $number”
# Is the guess high?
elif [ $number -lt $guess ]
then
echo “Guess lower: The number is less than $guess”
# Is the guess low?
elif [ $number -gt $guess ]
then
echo “Guess higher: The number is greater than $guess”
fi

The use of elif instead of a nested-if statement makes the code slightly shorter and easier to read. We can run the code to verify that it returns the same results:

[alice@sunshine ~]$ /opt/book/scripting/number_guess_v2
Guess higher: The number is greater than 2

User input

The values for all of the variables in the scripts we have looked at so far were hardcoded. They were defined in the script itself and the only way the values can be changed is by modifying the script. In many cases, this is fine, but you may also want values that are supplied by the user. There are two ways to accept input from the user: command line arguments and the read command.

Command line arguments

Like other commands that you execute in a terminal window, BASH scripts can accept program arguments. The arguments are automatically stored in special variables when the program executes. The variables are named with a number in the order the arguments were given on the command line:

Listing 8: /opt/book/scripting/user_input_ex1

#! /bin/bash
echo “The first argument: $1”
echo “The second argument: $2”
echo “The third argument: $3”

[alice@sunshine ~]$ /opt/book/scripting/user_input_ex1 42 “Hello World”
Earth
The first argument: 42
The second argument: Hello World
The third argument: Earth

Note that the second argument consists of two words (“Hello World”). The quotation marks around the group of words tells BASH that this is a single argument.

So, now we can accept arguments from the command line, but how can we ensure that the correct number of arguments are entered? BASH includes another special variable, $#, which stores the total number of arguments that were entered. This allows you to test the number of entered arguments against the expected number and print an error message if the test fails.

Listing 9: /opt/book/scripting/user_input_ex2

#! /bin/bash
if [ $# -eq 3 ]
then
echo “The first argument: $1”
echo “The second argument: $2”
echo “The third argument: $3”
else
    echo “Three arguments are required!”
fi
[alice@sunshine ~]$ /opt/book/scripting/user_input_ex2 42 Earth
Three arguments are required!

Reading user input

The other option for including user input into your scripts is the read command. read pauses the execution of a script until the user enters a value and presses return. To demonstrate the use of read, we will update the the number guessing script:

Listing 10: /opt/book/scripting/number_guess_v3

#! /bin/bash
#Prompt for user input
echo “Enter a number between 1 and 100 and press [ENTER]: ”
read guess

number=$(( ( $RANDOM % 100 ) + 1 ))
#Is the guess correct?
if [ $guess -eq $number ]
then
     echo “Correct guess: The number is $number”
# Is the guess high?
elif [ $number -lt $guess ]
then
echo “Guess lower: The number is less than $guess”
# Is the guess low?
elif [ $number -gt $guess ]
then
echo “Guess higher: The number is greater than $guess”
fi

[alice@sunshine ~]$ /opt/book/scripting/number_guess_v3
Enter a number between 1 and 100 and press [ENTER]: 15
Guess lower: The number is less than 15

Loops

One of the most useful aspects of BASH scripting (and computer programming in general) is the ability to reduce repetitive tasks down to a few simple commands. Instead of typing the same or similar commands over and over again, a loop allows you to write the commands you wish to execute once and then have the shell repeat them. We will be working with two types of loops available in BASH scripts:

1. for loops – These loops repeat the commands using a given list of input items
2. while loops – These loops repeat the commands while a given condition is true

For loops

For loops are the most basic and the most commonly used looping construct in BASH scripts. A for loop iterates over a list of items, executing any commands contained in the loop during each iteration. When the BASH intepretor reaches the keyword “done,” it jumps back to the beginning of the loop and begins the next iteration. During each successive pass through the loop, the value of the loop variable (var in the following example) is changed to the current element in the list. /opt/book/scripting/for_loop_example1 is an example of a simple for loop:

Listing 11: /opt/book/scripting/for_loop_example

#! /bin/bash
for var in “item1” “item2” “item3”
do
    echo “The current item is $var”
#More commands could be added here
done

When run, you can see that the value of $var changes with each iteration:

[alice@sunshine ~]$ /opt/book/scripting/for_loop_example1
The current item is item1
The current item is item2
The current item is item3

In addition to listing each item on the command line, you can also use the output of a command as the list of items to iterate over.

Listing 12: /opt/book/scripting/for_loop_example2

#! /bin/bash
for word in $(head -3 /opt/book/scripting/words.txt)
do
   echo “Original word: $word”
   echo “All uppercase: $(echo $word | tr ‘[:lower:]’ ‘[:upper:]’)”
done
[alice@sunshine ~]$ /opt/book/scripting/for_loop_example2
Original word: eyes
All uppercase: EYES
Original word: record
All uppercase: RECORD
Original word: explosive
All uppercase: EXPLOSIVE

As you can see, the first three lines in /opt/book/scripting/words.txt (eyes, record, explosive) were used as the list of items to iterate over. The first command in the loop is a simple echo statement, printing the value of $word, but the second command is a little more complex. In this command, we piped the value of the $word variable to tr and converted the lower case characters to upper case (echo $word | tr ‘[:lower:]’ ‘[:upper:]’), the output of that command is then printed to the screen.

Internal field separator

When reading the output from a command in a for loop, BASH determines the separation between each item by using a special internal variable $IFS, the internal field separator. The variable contains a list of characters which are used as field boundaries; when one is found, a new item for the for loop is created. The default values in $IFS are the white space characters (space, tab, and newline), but the list can be modified, for example, to parse a comma-separated list or to ignore one of the defaults as a separator and allow it as part of an item.

Listing 13: /opt/book/scripting/ifs_example1

!# /bin/bash
for line in $(tail -3 /etc/passwd)
do
    echo $line
done
[alice@sunshine ~]$ /opt/book/scripting/ifs_example1
russell.dacanay:x:1648:100: ”Russell
Dacanay
(Staff-Library)”:/home/staff/russell.dacanay:/bin/bash
daniel.saddler:x:1649:100: ”Daniel
Saddler
(Staff-Student
Services)”:/home/staff/daniel.saddler:/bin/bash
russell.lavigne:x:1650:100: ”Russell
Lavigne
(Staff-Academic
Affairs
VP
Office)”:/home/staff/russell.lavigne:/bin/bash

As you can see, using the default value for $IFS, the lines from /etc/passwd are broken up in the middle of fifth column because of the space or spaces in the text of that field. To fix this issue, we will set $IFS to contain only the newline character ($’ ’).

Listing 14: /opt/book/scripting/ifs_example2

!# /bin/bash
#Change IFS to the newline character only
IFS=$’
’
for line in $(tail -3 /etc/passwd)
do
    echo $line
done
[alice@sunshine ~]$ /opt/book/scripting/ifs_example1
russell.dacanay:x:1648:100: ”Russell Dacanay (Staff-Library)”: 
/home/staff/russell.dacanay:/bin/bash
daniel.saddler:x:1649:100: ”Daniel Saddler (Staff-Student Services)”: 
/home/staff/daniel.saddler:/bin/bash
russell.lavigne:x:1650:100: ”Russell Lavigne (Staff-Academic Affairs VP
Office)”: 
/home/staff/russell.lavigne:/bin/bash

Not that the backslash () in the output above is a line-continuation character, which is used because the output is too long to be printed on a single line. If you run the script in your Linux virtual machine, you will find the line with the backslash and the one after it are printed as a single line.

Sequences

You will often need to execute an action a specific number of times or use a sequence of numbers as the input for a loop. Since version 3.0,⁴ BASH has included a built-in syntax for generating a sequence of numbers as input for a for loop. Number sequences are surrounded by curly braces ({ }) and the arguments are separated by two periods (..). Sequences can be created using either two or three arguments – if two arguments are given, the first is the starting value and the second is the ending value. The loop is then executed using each integer from the starting to the ending value.

Listing 15: /opt/book/scripting/sequence_example1

#!/bin/bash
for number in {1..5}
do
    echo $number
done

[alice@sunshine ~]$ /opt/book/scripting/sequence_example1
1
2
3
4
5

You can also increment numbers backward by listing a higher number as the starting value and a lower number as the ending value.

Listing 16: /opt/book/scripting/reverse_sequence

#!/bin/bash
for number in {10..1}
do
    echo $number
done

[alice@sunshine ~]$ /opt/book/scripting/reverse_sequence
10
9
8
7
6
5
4
3
2
1

If three arguments are given,⁵ the third argument determines what the increment between each number in the series should be.

Listing 17: /opt/book/scripting/sequence_example3

#!/bin/bash
for number in {1..10..2}
do
   echo $number
done

[alice@sunshine ~]$ /opt/book/scripting/sequence_example3
1
3
5
7
9

Note that “10” was not part of the results returned. This is because the number sequence contains all of the numbers that are less than or equal to the ending value. Since the sequence is incrementing by two, the next number in the series would be eleven, but that is greater than our ending value of 10.

Break and continue

Under certain conditions, you may want to stop the processing of a loop or skip ahead to the next iteration of a loop. The break and continue keywords give you this ability. The break keyword stops the processing of a loop, skipping any remaining commands in the current iteration of the loop, and skipping all of the remaining items in the input list. Execution of the script is not interrupted; however, it will continue to execute any commands after the loop.

Listing 18: /opt/book/scripting/break_example

#!/bin/bash
for number in {1..5}
do
if [ $number -eq 4 ]
then
   echo “Stop!”
   break
fi
echo “$number”
done
echo “This command runs AFTER the loop is complete.”

[alice@sunshine ~]$ /opt/book/scripting/break_example
1
2
3
Stop!
This command runs AFTER the loop is complete.

Note that the loop is processed as expected for the first three numbers in the sequence. When the condition in the if statement ($number -eq 4) is met, “Stop!” is printed to the screen, the break keyword is reached, and execution of the loop ends. As another example, we will update the number guessing script to give the user five chances to guess the number and break if the number is guessed early.

Listing 19: /opt/book/scripting/number_guess_v4

#! /bin/bash
number=$(( ( $RANDOM % 100 ) + 1 ))
#Give the user 5 guesses
for loop in {1..5}
do
#Prompt for user input
echo “Enter a number between 1 and 100 and press [ENTER]: ”
read guess
echo “”
#Is the guess correct?
if [ $guess -eq $number ]
then
     echo “Correct guess: The number is $number”
     echo “You guessed it in $loop tries”
          break
# Is the guess high?
elif [ $number -lt $guess ]
then
     echo “Guess number $loop”
     echo “Guess lower: The number is less than $guess”
# Is the guess low?
elif [ $number -gt $guess ]
then
  echo “Guess number $loop”
  echo “Guess higher: The number is greater than $guess”
fi
end

[alice@sunshine ~]$ /opt/book/scripting/number_guess_v4
Enter a number between 1 and 100 and press [ENTER]: 15

Guess number 1
Guess lower: The number is lower than 15
Enter a number between 1 and 100 and press [ENTER]: 3

Guess number 2
Guess higher: The number is higher than 3
Enter a number between 1 and 100 and press [ENTER]: 5

Correct guess: The number is 5
You guessed it in 3 tries

The continue keyword skips the remaining commands in the current iteration of the loop and begins the next iteration. In the next example, we'll use the same code from listing 13 but replace the break keyword with continue.

Listing 20: /opt/book/scripting/continue_example

#!/bin/bash
for number in {1..5}
do
if [ $number -eq 4 ]
then
       echo “Stop!”
       continue
fi
echo “$number”
done

echo “This command runs AFTER the loop is complete.”

[alice@sunshine ~]$ /opt/book/scripting/continue_example
1
2
3
Stop!
5

This command runs AFTER the loop is complete.

Note the difference in the output between this script and the script in listing 18. Again, the first three iterations of the loop are processed as expected and the condition of the if statement is met on the fourth iteration. However, instead of exiting from the loop, processing continues with the fifth (and final) iteration of the loop.

While loops

Instead of operating on a list of items like a for loop, while loops will continue running until a specific condition is met. Before starting an iteration of the loop, the condition is tested. If the result is true, the commands inside the loop are executed. If it is false, the loop is skipped and the rest of the script is executed.

Listing 21: /opt/book/scripting/while_loop_example1

#! /bin/bash
counter=1
while [ $counter -le 5 ]
do
    echo $counter
    $(( counter=$counter + 1 ))
done

[alice@sunshine ~]$ /opt/book/scripting/while_loop_example1
1
2
3
4
5

As you can see, the output from this command is similar to some of the for loop examples that we have looked at previously. However, the script itself has some major differences. The first difference you'll notice is that unlike when using a for loop, we defined the initial value of the counter variable before executing the loop.

When the BASH interpreter reaches the while statement, if the current value of counter is less than or equal to five, the commands inside the loop are executed. During the loop, the current value of counter is printed to the screen and then increased by one ($((counter=$counter+ 1))). At this point, the value of counter is tested again and if it is still less than or equal to five, the loop continues.

When writing a script, you may need to create an infinite loop: a loop that continues until the user explicitly ends the script. They are typically used when you need to monitor something at regular intervals, such as the size of a file or the number of logged-in users. To create an infinite loop, you'll create a while loop whose test condition always evaluates as true. You can see the use of an infinite loop to monitor the size of a log file (/var/log/httpd/access_log) in listing 14. With each iteration, the time the file was checked and the size of the log file is printed to the screen. The script will loop until the user exits the script, either by pressing the “CTRL” and “C” keys together or closing the terminal window the script is running in.

Listing 22: /opt/book/scripting/while_loop_example2

#! /bin/bash
echo “This script will loop forever. Hit Control+C (CTRL+C) to exit.”
while [ true ]
do
    sleep 2
    echo “”
    date
    echo “$(wc -l /var/log/httpd/access_log)”
done

[alice@sunshine ~]$ /opt/book/scripting/while_loop_example2
This script will loop forever. Hit Control+C (CTRL+C) to exit.

Fri Jan 4 08:11:00 EST 2013
  7 /var/log/httpd/access_log

Fri Jan 4 08:11:02 EST 2013
  7 /var/log/httpd/access_log

Fri Jan 4 08:11:04 EST 2013
  8 /var/log/httpd/access_log

Fri Jan 4 08:11:06 EST 2013
  9 /var/log/httpd/access_log

To test this script, you'll need to open the web browser and visit http://www.sunshine.edu after starting the script. The number of entries in the log file will increase each time you load the web page.

Putting it all together

You've now seen the basic pieces of a shell script; let's look at a script that uses many of these elements to automate a process across all of the users on the system. The Linux virtual machine included with this text has over 1,000 accounts, far too many to support by hand. This script extracts important information for each account and displays it in an easy to read format.

Listing 23: /opt/book/scripting/user_info

#! /bin/bash
#This script returns import information about all users on the system

#Example line from /etc/passwd
#alice:x:501:501:Alice Adams:/home/alice:/bin/bash
for user in $(cut -d: -f1 /etc/passwd)
do
    IFS=$’
’
    #Grab the line from the password file that
    #contains this user's info. We append the
    #delimiter (:) to ensure we only get results
    #for this username and not similar users
    userinfo=$(grep $user: /etc/passwd)

    comment=$(echo $userinfo | cut -d: -f5)
    home=$(echo $userinfo | cut -d: -f6)
    groups=$(groups $user | cut -d: -f2)

   #We only want this to run on “regular” users,
   #not system accounts. Skip users that do not
   #have ‘/home’ in the path to their home directory
    if [ $(echo “$home” | grep -v ‘/home/’) ]
    then
        continue
    fi

    echo “Username: $user”
    echo “User Info: $comment”
    echo “Home Directory: $home”
    echo “Groups: $groups”

    echo “Disk usage: $(du -sh $home)”

    last=$(last $user | head -1)

    if [ $(echo $last | wc -c) -gt 1 ]
    then
        echo “Last login: ”
        echo “$last”
    else
        echo “User has never logged in!”
    fi
    echo “”
    echo “--”
    echo “”
done

[alice@sunshine ~]$ /opt/book/scripting/user_info
Username: alice
User Info: Alice Adams
Home Directory: /home/alice
Groups: alice sys
Disk Usage: 75M /home/alice
Last login:
alice pts/3 sunshine.edu Sun Jan 13 12:22 - 13:00 (0:48)
--
Username: bob
User Info: Bob Brown
Home Directory: /home/bob
Groups: bob
Disk Usage: 1.1M /home/bob
Last login:
bob pts/6 sunshine.edu Sun Jan 6 16:48 - 18:46 (1:58)
--

Let's look at this script in depth. In the first few lines, we set up a loop using all of the username on this system. The username is always the first column in /etc/password.

for user in $(cut -d: -f1 /etc/passwd)
do

For each account, we search the /etc/password file to find the account information for the user.

IFS=$’
’
userinfo=$(grep $user: /etc/passwd)

The next section of the script uses cut to separate the columns of the account information into usable variables. It also uses the groups command to get the list of groups the user is a member of and the du command to calculate the amount of storage their home directory is using.

comment=$(echo $userinfo | cut -d: -f5)
home=$(echo $userinfo | cut -d: -f6)
groups=$(groups $user | cut -d: -f2)
echo “Username: $user”
echo “User Info: $comment”
echo “Home Directory: $home”
echo “Groups: $groups”
echo “Disk usage: $(du -sh $home)”

The final portion of the script uses the last command to get the latest login for this user. If the user has never logged in, the result from last will be a blank line and the script will display an error message. If the user has logged in, the last time the user has logged in and the duration of that login session are displayed.

last=$(last $user | head -1)
if [ $(echo $last | wc -c) -gt 1 ]
then
  echo “Last login:”
  echo $last
else
  echo “User has never logged in!”
fi

Example case–Max Butler

In 1998, Max Butler was a 26-year-old computer enthusiast, earning over $100/hour testing the security of corporate clients while also volunteering for the San Francisco office of the FBI. That year, a critical security flaw was found in the most popular open-source DNS server used on the Internet – BIND. BIND was used on virtually all servers to map URLS such as “www.usf.edu” to IP addresses such as 131.247.88.80. The flaw would allow hackers to get complete control of any server running an unprotected version of BIND. In particular, almost all US Department of Defense servers ran BIND. If these servers were to be protected from attackers, they were to be patched before attackers got to them. But the military bureaucracy can be slow. How is a concerned security expert with all the innocence of a 20-year-old to fix the problem asap?

Enter scripting. A script could operate at the speed of a computer and instruct hundreds of computers every second to download a patch and fix themselves. Max Butler did just that, putting together a script that located any server running the unpatched version of BIND and updating it with the specified patch. While he was at it, Max also modified the patch so that it created a backdoor that only Max was aware of. This way, Max thought he was protecting the computers from attackers, while at the same time, giving him unrestricted access to the same computers so he could go in and fix them all by himself the next time a vulnerability was reported. No need to waste time contacting DoD administrators.

The good deed worked, but unfortunately for Max, the backdoor was not seen kindly. When DoD system administrators found about it, they prosecuted Max. On May 21, 2001, Max was sent to prison for 18 months for the deed.

This was not Max's last brush with cybercrime or prison. He later went on to command the majority of illicit online credit card marketplaces. On February 12, 2010, he was sentenced to 13 years in prison for the offense, then the longest sentence ever awarded for computer crime. This was later eclipsed by the sentence awarded to Albert Gonzales for the T.J. Maxx case. Max is currently serving the sentence at the Yankton Federal Prison Camp, a minimum security facility in South Dakota. He is scheduled to be released on 1/1/2019. CNBC produced a case file segment on him, called “American greed.”

Feb 17 08:00:08 inigo sshd[7049]: Failed
password for root from
61.136.171.198 port 59146 ssh2
Feb 17 08:00:09 inigo sshd[7049]: Received
disconnect from
61.136.171.198: 11: Bye Bye [preauth]
Feb 17 08:00:16 inigo sshd[7051]: pam_
unix(sshd:auth): authentication
failure; logname= uid=0 euid=0 tty=ssh
ruser= rhost=61.136.171.198
user=root
Feb 17 08:00:18 inigo sshd[7051]: Failed
password for root from
61.136.171.198 port 59877 ssh2
Feb   17   08:00:19   inigo   sshd[7051]:
Connection closed by 61.136.171.198
[preauth]
Feb 17  08:17:01  inigo  CRON[7296]: pam_
unix(cron:session): session
opened for user root by (uid=0)
Feb  17 08:17:01  inigo CRON[7296]: pam_
unix(cron:session): session
closed for user root