The shell program you use affects the feel of your command-line interface. Some of the features that can be built into the shell program include a simple arithmetic interpreter that lets you use the command line as a calculator; command aliasing, which lets you refer to standard Unix commands with other more convenient words; filename completion, which lets you type only the number of characters necessary to distinguish a file from other files in the directory, rather than typing the full filename; command editing and command history, which let you scroll back through the commands you've recently issued and edit them on the command line; spelling correction; and help functions for the shell program.

There are a number of common shell programs on Unix systems. You are automatically assigned a shell when your system administrator sets up your account. On Linux systems, the default shell program is the bash (Bourne Again) shell. However, you may prefer to use a shell other than bash. The two main classes of shell programs are shells derived from the Bourne shell, sh, and shells derived from the C shell csh. Bourne-type shells include sh, bash, ksh (the Korn shell), and zsh (the Z shell). C-type shells include csh and tcsh.

We tend to prefer C shells, for historical reasons. When we started working in Unix, the C shell was the best thing going, and the tcsh program has expanded the original csh into a powerful shell. tcsh implements most of the desirable shell features, including history, command aliasing, filename completion, command-line editing, arithmetic and functions, job control, and spelling correction. tcsh is also one of the most user-configurable shells. Therefore, we'll discuss the behavior of Unix commands from a C-shell perspective, as if you were using the tcsh program, which we use on our machines.

Your default shell will be listed as the last item in your entry in the /etc/passwd file. If you aren't certain which shell you are currently using, you can find out by typing:

% finger your-user-name

For user jambeck, this command shows the following information:

Login name: jambeck           In real life: Per Jambeck 
Directory: /home/jambeck      Shell: /bin/tcsh

This tells us that he is using tcsh as his default shell. For practical reasons, we will limit our discussions and most references to csh and tcsh. It must also be noted that many system processes (e.g., batch, at, and cron) use the Bourne shell by default, which makes it necessary to learn at least a minimal subset of its command language. On most systems there are commands to change your default shell as set in the passwd file. The chsh (change shell) command allows you to change your default login shell, if you're working on a Linux system.

The command line consists of the command itself, optional arguments that modify how the command works, and operands such as files upon which the command operates. For example, the chsh (change shell) command, which we just discussed briefly, has several possible options. The first is the -s option, which must be followed by the name of a shell program as its argument. The second is the -l option, which needs no argument, and which lists the shells that are available on your system. The operand for the chsh command is the username of the user whose shell is being changed. So, to change your default shell program, you might first type:

% chsh -l

which gives you a list of the shell programs available on the system:

/bin/bash 
/bin/sh 
/bin/ash 
/bin/bsh 
/bin/bash2 
/bin/tcsh 
/bin/csh 
/bin/ksh 
/bin/zsh 

Then, to actually change your shell to tcsh, you can type:

% chsh -s /bin/tcsh yourusername

Options can simply be single-letter codes, or they can have their own arguments. Options that take no arguments can be given as a group, while each option that takes an argument must be specified separately. Each option group and separate option must be preceded by a hyphen (-). The last option in a group, or separate options, can be followed by the option argument. The operands follow the final option in the list.

Many Unix commands have options that, frankly, you'll never use. And we're not going to talk about them. But there are ways of finding out more.

Unix has its own built-in reference manual, which is quite comprehensive and informative, and which will give you the correct information about the commands and options available on the particular system you're using.

The man command is one of the most useful Unix commands; it allows you to view Unix manual pages. While some Unix systems have implemented a web browser-like interface to the Unix manpages, you can't always count on this option being available. The man command is available on all types of Unix systems.

where name can be a Unix command, such as grep, or a system file, such as the password file /etc/passwd.

If you're not sure of the command you're looking for, you can sometimes find the right information using man's slightly smarter cousin, apropos. The apropos command locates commands by keyword lookup.

For instance, if you're concerned about disk usage on your system, you can enter apropos usage. The output of this command on our PC running Red Hat Linux is:

du (1)	- summarize disk usage 
getrlimit, getrusage, setrlimit (2)	- get/set resource limits and usage
quota (1)	- display disk usage and limits
quotacheck (8)	- scan a file system for disk usages

apropos doesn't always produce such brief and informative output. Entering a smart combination of keywords is (as always with such searches) the key to getting the output you want. If you want a predictable listing of Unix commands, it's probably best to pick up a comprehensive Unix book.

What should you do if you find the following text in a manpage?

This documentation is no longer being maintained and may be inaccurate or 
incomplete. The Texinfo documentation is now the authoritative source. 

The GNU^[*] set of Unix tools are adopting a documentation system, called texinfo, that is different from the traditional man system. If you come across this message, you should be able to read the up-to-date documentation on the program by typing in the command info progname. For instance, info info gives you a complete set of documentation on the use of info and even provides instructions for creating your own info documentation when you start writing your own programs.

By default, many Unix commands read from standard input and send their output to standard output. Standard input and output are file descriptors associated with your terminal. A program reading from standard input will simply hang out and wait for you to type something on your keyboard and press the Enter key. A program writing to standard output spews its output to your terminal, sometimes far faster than you can read it.

Some Unix commands read a hyphen (-) surrounded by whitespace on either side as "data from standard input." This construct can then be used in place of a filename in the command line. Absence of an output filename is sufficient to cause the program to write to standard output.

The standard input and output descriptors are useful because you can redirect both standard input and output, associating them with filenames, with no effects on the functioning of the program. Here are the most common redirection constructs used by the C shell:

The cat command reads the contents of a file and writes them to standard output. If you want to use the cat command to combine the contents of three files into one new file, you can use a redirector like this:

% cat file1 file2 file3 > file4 

This construct with cat would be useful if, for example, you'd just downloaded a bunch of individual sequence files from the NCBI web site and want to collect them into one large file that can be read by another program. (This is an example of something that seems like it should be simple, but is actually time-consuming and annoying to do with a standard PC word-processing program. Unix provides a neat solution that doesn't even require you to open any files).

You can also use redirectors to direct the contents of a file into a program at run-time, as standard input (useful if you are running a program that prompts you for input from the keyboard) or to capture output from a program that is normally written to standard output:

program < inputfile
program > outputfile

For example, let's say you've just finished an extensive BLAST search, and you want to send the results to your colleague. You can use the redirector < ("less than"), to scoop the file huge_blast_report out of your directory and mail it directly to your colleague:

% mail [email protected] < huge_blast_report

If you want to increase the chances of your colleague opening the message, you can add a subject header to the mail message using the mail option -s. The command reads:

% mail -s "surprise!" [email protected] < huge_blast_report

The reverse operation, sending the results of standard output (or text that's displayed on your screen) to a file, can be accomplished using > ("greater than"). Perhaps your colleague wants to write a quick reminder to herself to reply to your mail. She could do it using the cat command to take input from the keyboard and redirect it to a file, like this:

% cat > reminder_to_self
Ha! Send fifteen BLAST reports to colleague on Monday.
^D
%

Ctrl-D (^D) signals that you have finished entering text. Your colleague now has a file called reminder_to_self in her current working directory.

Operators are similar to redirectors in that they are ways of directing standard input and output. However, they direct input and output to and from other commands rather than to filenames.

The most commonly used operator is the pipe (|). The pipe directs standard output of one command into standard input for the next command. This allows you to chain together several different filtering commands or programs without creating input or output files each time.

You can use the cat command to direct the contents of a file into a program that reads information from standard input:

% cat inputfile | program

This command construct does the same thing as the example we showed earlier (program < inputfile). Both cause the output of the cat command to act as input for program. If you want to do a lot of runs of the same program using slightly different input, you can create multiple input files and then write a script that cat s each of those input files in turn and pipes their contents to program.

Pipes can carry out a complete set of file-processing options without writing to disk. For instance, imagine that you have a datafile consisting of multiple tables concatenated together. The first table in the file takes up the first 67 lines, the second table takes up the next 100 lines, and the rest of the file is taken up by a third table.^[†] You want the information that's contained in the second column of the middle table, which stretches from characters 30 -39 in the row. Using filters and pipes, you can construct the following command to crop out the data you need:

% head -167 protein1.pka | tail -100 | cut -c30-39 > protein1.pka.data 

In this example, head sends the top 167 lines of a specified file or files (in this case protein1.pka) to standard output; tail takes the last 100 lines of the output of head; and cut takes the correct column of characters out of the results of head and tail and then stores it in protein1.pka.data.

A useful construct Unix shells recognize is the presence of wildcard characters in filenames. The shell locates matches for any wildcards before passing filenames on to the program. The two most commonly used wildcards are the asterisk (*) and the question mark (?). * means "any sequence of zero or more characters, except for the / character." ? means "any single character." Thus, "every file in this directory" can be denoted by a lone *, which is a useful shortcut.

The shell recognizes other wildcards as well. The construct [cset ] refers to any characters in the specified set. If you want to move all files beginning with letters a through m to a new directory, you can structure the command as mv [a -m]* ../newdir. If you want to move all files beginning with a number to a new directory, enter mv [0 -9]* ../newdir.

On Unix systems running the X Window System, there are many commands available that initiate programs with functions that aren't command line-based. Once these programs, which can include anything from graphics viewers to complicated scientific applications, are called from the command line, they use the X Window System to open their own windows, which generally contain a complete, independent graphical user interface.

cat dumps the contents of a file onto the screen. If your file is short, or if you've successfully completed a speed-reading course, this utility works well. If you need to see what's on each page of a file, though, cat is less useful, since the contents of the file scroll by without pausing.

Instead of viewing text, cat is most useful for combining (or concat enating) files. For instance, if you have a series of files of program output named meercat1.txt, meercat2.txt, and meercat3.txt, and you want to combine them into a single file, you can type:

% cat meercat1.txt meercat2.txt meercat3.txt > big-meercat.txt 

This command appends the contents of meercat3.txt to the end of meercat2.txt, the contents of meercat2.txt to the end of meercat1.txt, and so on, combining them into one big file named big-meercat.txt. If you've thought to number the outputs sequentially (as we have with the meercats), and want them in that order in the file, you can just type:

% cat meercat*.txt > big-meercat.txt

and it will have the same effect. Wildcard characters such as * use a strict alphabetical order: if they exist, files meercat10.txt and meercat11.txt come before meercat2.txt.

cat can also append files to the end of an existing file. For example, if your program generates another output file you need to attach to the end of the collection, the command:

% cat meercat10.txt >> big-meercat.txt

does just that. If you use > instead of >> in this situation, instead of being added at the end of the file, the new file meercat.txt overwrites the entire contents of big-meercat.txt.

Incidentally, if you want a command that's the reverse of cat to print the lines of a file in backward order, you're in luck: the command is called tac. Sadly, the command acta, for printing a file inside out, hasn't yet been implemented.

more is a pager, which in Unix means a program that lets you view a file one page at a time. Suppose you have a file containing BLAST output named blast-first.txt. Typing:

% more blast-first.txt 

shows you the first page of the file blast-first.txt, and steps forward one page every time you press the space bar. To leave more, hit the q key; to view other more commands while within more, enter h.

more is smart about moving around files. If you know where you want to go in the file, you can specify the line number (using the + linenumber option). If, on the other hand, you want to start at the first occurrence of a certain word or pattern, use the +/ pattern option. When viewing a file in more, if you press the / key and then type a pattern to search for, more jumps to the next occurrence of that pattern in the file and repeats searches for each subsequent occurrence of that pattern every time you press / followed by the Enter key.

Here are some other useful options for more :

You can redirect the output of a program that generates more than a screen's worth of text to more, allowing you to page through the output one screen at a time. Let's say you want to know who is logged into your Unix system. If enough users are logged in, the output scrolls off the screen. By piping who to mor e :

% who | more

you can scroll through the output line-by-line using the Return key or screen-by-screen using the space bar.

more 's most significant shortcoming is that some versions can't move backward through a file. less is a utility that remedies this simple problem.

There is a superior pager command, less . Most importantly, less rectifies more 's biggest flaw: it lets you page backward as well as forward in a file. less also doesn't load a file into memory all at once, which makes it less likely that your computer will grind to a halt if you view a huge file with it. Finally, it also handles binary files more gracefully, displaying readable text as characters and representing unreadable control characters in the form ^X. less uses the same options as more, but it also takes additional options. Be sure to check info less to see which ones your local version takes. And finally, while it hardly bears mentioning, why is it called less ? Because less is more. Sigh.

Because it's a text-based operating system that has historically been used for software development and computation, Unix did not traditionally provide the kind of full-featured, "what you see is what you get" text editing that exists on personal computers, although now such editors are available. In fact, WYSIWYG text editors are of limited utility for programmers because they often introduce invisible markup characters into documents.

It's worth learning to use the plain-text editors that are provided for Unix. They have a fairly steep learning curve, but they are the right tools for the job if you're writing programs or looking at plain-text data. If you download sequence data from a web server and open and work with it in a plain-text editor, the file you write out should be readable by a sequence-analysis program. If you opened the same file and worked with it in a WYSIWYG editor, then wrote it out in the file format used by that editor, it would be unreadable by other programs.

The vi editor is a standard feature of most Unix systems. It's a full-screen editor; it allows you to see as many lines of the file that you are editing as will fit into the terminal screen or window in which you run it. The cursor can be moved through the file using keyed instructions, but it can't be moved with the mouse. The bottom line on the screen is called the status line. Error messages from vi appear in the status line.

In Section 5.6, we discuss the use of regular expressions for searching and replacement as a feature of the plain-text editor vi. The ability to use vi with the regular-expression language makes vi a powerful tool for file manipulation.

A few nice features have been added to vi in vim (vi improved). It's worth asking your system administrator to install vim if it's not already on your system, if only for the multiple undo feature that it introduces. We can't cover all the features of vim here, but we will present a few commands that will get you up and running.^[‡]

vim has three modes ; in each, input from the keyboard is interpreted differently:

Here are some of the most useful vim command-mode commands:

Here are some of the most useful vim status line mode commands:

vim is a fairly flexible editor, and you can certainly learn to make it do any text-editing task that you need to do. However, there are other options for text editing on Unix systems. The best of these is probably the Emacs editor. Emacs is an editing program made available by the Free Software Foundation. It contains not only a text-editing facility with special modes for T_EX and LaT_EX documents, programs in various programming languages, and outlines, but also a file manager, mail and news readers, and access to the online documentation browser info. Whole books have been written on Emacs (see the Bibliography) so we won't go into it here except to recommend that, if you're working on a Unix system, learning to use Emacs is one of the better uses of your learning-curve time.

In addition to the text files we've discussed up to now, there are also binary files that can't be read as text. They are almost always the output of a program or the executable form of a program itself (as opposed to the source code). Binary files and program executables aren't human-readable because they are in machine language. Because of this language gap, we'll unflinchingly make the prediction that, 9 times out of 10, it isn't worth the effort needed to read binaries. You'll have more luck taking another route, like talking to the person whose program created the file in the first place. Unfortunately, many programs today, such as commercial hidden Markov model software or data mining programs that directly write their internal representation of data structures to disk, use binary files to store proprietary data structures. For that tenth time, then, we present some tips on how to extract information from binaries without going crazy.

Your first step should be to use either the less command described earlier or the strings command. If any portions of the file are in plain text, they will be readable in less. The strings command cuts out any readable text characters in the file and prints them to the screen. For example, if you have an undocumented binary file named badger and want to see if it contains any clues as to what it does, try typing:

% strings -n 3 badger | less

(The -n option tells strings how many readable text characters in a row constitute a string. The default setting for -n is four). Piping the output to less will let you page through it if it's longer than one screen. If the output looks like:

ATCGTACTGATCGTCGATCGTCGATCATGCA
CGTAGCAGTCGATCATCATCGTACTAGCTAG
ATGCCTGAGCTATACACACTAGTCACGATGC

you might guess that badger contains some kind of binary encoding of data including a nucleotide sequence or a (not good) multiple sequence alignment.

Sometimes, it may be necessary to do more than just identify a binary file. In these cases, the od program may provide a first step in understanding the file's contents. Before looking at od itself, let's take a quick detour through the ways in which binary information is represented in a moderately more human-friendly form.

Rather than using conventional decimal (base-10) notation, binary data is usually represented using a base that is a power of two: either octal (base-8) or hexidecimal (base-16) digits. Octal numbers are usually preceded by a 0. For example, the decimal number 25 corresponds to octal 031.^[§] Hexidecimal digits, on the other hand, are usually preceded by a 0x and use the letters A through F to represent the decimal numbers 10 through 15. The decimal number 25 is 0x19 in hexidecimal.

If you want to delve into the heart of the binary file and see what's going on, you can use the od command to perform an octal dump (or hex dump) and see if your binary file is readily interpretable. Typing:

% od -c badger | less

creates an octal dump of badger you can step through a page at a time. It should look something like this:

0000000       001           006   R   T   D   C   Y   G  
0000020     006   R   T   D   C   Y   G        a      
0000040 001   1       003   A   R   G       001      
0000060 002   C   A       002   B   Z 270   R   ? 200       @

o d 's primary options are:

Unless you're a serious programmer, you're not likely to have to read binaries. However, on the off chance that you do, we hope these standard tools will help you start to get your questions answered.

Say you have a program that spits out a lengthy datafile that has several different tables of information concatenated together. Leaving aside the question of why anyone would write a program that creates such difficult output, there are commands that allow you to work with such data, and you need to know them. head is one such command.

By default, head sends the top 10 lines of the specified file or files to standard output. Checking the head of a file this way is an easy way to see if there's something in the file without opening it using an editor or doing a full cat of the file.

With the -number flag, head becomes a tool for selecting a specified number of records from the top of a file. Combinations of head and tail commands can extract any set of lines from a file provided that you know their location in the file.

The tail command outputs the last 10 lines of a file by default, or the last num lines of the file if specified. With the -f option, tail gives constantly updated output of the last few lines in the file. This provides a way to monitor what is being added to a text output file as it's written by another program.

The split command allows you to break up an existing file into smaller files of a specified size. Each file produced is uniquely named with a suffix (aa, ab...az, ba, etc.). The options to split are:

If you have a file called big-meercat.txt and you want to split it into subfiles of length 100 lines using single-letter suffixes and writing the files out to subfiles named meercat.*, the command form of the split command is:

% split -l 100 -a 1 big-meercat.txt meercat. 

csplit also splits files into subfiles, but is somewhat more flexible than split, because it allows the use of criteria other than number of lines or bytes for splitting. Here are csplit 's options:

Split criteria are formed in two ways: either a regular expression is supplied as the criterion, possibly modified by an offset, or a number of lines can be specified.

A biological sequence database in FASTA format may contain many records of the form:

 >identifying header information 
PROTEINORNUCLEICACIDSEQUENCEDATA 

The csplit command can split such a database into individual sequence files using the command:

% csplit -f dbrecord. -n 6 fastadbfile /^>/

The file is split into numbered subfiles, each containing a single sequence.

The cut command outputs selected parts of each line of an input file. A line in a file is simply any stretch of characters that ends with a specific delimiter; a delimiter is a special nontext character an operating system or program recognizes. Lines in files are terminated with an EOL (end-of-line) character; files themselves are terminated with an EOF (end-of-file) character. These characters are usually invisible to you when you're working with the file, but they are important in how a file is treated by programs that read it.

For example, say you have a file called sequence_data that contains the following:

ATC  TAC
ATG  CCC
GAT  TCC

Here's how to use cut to output the first character of each line in the file:

% cut -c 1 sequence_data 
A
A
G

And here's how to output the first line of fields 1 and 2:

% cut -f 1-2 sequence_data 
AAT  TAC

Portions of each defined line can be selected by character number in the line with the -c option, or by field with the -f option. Fields are stretches of characters within a line that are defined by delimiters. The most obvious delimiter for use within the text of a file is simply the space character, but other characters can be used as well. Fields are different from columns, which are strictly defined by numbering each character in the input line.

The list argument specifies the range of each line, whether in characters or in fields, to be selected.

The list is in the form of single numbers or of two numbers separated by a - character. Multiple single columns or ranges can be selected by separating them with commas. Either the first or the last number can be omitted, indicating that the cut starts at the beginning of the line or that it ends at the end of the line. Characters and fields in each line are numbered starting at 1.

When the -f option is used, indicating that cut is to count fields rather than characters, a delimiter other than the default tab character can be specified with the -d option. The -s option causes cut to ignore lines that don't contain the specified delimiter. This option can be useful, for example, for ignoring header lines in a table.

The paste command allows you to combine fields from several files into one larger file. Unlike the join command, which does a database-style merging of two files, paste is a purely mechanical combination of files. Lines are combined based solely on their line number in each file: i.e., the first line of file1 is pasted next to the first line of file2, regardless of the content of the lines. Pasted data is separated by a tab character unless another delimiter is specified with the -d option. With the -s option and only one input filename, paste joins all the lines in the input file into one long line.

paste can prepare datafiles to be read by data-analysis applications. If you have a group of files in the same format and you have used other filter commands to remove corresponding information from each of them, you can prepare one input file that allows you to plot the corresponding information from each of the files without reading them independently. In a previous example, we used piped commands to extract a column from a table in a complicated output file:

% head -167 protein1.pka | tail -100 | cut -c30-39 > protein1.pka.data 

If you have eight similar output files for proteins 1-8, you can process them all in the same way and then paste the results that you're interested in comparing into one big datafile:

% paste protein*.pka.data > allproteins.pka.data 

Each individual file in this example might look something like this:

3.8 
12.0 
10.8 
4.4 
4.0 
6.3 
7.9 

Each number represents the computed pKa value of one amino acid in a protein. If you have several sets of results that can be meaningfully combined into a table, paste creates a simple tab-delimited table that looks like this:

3.8 3.2 3.6 
12.0 12.9 12.5 
10.8 10.9 11.0 
4.4 4.2 4.5 
4.0 3.9 4.2 
6.3 6.5 6.2 
7.9 7.5 8.0 

It's up to you, however, to understand how your data can be meaningfully combined into a table and to use the paste command correctly to get the result you want.

join merges two files based on the contents of a specified join field, where lines from the two files having the same value in the join field are assumed to correspond. Files are assumed to have a tabular format consisting of fields and a consistent field separator, and are assumed to be sorted in increasing order of their join fields.

Command-line options for join include:

join is quite useful for constructing data tables from multiple files, and a sequence of join operations can construct a complicated file. In a simple example, there are three files:

mustelidae.color:
badger black 
ermine white 
long-tailed tan 
otter brown 
stoat tan

mustelidae.prey:
ermine mouse 
badger mole 
stoat vole 
otter fish 
long-tailed mouse

mustelidae.habitat:
river otter 
snowfield ermine 
prairie long-tailed 
forest badger 
plains stoat 

First, combine mustelidae.color and mustelidae.prey. The field both have in common is the name of the animal, which is the first field in each file. mustelidae.prey isn't yet sorted. The form of the join command needed is:

% sort mustelidae.prey | join mustelidae.color - > outfile

which produces the following output:

badger black mole 
ermine white mouse 
long-tailed tan mouse 
otter brown fish 
stoat tan vole

Now combine the resulting file with mustelidae.habitat. If you want the resulting output to be in the form habitat animal prey color, use the command construct:

% sort -k2 mustelidae.habitat | join -1 2 -2 1 -o 1.1,2.1,2.3,2.2 - outfile

This operates on the standard input and the output file from the previous step to produce the output:

forest badger mole black 
snowfield ermine mouse white 
prairie long-tailed mouse tan 
river otter fish brown 
plains stoat vole tan

The sort command can sort a single file, sort a group of files and simultaneously merge them into a single file, or check a file for sortedness. This function has many applications in data processing. Each line in the file is treated as a single field by default, but keys can also be defined by the user on the command line.

The main options for sort are:

Options that determine how keys are interpreted can be used as global options, but they can also be used as flags on a particular key. The key interpretation options for sort are:

Let's say you have two lists and, while they look similar, you can't tell by eye if they are exactly the same list. This can happen if you get a list of gene names back from database searches performed using two subtly different queries and want to know if they are equivalent. In order to compare them rigorously (and save your eyes in the process), you can try the semicomplementary commands cmp and diff. In short, cmp tells you whether two files are identical, and diff prints any lines that are different.

cmp is fairly simple-minded. Typing:

% cmp enolase1.list enolase2.list

produces no output if the two files are identical. Otherwise, cmp returns a message that the files differ and includes the character and line at which the first difference occurs.

diff is most useful for comparing different versions of a file to find exactly where the files differ. Before looking at diff ' s rather obtuse output, it's worth a moment to see how to decrypt it. Without options, diff responds with a list of differences in the form of the changes required to make file2 from file1:

In practice, the output looks like this (where enolase1.txt and enolase2.txt are lists of names of putative enolases produced by two database searches performed at different times):

% diff enolase1.list enolase2.list 
1a2
> ENO_MESCR
5a7
> ENOA_MOUSE 

Here are two of the more immediately useful options diff uses:

The info pages on diff and its variants are especially helpful. If you use this utility extensively, we strongly recommend you give them a look.

wc is a simple and useful utility for counting things in text files. Given a text file, wc counts the number of lines, words, and bytes (characters) that it contains. The default setting for wc is to count all three entities, so that typing it at the command prompt returns a line that looks like:

% wc meercat1.txt 
   27   98   559 meercat1.txt 

This output tells you that there are 27 lines, 98 words, and 559 bytes in meercat1.txt. If you pass multiple files to wc, it returns counts both for individual files and for all of them combined. For example, if you run wc on the three meercat files:

% wc meercat1.txt meercat2.txt meercat3.txt

(or, to save time, wc meercat*.txt, being appropriately careful using the wildcard), the output looks like:

   41   130   905 meercat1.txt   
   50   124   869 meercat2.txt   
   10    19   156 meercat3.txt   
  101   273  1930 total 

These are the options for wc :

Unix tools can often be used in combination to collect information you need. For instance, say you have a list of 1,000 files that need to be processed, and the output files are all saved together in the same directory. Instead of trying to list the contents of that directory using ls, you can use ls -1 dirname | wc to find how many output files have been created so far.

grep allows you to search for patterns (in the form of regular expressions) in a file or a group of files. GNU grep (the standard on Linux) searches for one of three kinds of patterns, depending on which of the following functions is selected:

Note that the -E and -F options can be explicitly selected by calling egrep or fgrep on some systems. If no files are specified to be searched, grep searches the standard input for the pattern, allowing the output of another program to be redirected to grep if you are looking for a pattern in the output.

As a simple example, consider the following commands:

% grep -c '>' SP-caspases-A.fasta SP-caspases-B.fasta 
% grep '>' SP-caspases-A.fasta SP-caspases-B.fasta

These both search through a file of FASTA-formatted sequences (whose header lines, you will remember, begin with the > symbol). The first command returns the number of sequences in each file, while the second returns a list of the sequence headers. Be sure to enclose the > in quotes, though. Otherwise, as one of us once found out the hard way, the command is interpreted as a request for grep to search the standard input for no pattern and then redirect the resulting empty string to the files listed, overwriting whatever was already there.

grep takes dozens of options. Here are some of the more useful ones:

In protein structure files, protein sequence information is stored as a sequence of three-letter codes, rather than in the more compact single-letter code format. It's sometimes necessary to extract sequence information from protein structure files. In real life, you can do this with a simple Perl program and then go on to translate the sequence into single-letter code. But you can also extract the sequence with two simple Unix filter commands.

The first step is to find the SEQRES records in the PDB file. This is done using the grep command:

% grep SEQRES pdbfile > seqres

This gives you a file called seqres containing records that look like this:

SEQRES 1 357 GLU VAL LEU ILE THR GLY LEU ARG THR ARG ALA VAL ASN 2MNR 106 
SEQRES 2 357 VAL PRO LEU ALA TYR PRO VAL HIS THR ALA VAL GLY THR 2MNR 107 
SEQRES 3 357 VAL GLY THR ALA PRO LEU VAL LEU ILE ASP LEU ALA THR 2MNR 108

Not all the characters in each record belong to the amino-acid sequence. Next, you need to extract the sequences from the records. This can be done using the cut command:

% cut -c20-70 seqres > seqs

The output of this command, in the file seqs, looks like this:

GLU VAL LEU ILE THR GLY LEU ARG THR ARG ALA VAL ASN 
VAL PRO LEU ALA TYR PRO VAL HIS THR ALA VAL GLY THR 
VAL GLY THR ALA PRO LEU VAL LEU ILE ASP LEU ALA THR

If you don't want to create the intermediate file, you can pipe the commands together into one command line:

% grep SEQRES pdbfile | cut -c20-70 | paste -s > seqs. 

Addition of the paste -s command joins the individual lines in the file into one long line.

The easiest way to communicate with other computers is via the Web. Most distributions of Linux include web browser software—usually Netscape—which, if you select it from the list of installation options, is automatically installed for you. Setting up a web browser on a Linux system is the same as setting up a browser on other computers; you need to set the browser's preferences and tell it where the correct utilities are located to open different kinds of file attachments.

You may want to maintain a web page on your machine, and in order to do that, you need to install web server software. Again, most Linux distributions allow you to install the Apache web server software as one of your installation options. If you choose to install the Apache web server, you can publish a simple web site by placing the appropriate HTML files in the /home/httpd/html directory.

In the world of the Internet, computers recognize each other by their Internet Protocol (IP) addresses. Computers that are constantly connected to the Internet have permanently allocated IP addresses and hostnames, while computers that only connect to the Internet occasionally may have dynamically allocated IP addresses, or no IP address at all, depending on the protocol they use to connect.

IP addresses consist of four numbers separated by dots (e.g., 128.174.55.33). These are interpreted as directions to the host (a computer that communicates with other computers) by network software. Computers also have hostnames, such as gibas.biotech.vt.edu. Name servers are dedicated machines that maintain information about the relationships among IP addresses and hostnames.

The telnet command opens a shell on a remote Unix machine; the workstation on which the command is issued becomes a terminal for that machine. To telnet to another Unix machine, you must have a login on that machine. Once you're logged in to the remote host, the shell works just as if you were working directly on the remote machine.^[#]

A "login:" prompt should appear, followed by a "password:" prompt after your ID is entered.

The File Transfer Protocol (ftp) is a method for transferring files from one computer to another. You may be familiar with Fetch, Interarchy, or other PC-based FTP clients; Unix ftp is conceptually similar to these programs (and many of them have analogs that run under Linux, if you like their graphical user interfaces). When you use ftp to connect to another host, you will find yourself in an operating environment that is unique to ftp. Unix commands don't always work in the ftp environment, although the commands ls and cd have similar functions.

Again, a "login:" prompt appears, followed by a "password:" prompt. If you are accessing an anonymous FTP server (a common way to distribute software), the standard username is anonymous, and your email address is the password. Once in the FTP environment, the most important commands to know are:

Sometimes you need to run an X program on another computer and have it display on your terminal. This is relatively simple to do. First, you need to set your own terminal to allow remote displays from other hosts. This is done using the xhost command:

% xhost + 

A confirmation that access is allowed from other hosts is then printed to standard output.

Next, you need to change the display environment on the remote machine. This is done with the setenv command:

% setenv DISPLAY yourmachine.yoursubnet.wherever.edu:0 

Not all X applications running on a remote server can use your terminal for display, generally because the remote machine and your machine don't have the same graphics capabilities. For instance, programs running on a remote Silicon Graphics machine can't display on your local Linux workstation, because Silicon Graphics uses proprietary graphics libraries that aren't currently available to Linux users. However, even if both machines are compatible, bandwidth limitations can make running large X programs over the network extremely slow.

One of the biggest inconveniences for Linux users in a primarily Mac/PC environment is the sharing of files generated by PC productivity software with other users. While it's not our purpose to teach you to use these packages here, we can mention a few options that will help you handle communication with non-Unix users.

Fortunately, there are relatively low-cost software products available for Linux that make it possible to work with common file types, such as Microsoft Word and rich-text format (RTF) documents, PowerPoint presentations, and Excel spreadsheets. Sun's StarOffice (http://www.staroffice.com) and Applix's Applixware (http://www.vistasource.com) are two possibilities; at the time of this writing, StarOffice seemed to do the cleanest job of converting files generated by Microsoft Word and other commonly used programs. Adding one of these packages to your Linux system will add most of the basic PC functions (word processing, electronic presentations, etc.) that may be vital to your work.

Most kinds of graphics files are easily handled and converted on Linux systems. One powerful tool for manipulating graphics files is called the GIMP (Gnu Image Manipulation Program, http://www.gimp.org). The GIMP is commonly included in Linux distributions, so be sure to select it as part of your installation if you will be doing anything with graphics files. The GIMP is analogous to Adobe Photoshop program and shares most of the same functionality.

Linux users can read and write files on Microsoft-formatted floppy disks and Zip disks. A floppy or Zip disk is treated as an additional filesystem on your computer. The most basic way to access this filesystem is to mount it using the mount command. To do this, you need to know the device ID of the disk you are trying to mount and establish a mount point for the new filesystem.

Determining the device IDs of the various drives is usually straightforward. One way is to open the file /var/log/dmesg. This file contains the system information that is printed to standard output when the machine is booted. Scan through the file and find the drive information, which should look like this:

hdc: SAMSUNG SC-140B, ATAPI CDROM drive 
hdd: IOMEGA ZIP 250 ATAPI, ATAPI FLOPPY drive 
hdc: ATAPI 40X CD-ROM drive, 128KB Cache 
Floppy drive(s): fd0 is 1.44M 

This section of the file contains information about IDE devices. On this particular machine, the IDE devices include a CD-ROM drive, a Zip drive, and a floppy drive. The three-letter codes hdc, hdd, and fd0 are the device IDs.

The next section of the file contains information about SCSI devices. On this particular machine, the main hard disk is a SCSI drive, and its ID is sda. sda1, sda2, etc., are the individual IDs of the partitions on the hard drive:

Detected scsi disk sda at scsi0, channel 0, id 0, lun 0 SCSI device 
sda: hdwr sector= 512 bytes. Sectors= 35566499 [17366 MB] [17.4 GB] 
sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 >

Once you know the device IDs, mounting these new filesystems is simple. If you're the root user of your own machine, the command is:

mount -t [filesystem type] devicefile mount point
               

For example, to mount a PC-formatted floppy disk at /mnt/floppy, the command is:

% mount -t msdos /dev/fd0 /mnt/floppy

You can find a listing of allowed file types in the manpages for mount.

As a shortcut, you can modify your /etc/fstab file to contain the following lines:

/dev/fd0         /mnt/floppy       vfat  noauto,owner  0 0 
/dev/hdd4        /mnt/zip          vfat  noauto,owner  0 0 

On this system, the Zip drive is located at /dev/hdd. All PC-formatted Zip disks use partition number 4, and the device file for that partition is /dev/hdd4. The noauto flag means that these disks aren't mounted automatically at boot time. Once these lines are added to /etc/fstab, the devices can be mounted with the shortened command mount devicefilename.

Once the Zip or floppy is mounted as a partition, the files on that disk can be treated like any other file on the system.

Getting some of these devices working isn't as straightforward as we'd like it to be. For further help, you can search the Web for the Linux how-to pages for the particular device you're using.

If you install the utility package mtools and its graphical frontend mfm, you can run mfm and move files to Zip or floppy disks, using a graphical interface similar to that on a PC. However, if you use this method to access devices, you can't run Unix commands on the files stored on your media until you move them onto the local hard disk.

By default, processes to access media may be run only by the root user. It's possible to configure your system so that other users can write to floppy and Zip drives. However, this creates a security hole in your system. You have to decide for yourself whether the benefits of easy disk access outweigh any potential risks.

A Unix system carries out many different operations at the same time. Each of these operations, or processes, has a unique process ID that allows the user or the administrator to interact with that process directly.

There are a minimum number of processes that run on a system regardless of whether you actively initiate them. Each shell program, whether idle or active, has a process ID attached to it. Several system (or root) processes, sometimes known as daemons , are constantly active on the system. These processes often lie in wait for you to initiate some activity they handle: for instance, printing files, sending email, or initiating a telnet session.

Above and beyond this minimal system activity level are any processes you initiate by typing a command or running a program. The Unix kernel manages these processes, allocating whatever resources are available to the processes according to their needs.

Each process uses a percentage of the processing capacity of the system's CPU or CPUs. It also uses a percentage of the system's memory. When the processes running on a machine require more than 100% of the CPU's capacity to execute, each individual process will execute more slowly. While Unix does an extremely good job of juggling hundreds of processes that run at the same time without having the machine roll over and die, eventually you will see a situation where the load on the machine increases to the point that the machine becomes useless. The operating system uses many techniques to prevent this, such as limiting the absolute number of processes that can be started and swapping idle jobs out of memory. Even on a single processor system, it's possible to have multiple processes running concurrently as long as there is enough space for both jobs to remain in memory. At the point at which the CPU has to constantly wait for data to get loaded from the swap space on the hard drive, you will see a great drop in efficiency. This can be monitored using the top command, which is described in Section 5.9.1.3. Many machines are more limited by lack of memory than they are by a slow CPU, and it's often now more cost-effective to put money into additional RAM than to buy the latest, greatest, and fastest CPU.

2:55pm up 37 days, 4:50, 4 users, load average: 1.00, 1.02, 2.00 
USER    TTY    FROM        LOGIN@    IDLE   JCPU   PCPU WHAT 
jambeck tty1              22Jan99  37days  3:55m  0.06s startx 
jambeck ttyp0  :0.0       Wed 5pm   1:34m  0.22s  0.22s -csh 
jambeck ttyp3  :0.0       21Feb99    3:47  9.05s  8.51s telnet weasel 
god     ttyp2  around      2:52pm   0.00s  0.55s  0.09s create world 

  PID TTY    TIME CMD
36758 ttyq10 0:02 tcsh    
43472 ttyq10 0:00 ps    
42948 ttyq10 4:24 xemacs-20    
42967 ttyq10 1:21 fermats-last-theorem-solver 

4:34pm up 37 days, 6:29, 4 users, load average: 0.25, 0.07, 0.02 
42 processes: 39 sleeping, 3 running, 0 zombie, 0 stopped 
CPU states: 42.9% user, 6.4% system, 0.0% nice, 51.0% idle 
Mem:  39092K av, 38332K used,   760K free, 13568K shrd,  212K buff 
Swap: 33228K av, 20236K used, 12992K free               8008K cached

  PID USER    PRI NI  SIZE  RSS SHARE STAT LIB %CPU %MEM   TIME COMMAND
  516 jambeck  15  0  4820 3884  1544 R      0 30.4  9.9   4:23 emacs-fgyell
  415 root      9  0 10256 9340   888 R      0 15.5 23.8 161:41 /usr/X11R6/b 
10756 cgibas    5  0   716  716   556 R      0  2.3  1.8   0:01 top-ci

The cron daemon, crond, is a standard Unix process that performs recurring jobs for the system and individual users. System activities such as cleanup of the /tmp directory and system backups are typically functions controlled by the cron daemon. Normal users can also submit their own jobs to cron, assuming they have permission to run cron jobs. Details about cron permissions are found in the crontab manpage. Since the at and batch commands, which are discussed later, are also controlled by cron, most systems are configured to allow users to use cron by default.

# Format of lines: 
#min  hour  daymo  month  daywk  cmd 
   50     2      *      *      *  /home/jambeck/runme 

01 4 * * * find /data/seq/ -name "*.seq" -type f -mtime -1 -exec 
/usr/bin/csh /usr/local/bin/blastall -p blastx -d nr -i '{' ";'

> at 3:07am 
Mail -s "big breakthrough" boss@wherever < /home/jambeck/news 
<Ctrl-d> 

As fast as available disk space on a system expands, users seem to be able to expand their files to fill it. Software takes up more space; output files become larger and more complex; more layers of analysis can be created. Since the infinitely large data-storage medium has yet to be invented, you can still run up against disk-space limitations. So, you need to be able to monitor how much space you are using and, as we'll discuss in Section 5.9.4, how to make data archives and store them on appropriate media.

Filesystem            Type    blocks       use    avail %use Mounted on 
/dev/root              xfs  17506496  14113592  3392904   81 / 
/dev/xlv/xlv_raid_home xfs  62783488  39506328 23277160   63 /scratch-res1 
/dev/dsk/dks0d5s0      xfs  17259688  15000528  2259160   87 /mnt/root-6.4 
/dev/dsk/dks12d1s7     xfs  17711872  11773568  5938304   67 /ip410 
/mnt/local/jmd/balder: NFS server balder not responding 
zeus:/hamr             nfs   2205816    703280  1502536   32 /nfs/zeus/hamr 
zeus:/hamrscr          nfs   4058200   2153480  1904720   54 /nfs/zeus/hamrscr 
zeus:/lcascr1          nfs 142241472 103956480 38284992   74 /nfs/zeus/lcascr1 

So, after months of your time, hundreds of megabytes of files, and several layers of subdirectories, the otter project is finally complete. Time to move on to the next project with a clean slate. But as refreshing as it may sound, you can't just type:

% rm -rf otter/

Other people may need to look back at your findings or use them as a starting point for their own research. At the other extreme, you can't leave your files lying around or laboriously copy them a few at a time to another location. Not every file needs to be accessible at all times; some files are replaced, while others are more conveniently stored elsewhere. This section covers the tools provided by Unix for archiving your data so you don't have to worry about it on a day-to-day basis but can find things later when you need them.

% tar cvf otter/

% tar cvf /dev/nftape otter/

ftp command "| command" filename

put "|tar cvBf - *" filename

get filename.tar "|tar xvf - dirname"

get filename.tar "|tar t - *"

% compress -v stoat.txt otter.tar