VMware currently comes in two basic flavors, VMware Server and VMware ESX Server. VMware Server is a free version of VMware that offers basic virtual server capabilities and runs inside Linux or Windows. Each virtual machine is represented as a series of files in a subdirectory of a standard filesystem that you specify; the subdirectory carries the name of the virtual machine. For example, if you’ve chosen to store your virtual machines in /vmachines, and you have a virtual host called Windows 2000, its files will be located in /vmachines/Windows 2000.

While VMware Server runs inside standard Linux or Windows, VMware ESX Server uses a custom Linux kernel and a custom filesystem, VMFS, to store virtual machine files. You can also store virtual machine files on raw disk partitions. Neither the raw disk partitions or files in a VMFS filesystem can be accessed by all backup commands, so you probably need to back them up in a special way.

When backing up a VMware machine, you have to back up the operating system of the VMware server (known as the service console on ESX systems) and the VMware application itself. You also need to back up each virtual machine’s files. However, you cannot simply back up the virtual machine files or raw disks with a standard backup program. The virtual disks are constantly open and changing while the virtual machines are running, and you will not receive a consistent backup of them. Even open file agents won’t necessarily work properly if the virtual machines are gigabytes in size. You therefore have three options for backing up virtual machines running inside VMware:

C:> vmware-cmd path-to-config/config.vmx suspend

C:> vmware-cmd path-to-config/config.vmx start

One of the really nice things about using VMware (or other virtual server solutions) is that you don’t have worry about bare-metal recovery of the virtual servers. As long as you can get them to not change during a backup, all you have to do is back up their files.

However, you can use the bare-metal recovery procedure documented in Chapter 11 to migrate physical machines into virtual machines. We just did this and turned 25 very old physical servers into one very nice VMware server. The following is the story of that migration.

I get asked all kinds of questions about backup products and how they behave on different operating systems and applications, and I use a lab to answer these questions. In addition to the usual backup hardware (SAN, tape libraries, VTLs), it consists of some Sun, IBM, and HP hardware running Solaris, AIX, and HP-UX. Up until just recently, we also had about 25 Intel machines running various versions of Linux and Windows and their associated applications (Exchange, SQL Server, Oracle, etc.). I never had enough machines, and I never had the right machines connected to the right hardware. We were constantly swapping SCSI and Fibre Channel cards, as well as installing and uninstalling applications. I could have used 100 machines, but that would obviously be prohibitive in many ways. (The cooling alone would be crazy.)

So we recently decided to see if we could get rid of all these servers with VMware. We bought a white box with a 3.5 GHz Dual Core AMD processor, 4 GB DDR2 RAM and 1.75 TB of internal SATA disks. I installed into that server two existing Fibre Channel cards and two SCSI cards. I then followed the alt-boot recovery method to move all of those physical servers into virtual servers, virtually upgrading each of their CPUs, storage, and memory in the process. Here are the steps I followed for each server:

With 4 GB of RAM and a 3.5 GHz dual-core processor, I can run about eight virtual servers at a time without swapping. I typically only need a few at a time, and what’s important is that I have Exchange 2000, SQL Server X, or XYZ x.x running; they don’t need to run that fast. (That’s how I was able to get by with those old servers for so long.) Each virtual server can have access to either one of the Fibre Channel cards or SCSI cards, which gives them access to every physical and virtual tape drive in the lab. They will also have more CPU, disk, and RAM than they ever had in their old machine. (I can even temporarily give any of them the entire 3.5 GHz processor and almost all of the 4 GB of RAM if I need to, and I don’t have to swap chips or open up any CPU thermal compound to do it!)

I also get to have hundreds of virtual servers and not have any logistical or cooling issues, since each server only represents 20–50 GB of space on the hard drive. I can have a Windows 2000 server running no special apps, one running Exchange 5, one running SQL Server 7, a server running Windows 2003 with no special apps, one with Exchange 2000, and one running Windows Vista. I could have servers running every distribution of Linux, FreeBSD, and Solaris x86—and all of the applications those servers support. I think you get the point. I’ve got enough space for about 300 virtual server combinations like that. It boggles the mind.

Files that are changing during the backup do not always make it to the backup correctly. This is especially true if the filename or inode changes during the backup. The extent to which your backup is affected by this problem depends on what type of utility you’re using and how volatile the filesystem is.

For example, suppose that the utility performs the equivalent of a find command at the beginning of the backup. This utility then begins backing up those files based on the list that it created at the beginning of the backup. If a filename changes during a backup, the backup utility receives an error when it attempts to back up the old filename. The file, with its new name, is simply overlooked.

Another scenario occurs when the filename does not change, but its contents do. The backup utility begins backing up the file, and the file changes while being backed up. This is probably most common with a large database file. The backup of this file would be essentially worthless because different parts of it were created at different times. (This is actually what happens when backing up Oracle database files in hot-backup mode. Without Oracle’s ability to rebuild the file, the backup of these files would be worthless.)

This is similar to the corrupted files problem but on a filesystem level. Backing up a particular filesystem may take several hours. This means that different files within the backup are backed up at different times. If these files are unrelated, this creates no problem. However, suppose that two different files are related in such a way that if one is changed, the other is changed. An application needs these two files to be related to each other. This means that if you restore one, you must restore the other. It also means that if you restore one file to 11:00 p.m. yesterday, you should restore the other file to 11:00 p.m. yesterday. (This scenario is most commonly found in databases but can be found in other applications that use multiple, interrelated files.)

Suppose that last night’s backup began at 10:00 p.m. Because of the name or inode order of the files, one is backed up at 10:15 p.m. and the other at 11:05 p.m. However, the two files were changed together at 11:00 p.m., between their separate backup times. Under this scenario, you would be unable to restore the two files to the way they looked at any single point in time. You could restore the first file to how it looked at 10:15, and the second file to how it looked at 11:05. However, they need to be restored together. If you think of files within a filesystem as records within a database, this would be referred to as a referential integrity problem.

If the filesystem changes significantly while it is being backed up, some utilities may actually create a backup that they cannot read. This is obviously one of the most dangerous things that can happen to a backup, and it would happen only under the most extreme circumstances.

In 1991, Elizabeth Zwicky did a paper for the LISA^[1] conference called “Torture-testing Backup and Archive Programs: Things You Ought to Know But Probably Would Rather Not.” Although this paper and its information are somewhat dated now, people still refer to it when talking about this subject. Elizabeth graciously consented to allow us to include some excerpts in this book:

Many people use tar, cpio, or some variant to back up their filesystems. There are a certain number of problems with these programs documented in the manual pages, and there are others that people hear of on the street, or find out the hard way. Rumors abound as to what does and does not work, and what programs are best. I have gotten fed up, and set out to find Truth with only Perl (and a number of helpers with different machines) to help me.

As everyone expects, there are many more problems than are discussed in the manual pages. The rest of the results are startling. For instance, on Suns running SunOS 4.1, the manual pages for both tar and cpio claim bugs that the programs don’t actually have any more. Other “known’’ bugs in these programs are also mysteriously missing. On the other hand, new and exciting bugs—bugs with symptoms like confusions between file contents and their names—appear in interesting places.

Elizabeth performed two different types of tests. The first type were static tests that tried to see which types of programs could handle strangely named files, files with extra-long names, named pipes, and so on. Since at this point we are talking only about volatile filesystems, I will not include her static tests here. Her active tests included:

Elizabeth explains how the degree to which a utility would be affected by these problems depends on how that utility works:

Programs that do not go through the filesystem, like dump , write out the directory structure of a filesystem and the contents of files separately. A file that becomes a directory or a directory that becomes a file will create nasty problems, since the content of the inode is not what it is supposed to be. Restoring the backup will create a file with the original type and the new contents.

Similarly, if the directory information is written out and then the contents of the files, a file that is deleted during the run will still appear on the volume, with indeterminate contents, depending on whether or not the blocks were also reused during the run.

All of the above cases are particular problems for dump and its relatives; programs that go through the filesystem are less sensitive to them. On the other hand, files that shrink or grow while a backup is running are more severe problems for tar , and other filesystem based programs. dump will write the blocks it intends to, regardless of what happens to the file. If the block has been shortened by a block or more, this will add garbage to the end of it. If it has lengthened, it will truncate it. These are annoying but nonfatal occurrences. Programs that go through the filesystem write a file header, which includes the length, and then the data. Unless the programmer has thought to compare the original length with the amount of data written, these may disagree. Reading the resulting archive, particularly attempting to read individual files, may have unfortunate results.

Theoretically, programs in this situation will either truncate or pad the data to the correct length. Many of them will notify you that the length has changed, as well. Unfortunately, many programs do not actually do truncation or padding; some programs even provide the notification anyway. (The “cpio out of phase: get help!” message springs to mind.) In many cases, the side reading the archive will compensate, making this hard to catch. SunOS 4.1 tar, for instance, will warn you that a file has changed size, and will read an archive with a changed size in it without complaints. Only the fact that the test program, which runs until the archiver exits, got ahead of tar, which was reading until the file ended, demonstrated the problem. (Eventually the disk filled up, breaking the deadlock.)

What if you could back up a very large filesystem in such a way that its volatility was irrelevant? A recovery of that filesystem would restore all files to the way they looked when the entire backup began, right? A technology called the snapshot allows you to do just that. A snapshot provides a static view of an active filesystem. If your backup utility is viewing a filesystem through its snapshot, it could take all night long to back up that filesystem; yet it would be able to restore that filesystem to exactly the way it looked when the entire backup began.

The dump utility is very filesystem-specific, so there may be slight variations in how it works on various Unix platforms. For the most part, however, the following description should cover how it works because most versions of dump are generally derived from the same code base. Let’s first look at the output from a real dump. We’re going to look at an incremental backup because it has more interesting messages than a level-0 backup:

# /usr/sbin/ufsdump 9bdsfnu 64 80000 150000 /dev/null /
  DUMP: Writing 32 Kilobyte records
  DUMP: Date of this level 9 dump: Mon Feb 15 22:41:57 2006
  DUMP: Date of last level 0 dump: Sat Aug 15 23:18:45 2005
  DUMP: Dumping /dev/rdsk/c0t3d0s0 (sun:/) to /dev/null.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Estimated 56728 blocks (27.70MB) on 0.00 tapes.
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 56638 blocks (27.66MB) on 1 volume at 719 KB/sec
  DUMP: DUMP IS DONE
  DUMP: Level 9 dump on Mon Feb 15 22:41:57 2006

In this example, ufsdump makes four main passes to back up a filesystem. We also see that Pass II was performed three times. What is dump doing during each of these passes?

Let’s review the issues raised earlier in this section.

As described earlier, using dump to back up a mounted filesystem can dump files that are found to be corrupt when restored. The likelihood of that occurring rises as the activity of the filesystem increases. There are also situations that can occur where data is backed up safely, but the information in the dump is inconsistent. For these inconsistencies to occur, certain events have to occur at the right time during the dump. And it is possible that the wrong file is dumped during the backup; if that file is restored, the administrator will wonder how that happened!

The potential for data corruption to occur is pretty low but still a possibility. For most people, dumping live filesystems that are fairly idle produces a good backup. Generally, you will have similar success or failure performing a backup with dump as you will with tar or cpio.

If you’ve just been handed a volume and need to read it right now, ignore this paragraph. If you work in a heterogeneous environment and might be reading volumes on different types of platforms, read it carefully now. Reading a volume on a platform other than that on which it was created is always difficult.

In fact, except for circumstances like a bad backup drive or data corruption, the only sure way to read a volume easily every time is to read it on the machine that made it. Do not assume that you will be able to read a volume on another system because the volume is the same size, because the operating system is the same, or even if the utility goes by the same name. In fact, don’t assume anything.

If it is likely that you are eventually going to have to read a volume on another type of system or another type of drive, see if it works before you actually need to do it. Also, if you can keep one or two of the old systems and drives around, you will have something to use if the new system doesn’t work. (I know of companies that have 10- or 15-year-old computers sitting around for just this purpose.) If you test things up front, you might find out that you need to use a special option to make a backup that can be read on other platforms. You may find that it doesn’t work at all. Of course, finding that out now is a lot better than finding it out two years from now when you really, really, really need that volume!

Many media types look similar but really are not. DLT, LTO, AIT, and other drives all have different generations of media that work in different generations of drives. If the volume is a tape, and its drive has a media recognition system (MRS), it may even spit the tape back out if it is the wrong type. Sometimes MRS is not enabled or not present, so you assume that the tape should work because it fits in the drive. Certain types of media are made to work in certain types of drives, and if you’ve got the wrong media type for the drive that you are using, the drive will not be able to read it. Sometimes this is not initially obvious because the drive reports media errors.

Problems involving incompatible media types sometimes can be corrected by using the newest drive that you have available. That is because many newer drives are able to read older tapes created with previous generations of drives. However, this is not always the case and can cause problems.

If the drive types and media types are the same but one drive cannot read the other drive’s tapes, then the drive could be defective or just dirty. Try a cleaning tape, if one is available. If that does not work, the drive could be defective. It also is possible that the drive that wrote the tape was defective. A drive with misaligned heads, for example, may write a backup image that can’t be read by a good drive. For this reason, when you are making a backup volume that is going to be stored for a long time, you should verify right away that it can be read in another drive.

Although less common, there are also tape cleaning machines. The machines look like tape drives. They load the tape and run the entire tape through a clean and vacuum process. Sometimes when a tape is unreadable in any drive, cleaning the tape like this can allow the tape to be read. It would be handy to have one of these machines to prepare for such a scenario.

This is related to the media-types problem. Not all drives that look alike are alike. For example, not all tapes are labeled with the type of drive they should go into. Not all drives that use hardware compression are labeled as such, either. The only way to know for sure is to check the model numbers of the two different drives. If they are different manufacturers, you may have to consult their web pages or even call them to make sure that the two drive types are compatible.

Usually, drives of the same type use the same kind of compression. However, some value-added resellers (VARs) sell drives that have been enhanced with a proprietary compression algorithm. They can get more compression with their algorithm, thus allowing the drive to write faster and store more. If all of your drives are from the same manufacturer, this may not be a problem—as long as the vendor stays in business! But if all your drives aren’t from the same manufacturer, you should consider using an alternate compression setting if they have one, such as IDRC or DCLZ. Again, this goes back to proper planning.

Differences exist among machines of different architectures that may make moving volumes between them impossible. These differences include whether the machine is big-endian, little-endian, ones complement, or twos complement. For example, Intel-based machines are little-endian, and RISC-based machines are big-endian. Moving volumes between these two types of platforms may be impossible.

Most big Unix machines are big-endian, but Intel x86 machines and older Digital machines are little-endian (see Table 23-1). That means that if you are trying to read a backup that was written on an NCR 3b2 (a big-endian machine), and you are using a backup drive on an NCR Intel SVr4 (little-endian) box, you may have a problem. There is also the issue of ones-complement and twos-complement machines, which are also different architectures. It is beyond the scope of this book to explain what is meant by big-endian, little-endian, ones complement, and twos complement. The purpose of this section is merely to point out that such differences exist and that if you have a volume written on one platform and are trying to read it on another, you may be running into this problem. Usually, the only way to solve it is to read the volume on its original platform.

Most backup formats use an “endian-independent” format, which means that their header and data can be read on any machine that supports that format. Usually, tar and cpio can do this, especially if you use the GNU versions. I have read GNU tar volumes on an Intel Unix or Linux (i.e., little-endian) box that were written on HPs and Suns (i.e., big-endian machines). For example, it is quite common to ftp tar files from a Unix machine to a Windows machine, then use WinZip to read them. Again, your mileage may vary, and it helps if you test it out first.

Some people talk about reading a volume with dd and using its conv=swab feature to swap the byte order of a volume. This may make the header readable but may make the data itself worthless. This is because of different byte sizes (8 bits versus 16 bits) and other things that are beyond the scope of this book. Again, the only way to make sure that this is not preventing you from reading a volume is to make sure that you are reading the volume on the same architecture on which it was written.

Tape volumes are written in different block sizes, and you often need to know the block size of a tape before you can read it. This section describes how block sizes work, as well as how to determine your block size.

When a program reads or writes data to or from a device or memory, it is referred to as an I/O operation . How much data is transferred during that I/O operation is referred to as a block. Since the actual creation of each block consumes resources, a larger block usually results in faster I/O operations (i.e., faster backups). When an I/O operation writes data to a disk, the block size that was used for that operation does not affect how the data is physically recorded on the disk; it affects only the performance of the operation. However, when an I/O operation writes to a tape drive, each block of data becomes a tape block, and each tape block is separated by an interrecord gap. This relationship is illustrated in Figure 23-1.

Figure 23-1. Tape blocks and interrecord gaps

All I/O operations that attempt to read from this tape must understand its block size, or they will be unsuccessful. If you use a different block size, three potential scenarios can occur:

Block size is a multiple of the original block size

For example, a tape was recorded with a block size of 1,024, and you are reading it with a block size of 2,048. This scenario is actually quite common and works just fine. Depending on a number of factors, the resulting read of the tape may be faster or slower than it would have been if it used the original block size. (Using a block size that is too large can actually slow down I/O operations.)

Block size is larger than the original block size (but not a multiple)

For example, a tape was recorded with a block size of 1,024, and you are reading it with a block size of 1,500. What happens here depends on your application, but most applications will return an I/O error. The read operation attempts to read a whole block of data, and when it reaches the end of the block that you told it to read, it does not find an interrecord gap. Most applications will complain and exit.

Block size is smaller than the original block size

For example, a tape was recorded with a block size of 1,024, and you are reading it with a block size of 512. This will almost always result in an I/O error. Again, the application attempts to read a block of 512 bytes, then looks for the interrecord gap. If it doesn’t see it, it complains and exits.

Interrecord gaps actually take up space on the tape. If you use a block size that is too small, you will fill up a lot of your tape with these interrecord gaps, and the tape actually will hold less data.

Each tape drive on each server has an optimal block size that allows it to stream best. Your job is to find which block size gives you the best performance. A block size that is too small decreases performance; a block size that is too large may decrease performance as well because the system may be paging or swapping to create that large block size. Some operating systems and platforms also limit the maximum block size.

Use the trick described in Chapter 3 in the section “Using dd to Determine the Block Size of a Tape” to determine your block size. If you’re reading a tar or dump backup, you’ll need to determine the blocking factor. If the backup utility is tar, the blocking factor usually is multiplied by 512. dump’s blocking factor usually is multiplied by 1,024. Read the manpage for the command that you are using and determine the multiplier that it uses. Then, divide the block size by that multiplier. You now have your blocking factor.

For example, you read the tape with dd, and it says the block size is 32,768. The manpage for dump tells you that the blocking factor is multiplied times 1,024. If you divide 32,768 by 1,024, you will get a blocking factor of 32. You then can use this blocking factor with restore to read the tape.

Some operating systems, such as AIX, allow you to hardcode the block size of a tape device. This means that no matter what block size you set with a backup utility, the device will always write using the hardcoded block size. During normal operations, most people set the block size to 0, allowing the device to write in any block size that you specify with your backup utility. (This is also known as variable block size.) However, during certain operations, AIX automatically sets the block size to 512. This normally happens when performing a mksysb or sysback backup, and the reason this happens is that a block size of 512 makes the mksysb/sysback tape look like a disk. That way, the system can boot off the tape because it effectively looks like the root disk. Most mksysb/sysback scripts set the block size back to when they are done, but not all do so. You should check to make sure that your scripts do, to prevent you from unintentionally writing other tapes using this block size.

Why can’t you read, on other systems, tapes that were written on AIX (with a block size of 512)? The reason is that AIX doesn’t actually use a block size of 512. What AIX really does is write a block of 512 bytes and then pad it with 512 bytes of nulls. That means that they’re really writing a block size of 1024, and half of each block is being thrown away! Only the AIX tape drives understand this, which means that a tape written with a block size of 512 can be read only on another AIX system.

However, if you set the device’s hardcoded block size to 0, you should have no problem on other systems—assuming the backup format is compatible. Setting it to 0 makes it work like every other tape drive. The block size you set with the backup utility is the block size the tape drive writes in. (If you want to check your AIX tape drive’s block size now, start up smit and choose Devices, then Tape Drives, then Change Characteristics, and make sure that the block size of all your tape drives is set to 0!)

You can even set the block size of a device to 1,024 without causing a compatibility problem. Doing so will force the device to write using a block size of 1,024, regardless of what block size you specify with your backup utility. However, this is a “normal” block, unlike the unique type of block created by the 512-byte block size. Assuming that the backup format is compatible, you should be able to read such a tape on another platform. (I know of no reason why you would want to set the block size to 1,024, though.)

To set the block size of a device back to 0, run the following command:

# chdev -l device_name -a block_size=0

Obviously, when you are handed a foreign volume, you have no idea what backup utility was used to make that volume. If this happens, start by finding out the block size; it will come in handy when trying to read an unknown format. Then, use that block size to try and read the volume using the various backup formats, such as tar, cpio, dump, and pax. I would try them in that order; foreign volumes are most likely going to be in tar format because it is the most interchangeable format.

One trick to finding the type of backup format is to take a block of data off of the volume and run the file command on it. This often will come back and say cpio or tar. If that happens, great! For example, if you used the block size-guessing command shown previously, you would have a file called /tmp/sizefile that you could use to determine the block size of the tape. If you haven’t made this file, do so now, then enter this command:

# file /tmp/sizefile

If it just says “data,” you’re out of luck. But you just might get lucky, especially if you download from the Internet a robust magic file:

# file -f /etc/robust.magic /tmp/sizefile

In this case, file helps reveal the format for commands and utilities not native to the immediate platform.

Sometimes, two commands sound the same but really aren’t. This can be as simple as incompatible versions of cpio, or at the worst, completely incompatible versions of dump. Format inconsistencies between tar and cpio usually can be overcome by the GNU versions because they automatically detect what format they are reading. However, if you are using an incompatible version of dump (such as xfsdump from IRIX), you are out of luck! You will need a system of that type to read the volume. Again, your mileage may vary. Make sure you test it up front.

One of the most common questions I see on Usenet is, “I accidentally typed tar cvf when I meant to type tar xvf. Is there any way to read what’s left on this volume?” The quick answer is no. Why is that?

Each time a backup is written to a tape, an end-of-media (EOM) mark is made at the end of the backup. This mark tells the tape drive software, “There is no more data after this mark—no need to go any further.” No matter what utility you try, it will always stop at the EOM mark because it thinks this is the last backup on the tape. Of course, the tape could just be damaged or corrupted. One of the tricks I’ve seen used in this scenario is to use cat to read the corrupted tape:

# cat device/tmp/somefile

This just blindly reads in the data into /tmp/somefile, so you can read it with tar, cpio, or dump.

One of the fun things about being a backup specialist is that everyone tells you their favorite backup and recovery horror stories. One day a friend told me that he was having a really hard time reading a particularly flaky tape. The system would read just so far into the tape and then quit with an I/O error. However, if he tried reading that same section of tape again, it would work! He really needed the data on this particular tape, so he refused to give up. He wrote a shell script that would read the tape until it got an error. Then it would rewind the tape, fast-forward (fsr) to where he got the error, and try again. This script ran for two or three days before he finally got what he needed. I had never heard of such dedication. I told my friend Jim Donnellan that he had to let me put the shell script in the book. The shell script in Example 23-1 was called read-tape.sh and actually did the job. Maybe this script will come in handy for someone else.

# !/bin/sh

DEVICE=/dev/rmt/0cbn
# Set this to a non-rewinding tape device

touch rawfile
# The rawfile might already be there, but just in case
while true ; do

 size=Qls -l rawfile | awk '{print $5}'Q # Speaks for itself

 blocks=Qexpr "$size" / 512Q

 full=Qdf -k . | grep <host> | awk '{print $6}'Q 
 # Unfortunately, this only gets checked once per glitch. Maybe a fork?

 echo $size
 # Just so I know how it's going

 echo $blocks

 echo $full

 if [ $full -gt 90 ] ; then
      echo "filesystem is filling up"
      exit 1
 fi

 mt -f $DEVICE rewind
 # Let's not take chances. Start at the beginning.

 sleep 60
 # The drive hates this tape as it is. Give it a rest.

 mt -f $DEVICE fsr $blocks

 # However big rawfile is already, we can skip that on the tape

 dd if=$DEVICE bs=512 >> rawfile # Let's get as much as we can

 if [ $? -eq 0 ] ; then
  # If dd got clipped by a tape error, there's still work to do,
  echo "dd exited cleanly"

  # if not, it must have gotten to the end of the file this time
  # without a hitch. We're done.
  exit 0
 fi

done

If you’ve got tips on how to read corrupted or damaged volumes, I want to hear them. If I use them in later editions of the book, I will credit your work! (I also will put any new ones I receive on the web site for everyone to use immediately.)

This one is more of a gotcha than anything else. Always remember that when a backup is sent to tape, it could have more than one partition on that tape. If you are reading an unknown tape, you might try issuing the following commands:

# mt -t device rewind
#mt -t device fsf 1

Then, try again to read this backup. If it fails with I/O error, there are no more backups. (That’s the EOM marker again.) If it doesn’t fail, try the same commands that you tried in the beginning of the tape to read it. Do not assume that it is the same format as the first partition on the tape. Also understand that every time you issue a command to try and read the tape, you need to rewind it and fast-forward it again using the two preceding commands.

Then perhaps failure is your style! That doesn’t mean that you have to stop trying to read that volume. Remember that the early bird gets the worm, but the second mouse gets the cheese. The next time you’re stuck with a volume you can’t read, remember my friend Jim and his flaky tape.