

[[backup.server]]
Backup Server
~~~~~~~~~~~~~

You can set up a backup server for this process. Scripts on the backup
server interact with the target machines (including itself) via SSH.
They assume that your backup server user (root here, for simplicity) can
log in with no password to the targets. This is necessary for unattended
backups.

First, create a suitable directory to keep all the backups in. We'll
call it `backs`. In backs, create a directory for each target computer.
The first field in the directory should be the host name. Subsequent
fields can be other useful information. For example, to preserve the
last backup of a target before an installation of a new version of the
distribution, I use an abbreviation for the distribution, e.g. ``fc5''.
Fields are separated with periods (``.''). So, for example, `tester.f7`.
The host name is required because the scripts use that to determine
which host to back up.

Copy the scripts `get` and `restore` into each target's directory. Then
customize them for each host as needed.

Also create in `backs` a directory called `scripts` and put in it the
script `get.target`. This is a library for the backup and restore
scripts. It performs actions common to all backups and restorations.

[[CreatingtheStage1BackUp]]
Creating the Stage 1 Back Up
----------------------------

Having made your production backups, you need to preserve your partition
information so that you can rebuild your partitions.

The script link:#make.fdisk[`make.fdisk`] scans a hard drive for
partition information, and saves it in four files. The first is an
executable script, called link:#make.dev.hda[`make.dev.x`] (where ``x''
is the name of the device file, e.g. hda). Second is
link:#mount.dev.hda[`mount.dev.x`], which creates mount points and
mounts the newly created partitions on them. The next,
link:#dev.hda[`dev.x`], is the commands necessary for `fdisk` to build
the partitions. Last is an input file for `sfdisk` to create partions.
(`sfdisk` is preferable and used if found.) You specify which hard drive
you want to build scripts for (and thus the file names) by naming the
associated device file as the argument to
link:#make.fdisk[`make.fdisk`]. For example, on a typical IDE system,

....
bash# make.fdisk /dev/hda
....

spits out the scripts link:#make.dev.hda[`make.dev.hda`],
link:#mount.dev.hda[`mount.dev.hda`] and the input files for `fdisk` and
`sfdisk`, link:#dev.hda[`dev.hda`] and link:#dev.hda.sfd[`dev.hda.sfd`],
respectively.

In addition, if link:#make.fdisk[`make.fdisk`] encounters a FAT
partition, it preserves the partition's boot sector in a file named
`dev.xy`, where x is the drive's device name (e.g. sdc, hda) and y is
the partition number. The boot sector is the first sector, 512 bytes, of
the partition. This sector is restored at the same time the partitions
are rebuilt, in the script link:#make.dev.hda[`make.dev.hda`].

Fortunately, the price of hard drives is plummeting almost as fast as
the public's trust in politicians after an election. So it is good that
the output files are text, and allow hand editing. That's the most
difficult but most flexible way to rebuild on a larger replacement
drive. (See the link:#todo[To Do list].)

Other metadata are preserved by the script
link:#save.metadata[`save.metadata`]. The script saves the partition
information in the file `fdisk.hda` in the root of the ZIP disk. It is a
good idea to print this file and your `/etc/fstab` so that you have hard
copy should you ever have to restore the partition data manually. You
can save a tree by toggling between two virtual consoles, running
`fdisk` in one and catting `/etc/fstab` or `/fdisk.hda` as needed.
However, doing so is error prone.

You will also want to preserve files relevant to your restoration
method. For example, if you use NFS to save your data, you will need to
preserve hosts.allow, hosts.deny, exports, etc. Also, if you are using
any network-backed restoration process, such as Amanda or Quick Restore,
you will need to preserve networking files like HOSTNAME, hosts, etc.
and the relevant software tree.

The simplest way to handle these and similar questions is to preserve
the entire `/etc` directory.

There is no way a 100 MB ZIP drive is going to hold a server
installation of a modern distribution of Linux. We have to be much more
selective than simply preserving the whole kazoo. What files do we need?

* The boot directory.
* The /etc directory and subdirectories.
* Directories needed at boot time.
* Device files in /dev.

To determine the directories needed at boot, we look at the boot
initialization file `/etc/rc.sysinit`. It sets its own path like so:

....
....

Trial and error indicated that we needed some other directories as well,
such as `/dev`. In Linux, you can't do much without device files.

In reading the script link:#save.metadata[`save.metadata`], note that we
aren't necessarily saving files that are called with absolute paths.

We may require several iterations of back up, test the bare metal
restore, re-install from CD and try again, before we have a working
backup script. While I worked on this HOWTO, I made five such iterations
before I had a successful restoration. That is one reason why it is
essential to use scripts whenever possible. Test thoroughly!

One thing you can do on an RPM based system is use the `rpm` program to
determine which files are where. For example, to get a complete list of
the files used by the openssh package, run:

....
bash# rpm -ql openssh
....

There are some things you don't need, like the man pages. You can
inspect each one and decide whether to back it up or not.

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

The second stage of restoration is run without overwriting previously
restored files. This means that the files restored in the first stage
are the ones that will be used after full restoration. So update your
bare metal backups whenever you update files in these directories!
____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

Recent kernels have incorporated a new ATA (IDE) hard drive driver,
libata. Because of this, parallel ATA drives (PATA) now show up as SCSI
drives, as serial ATA (SATA) have always done. However, not all rescue
distributions (e.g. finix) use this new driver. There is a line toward
the bottom of link:#save.metadata[`save.metadata`] wich very carefully
replaces "/dev/sda" with "/dev/hda". Use this as a template if you have
multiple IDE hard drives. Comment it out or delete it if this is not an
issue for you.

Note that there is no guaranteed mapping! Systems with multiple hard
drives may have confusing mappings. Be sure to edit this line carefully.
Check it if you add or remove a hard drive of any interface type to or
from your system!

N.B: if you have libata IDE drive issues, the grub-install line at the
end of link:#restore.metadata[`restore.metadata`] won't work. If it
doesn't, use your rescue disk to do the same. Or burn and boot to the
boot image that is made as part of the first stage backup. Boot to it
and do the second state restore as usual. The second state restore
should re-run `grub-install` or you can run it manually.
_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

The version of `tar` included in http://www.toms.net/rb[tomsrtbt] does
not preserve ownership when it restores. This may cause problems for
applications like link:#amanda[Amanda]. A backup and restoration tool,
Amanda has several directories owned by its own eponymous user. The
solution is:

* Note which directories and files are not owned by root.
* Note their owners.
* Arrange to set the ownership correctly as part of the restoration
process. E.g:
+
....
bash# chown -R amanda:disk /var/lib/amanda
....
+
You can also add that line to your scripts for second state restoration,
such as link:#restore[`restore`].
___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

http://www.toms.net/rb[tomsrtbt] does not support restoring owners by
UID/GID. To make backups suitable for restoring with
http://www.toms.net/rb[tomsrtbt], remove the tar command line option
``--numeric-owner'' from the command line options for tar in the
function crunch in the script link:#save.metadata[`save.metadata`].
____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

The Archive
~~~~~~~~~~~

All of this gets stored into an archive under
`/var/lib/bare.metal.recovery`. Each day a first stage backup is made a
new directory is prepared, with the date encoded as YYYYMMDD, and the
day's archive deposited therein. It is up to you to prune obsolete
archives. It is a good idea to keep at least one old archive around in
case the computer crashes while you are making an archive. If a second
archive is made in a day, the earlier one for that day is replaced.

The files in the archive directory include a `README.txt`, which has
information about the backup and the computer the backup was made on.
Other files are there in case hand intervention is required.

Below the daily archive directory are several text files and three
directories. The scripts reside in `bin`, the tarballs in `data`, and
information about the system such as partitions and LVM volume backups
are in `metadata`.

To create a CD, simply use a script or graphical tool to create a CD
starting at the daily archive directory. It is up to you to be sure your
archive will fit onto your medium, or to make other arrangements.

[[ThemeAndVariations]]
Theme And Variations
~~~~~~~~~~~~~~~~~~~~

No ZIP drive
^^^^^^^^^^^^

This backup process used to require you to have the ZIP disk drive
present at each backup. It now creates the archive in a directory, which
you can back up over the net. Then you only need to build a ZIP disk
(with `cp -rp`) on the backup server when you need to restore.

The backup process will be faster than directly writing to the ZIP
drive, but you should check that the resulting directory will fit on
your ZIP disk (with the output of `du -hs $target.zip` in the script
link:#save.metadata[`save.metadata`])! See the definition of the
variable `zip` in that script.

One of my laptops has problems running both a network card and a ZIP
drive, so this is the process I use to back it up. I keep a backup image
as well as the current one, so that I have a fallback in case the
computer crashes during a backup.

CD-ROM
^^^^^^

This is similar to the no ZIP drive option above. Save your backups to a
directory on your hard drive, as noted. Then use `mkisofs` to create an
ISO 9660 image from that directory, and burn it. This does not work with
some CD-ROM based restoration Linuxes, like
http://www.knoppix.org/[Knoppix], because the Linux has to have the
CD-ROM drive. Unless you have two CD-ROM drives, say one in a USB
clamshell. I have a DVD burner set up this way with exactly this in
mind. Or have http://www.finnix.org/[Finnix] load itself into memory on
boot and then use the CD-ROM drive from which you booted.

These remarks should also apply to DVDs.

Also, look at
http://www.knoppix.net/wiki/Knoppix_Remastering_Howto[remastering]
Knoppix with your first and second stage backups on the CD-ROM. You
should also be able to http://www.finnix.org/Remastering_Finnix[remaster
Finnix].

These days many computers come with a CD-ROM drive but no floppy
diskette. And floppy drives do fail. So it's a good idea to burn your
CD-ROM with a bootable image on it. The bad news is that the ``El
Torito'' format supports 1.2 MB, 1.44 MB and 2.88 MB floppy images, and
http://www.toms.net/rb[tomsrtbt] uses a 1.7 MB floppy. The good news is
that you can get a 2.88 MB version, `tomsrtbt-2.0.103.ElTorito.288.img`,
from the same mirrors where you get the floppy image. Place a _copy_
footnote:[I emphasize copy because `mkisofs` will mung the file in the
directory from which it makes the ISO image.] in the root directory of
the backup files. Then use the `mkisofs` command line option -b to
specify `tomsrtbt-2.0.103.ElTorito.288.img` as the boot image file.

The only down side of this process is that many older BIOSes do not
support 2.88 MB floppy images on CD-ROMs. Most of those will boot to a
http://www.toms.net/rb[tomsrtbt] floppy.

An alternative is to use http://syslinux.zytor.com/[Syslinux]. It is not
dependent on a floppy diskette image, and you can build your own CD with
a number of tools, such as http://www.toms.net/rb[tomsrtbt], on it.

You may have to adjust the BIOS options to allow the computer to boot to
CD-ROM drive. If you can't do that, either because the BIOS won't
support booting to CD-ROM, or because you can't get into the BIOS, see
http://btmgr.webframe.org/[Smart Boot Manager (SBM)] as described in the
link:#resources[Resources].

One variant is to dispense with the tarballs in the first stage, and
create a tarball of the entire system. When you build your restoration
CD, put the monster tarball in the data directory of the CD. The scripts
will pick that up and restore for you, combining the first and second
stages. This eliminates a lot of the cruft related to permissions and
ownership in link:#restore.metadata[`restore.metadata`] and
link:#save.metadata[`save.metadata`]

_Test_ your CDs on the drive you will use at restoration time. If you
find you need to hack the scripts, you can copy them to `/tmp`, usually
a RAM drive, and edit them there. The scripts will run there. As a RAM
disk is volatile, be sure to save your changes before you reboot!

NFS
^^^

If you back up across your network to a backup server, you will have all
the files on it you need. Set up the directory where you keep all your
backups as an NFS export.

Then, on http://www.finnix.org/[Finnix], do the following (tab
completion is very nice here):

....
# mkdir /mnt/nfs
# /etc/init.d/portmap start
# mount server:/path/of/exportedfs /mnt/nfs
# cd /mnt/nfs/.../bin
....

Now restore as usual.

There are several advantages to NFS for this job: You don't have to
worry about space on a CD-ROM or
http://www.iomega.com/zip/products/par100_250.html[ZIP drive]. You can
edit scripts on the server and they are preserved when you reboot the
target.

Multiple ZIP disks
^^^^^^^^^^^^^^^^^^

By splitting up the two first stage scripts,
link:#restore.metadata[`restore.metadata`] and
link:#save.metadata[`save.metadata`], you could spread the first stage
metadata across multiple ZIP disks.

Excluding From First Stage Saving
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There are time when you need to squeeze a few megabytes from the first
stage data, especially when you are pushing the limit of your ZIP disk.
The function `crunch` in the script link:#save.metadata[`save.metadata`]
takes multiple parameters to feed to `tar`. It can also take the
`--exclude` parameter. So, for example, you can exclude the `samba` and
`X11` directories under `/etc` like so:

....
....

Why those two? Because they're hard drive space hogs and we don't need
them when booting after the first stage.

If you keep multiple kernels around, you can eliminate the modules for
all of the kernels you won't boot to. Check your `lilo.conf` or
`/boot/grub/menu.lst` to see which kernel you will use, and then check
`/lib/modules` for module directories you can exclude.

How to find more good candidates for exclusion? List the target
directories with `ls -alSr` for individual files, and `du | sort -n` for
directories.

Another (probably neater) way to exclude directories is to put a
complete list of directories into a file, then refer to it via the tar
option `--exclude-from=FILENAME`.

[[initrd]]
Initrd
^^^^^^

If your system uses an initial RAM disk, or initrd, to boot, make sure
that link:#restore.metadata[`restore.metadata`] creates the directory
`/initrd`. The easiest way to do this is to ensure that it is included
in the list of directories used in the directory creating loop toward
the end.

Your system will probably use an initrd if it boots from a SCSI drive or
has root on an ext3fs partition. Check `/etc/lilo.conf` or
`/boot/grub/menu.lst` to see if it calls for one.

[[firststagerestore]]
First Stage Restore
-------------------

[[Booting]]
Booting
~~~~~~~

The first thing to do is to verify that the hardware time is set
correctly. Use the BIOS setup for this. How close to exact you have to
set the time depends on your applications. For restoration, within a few
minutes of exact time should be accurate enough. This will allow
time-critical events to pick up where they left off when you finally
launch the restored system.

[[Bootingtomsrtbt]]
tomsrtbt
^^^^^^^^

Before booting http://www.toms.net/rb[tomsrtbt], make sure your ZIP
drive is installed on a parallel port, either `/dev/lp0` or `/dev/lp1`.
The start-up software will load the parallel port ZIP drive driver for
you.

The next step is to set the video mode. I usually like to see as much on
the screen as I can. When the option to select a video mode comes, I use
mode 6, 80 columns by 60 lines. Your hardware may or may not be able to
handle high resolutions like that, so experiment with it.

[[Bootingknoppix]]
Knoppix
^^^^^^^

These instructions will probably work with other CD-ROM or USB pen
Linuxes, but you may have to vary them to suit.

Before booting http://www.knoppix.org/[Knoppix], make sure your ZIP
drive (or substitute) is installed on a parallel port, either `/dev/lp0`
or `/dev/lp1`. Knoppix does not load the parallel port ZIP drive driver
for you. Instead, use the command `modprobe ppa` (as root) to install
it.

Boot http://www.knoppix.org/[Knoppix] as usual. I find it faster and
more useful to boot to a console. At the boot menu, use the command
``knoppix 2''. Then become the root user, with `su -`. For the password,
just hit return.

[[bootingfinnix]]
Finnix
^^^^^^

One option for booting http://www.finnix.org/[Finnix] is the "toram"
option, which lets you move the whole kazoo into RAM. That in turn
should let you load another CD, with your first stage data, into the CD
drive.

[[restoration]]
Restoration
~~~~~~~~~~~

These instructions assume you are running
http://www.toms.net/rb[tomsrtbt]. If you are using a different Linux for
your restore system, you may have to adjust these instructions a bit.
For example, you should always run these scripts as root even if some
other user gives you the requisite privileges.

Once the restoration Linux has booted and you have a console, mount the
ZIP drive. It is probably a good idea to mount it read only. On
http://www.toms.net/rb[tomsrtbt]:

....
# mount /dev/sda1 /mnt -o ro
....

Check to be sure it is there:

....
# ls -l /mnt
....

On http://www.knoppix.org/[Knoppix] or http://www.finnix.org/[Finnix],
you may want to make a directory under `/mnt` and mount it there, like
so:

....
# mkdir /mnt/zip
# mount /dev/sda1 /mnt/zip -o ro
....

Now cd into the mounted device, and into the `bin` directory below it.
On http://www.finnix.org/[Finnix], for example:

....
# cd /mnt/zip/bin
....

The scripts assume you are in this directory, and call data files
relative to it. At this point, you can run the restoration automatically
or manually. Use the automated restore if you don't need to make any
changes as you go along.

One consideration here is whether you have multiple hard drives. If your
Linux installation mounts partitions on multiple hard drives, you must
mount the root partition first. This is to ensure that mount point
directories are created on the partition where they belong. The script
link:#first.stage[first.stage] will run the scripts to mount the drives
in the order in which they are created. If you have created them (in the
script `save.metadata`) in the order in which they cascade from root,
the mounting process should work just fine.

If you have multiple hard drives, and they cross-mount, you are on your
own. Either combine and edit the scripts to mount them in the correct
order, or do it manually.

Automated
^^^^^^^^^

The automatic process calls each of the manual scripts in proper order.
It does not allow for manual intervention, say for creating file systems
that this HOWTO does not support. To run the first stage restore
automatically, enter the command:

....
# first.stage
....

If you want to check for back blocks, add the `-c` option.

Manually
^^^^^^^^

Run the script(s) that will restore the partition information and create
file systems. You may run them in any order, so long as they build
dependencies in the correct order. You can read the script
link:#first.stage[first.stage] to get an idea of the order. e.g.:

....
# ./make.dev.hda
....

If you want to check for back blocks, add the `-c` option.

This script will:

* Clean out the first 1024 bytes of the hard drive, killing off any
existing partition table and master boot record (MBR).
* Recreate the non-LVM partitions from the information gathered when you
ran link:#make.fdisk[`make.fdisk`].
* Make ext2 and ext3 file systems on non-LVM partitions and Linux swap
partitions as appropriate. If you provide the `-c` option to the script,
it will also check for bad blocks.
* Make some types of FAT partitions.

Now is a good time to check the geometry of the drive. Sometimes
different versions of Linux pick up different geometries, so the
geometry implicit in the file `dev.hdX` is incorrect. To force it to be
correct on http://www.knoppix.org/[Knoppix], edit
link:#make.dev.hda[`make.dev.x`]. Use the -C, -H and -S options to fdisk
to specify the cylnders, heads and sectors, respectively. Those you can
get from the file `fdisk.hdX` in the root directory of the ZIP drive.
Then re-run it.

_______________________________________________________________________________________________________________________________________________________________________________________________
Note: If you have other operating systems or file systems to restore, now is a good time to do so. When you've done that, reboot to your restoration Linux and continue your first stage restoration.
_______________________________________________________________________________________________________________________________________________________________________________________________

If you have LVM volumes to restore, now is the time to run `make.lvs`
and `mount.lvs`.

Now run the script(s) that create mount points and mount the partitions
to them.

....
# ./mount.dev.hda
....

Once you have created all your directories and mounted partitions to
them, you can run the script link:#restore.metadata[`restore.metadata`].

....
# ./restore.metadata
....

This will restore the contents of the ZIP drive to the hard drive to
give you a minimal bootable system.

You should see a directory of the ZIP disk's root directory, then a list
of the archive files as they are restored. Tar on
http://www.toms.net/rb[tomsrtbt] will tell you that tar's block size is
20, and that's fine. You can ignore it. Be sure that lilo prints out its
results:

....
....

That will be followed by the output from a ```df -m`'' command.

Finishing Touches
^^^^^^^^^^^^^^^^^

If you normally boot directly to X, you could have some problems. To be
safe, the first stage script changes the run level in
`/target/etc/inittab` to 3. Note: different distributions use different
run level schemes. 3 works on Red Hat derived distributions; it may not
on others.

You can now gracefully reboot. Remove the medium from your boot drive if
you haven't already done so, and give the computer the three fingered
salute, or its equivalent:

....
# shutdown -r now
....

or

....
# reboot
....

The computer will shut down and reboot.

[[SecondStageRestoration]]
Second Stage Restoration
------------------------

As the computer reboots, go back to the BIOS and verify that the clock
is more or less correct.

Once you have verified the clock is correct, exit the BIOS and reboot to
the hard drive. You can simply let the computer boot in its normal
sequence. You will see a lot of error messages, mostly along the lines
of ``I can't find blah! Waahhh!'' If you have done your homework
correctly up until now, those error messages won't matter. You don't
need linuxconf or apache to do what you need to do.

___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

As an alternative, you can boot to single user mode (at the lilo prompt,
enter `linux single`), but you will have to configure your network
manually and fire up sshd or whatever daemons you need to restore your
system. How you do those things is very system specific.
___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

You should be able to log into a root console (no X, no users, sorry).
You should now be able to use the network, for example to NFS mount the
backup of your system.

If you did the two stage backup I suggested for Arkeia, you can now
restore Arkeia's database and executables. You should be able to run

....
/etc/rc.d/init.d/arkeia start
....

and start the server. If you have the GUI installed on another computer
with X installed, you should now be able to log in to Arkeia on your
tape server, and prepare your restoration.

_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
*Note*

When you restore, read the documentation for your restoration programs
carefully. For example, tar does not normally restore certain
characteristics of files, like suid bits. File permissions are set by
the user's umask. To restore your files exactly as you saved them, use
tar's p option. Similarly, make sure your restoration software will
restore everything exactly as you saved it.
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

To restore the test computer:

....
bash# restore.all
....

If you used tar for your backup and restoration, and used the -k (keep
old files, don't overwrite) option, you will see a lot of this:

....
....

This is normal, as tar is refusing to overwrite files you restored
during the first stage of restoration.

Then reboot. On the way down, you will see a lot of error messages, such
as ``no such pid.'' This is a normal part of the process. The shutdown
code is using the pid files from daemons that were running when the
backup was made to shut down daemons that were not started on the last
boot. Of course there's no such pid.

Your system should come up normally, with a lot fewer errors than it had
before; ideally no errors. The acid test of how well your restore works
on an RPM based system is to verify all packages. During the first stage
backup, a verification was performed on the system, producing the file
`rpmVa.txt`. Verify your system again, and compare the results to the
one made earlier. E.g.:

....
bash# rpm -Va | sort +2 -t ' ' | uniq > ~/foo.txt
diff /mnt/zip/metadata/rpmVa.txt ~/foo.txt
....

Prelinking error messages are normal and you can ignore them. Do not
first run the command `/etc/cron.daily/prelink` to remove them. Doing so
may introduce new errors in the verification results that will skew your
results.

Some files, such as configuration and log files, will have changed in
the normal course of things, and you should be able to mentally filter
those out of the report. Emacs users should check out its ediff
facilities.

Now you should be up and running. It is time to test your applications,
especially those that run as daemons. The more sophisticated the
application, the more testing you may need to do. If you have remote
users, disable them from using the system, or make it ``read only''
while you test it. This is especially important for databases, to
prevent making any corruption or data loss worse than it already might
be.

If you normally boot to X, it was disabled as part of the first stage
restoration. Test X before you re-enable it. Re-enable it by changing
that one line in `/etc/inittab`. Find the line that looks like this:

....
....

and change it to this:

....
....

Or just run this on the target to change it back. Note: different
distributions use different run level schemes. These values work on Red
Hat derived distributions; they may not on others.

....
sed -i s/id:.:initdefault:/id:5:initdefault:/g /etc/inittab
....

You should now be ready for rock and roll -- and some aspirin and a
couch.

[[DistributionSpecificNotes]]
Distribution Specific Notes
---------------------------

Below are distribution notes from past experiences. If you have
additional notes that you would like to add for other distributions,
please forward them to me.

[[fedora3]]
Fedora
~~~~~~

The scripts now reflect Fedora 7, so you should not have to make any
changes to these link:#thescripts[scripts].

________________________________________________________________________________________________________________________________________________________________
*Note*

I tested the above on a fresh installation of FC3. I had problems with
devices after booting when I worked with a system that had been upgraded
from FC2 to FC3.
________________________________________________________________________________________________________________________________________________________________

[[knoppix]]
Knoppix
~~~~~~~

I used to use http://www.knoppix.org/[Knoppix].
mailto:pon at iki dot fi[Pasi Oja-Nisula] reports:

______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
For me the best thing about using Knoppix is that I don't need a
specific boot medium for each machine, but I can use the same tools all
the time. And hardware support in Knoppix is really great. I don't have
that much experience with different platforms, but all the machines I've
tried have worked fine, scsi drivers are found and so on.

I'm doing this recovery thing by copying the backups over the network to
other machine. The restore involves booting the Knoppix cd, fetching the
metadata.tar.gz from the network machine. Then make.dev, mount.dev,
fetching the other tar.gz files, grub and reboot. Some typing involved
but thanks to your scripts it's quite straighforward. Unless changing
from ide to scsi or something, but even then it's not that difficult,
since Linux is easy to restore to different hardware.
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Let me add to that that http://www.knoppix.org/[Knoppix] detects USB
devices for you, which is really nice. They make excellent (and roomier)
substitutes for the ZIP drive.

Also see
http://www-106.ibm.com/developerworks/linux/library/l-knopx.html?ca=dgr-lnxw04Knoppix[``System
recovery with Knoppix''].

Do your restore as user ``root'' rather than as user ``knoppix''.
Otherwise you may get some directories and files owned by an oddball
user or group. Also, for http://www.knoppix.org/[Knoppix], we tar the
first stage stuff saving numeric user & group values instead of by name.
The names may point to different numbers on knoppix, so we would be
restoring the files with incorrect user and group IDs.

Finnix
~~~~~~

http://www.finnix.org/[Finnix] has some of the same advantages of
Knoppix. In addition, it runs in command line mode with mouse support,
which is great for the task at hand. It's small, under 100 MB as of this
writing, so you can remaster it with your first stage data on it. It
boots quickly. And it has LVM support. And Zile, a subset of Emacs. I am
pleased with http://www.finnix.org/[Finnix] for this use, and it is now
my standard first stage restoration Linux.

[[ApplicationSpecificNotes]]
Application Specific Notes
--------------------------

Here are some notes about backing up particular applications.

[[lvm]]
Logical Volume Manager
~~~~~~~~~~~~~~~~~~~~~~

Handling logical volumes turns out to be a bit of a trick: use the
http://www.finnix.org/[Finnix] distribution's startup code to turn LVM
on and off. This results in distribution specific code for the first
stage of restoration. It is generated in link:#make.fdisk[`make.fdisk`].
To edit it, search link:#make.fdisk[`make.fdisk`] for ``Hideous''.

LVM required the addition of two new LVM specific scripts,
link:#make.lvs[`make.lvs`] and link:#mount.lvs[`mount.lvs`]. They are
only generated and used if there are logical volumes present.

[[selinux]]
Selinux
~~~~~~~

Selinux is disabled on the test machines. `/selinux` is not backed up in
any of these scripts. At a guess, you should probably disable selinux
after the first stage restoration, and you will probably have some
selinux specific tasks to perform before turning it back on.

[[grub]]
GRUB
~~~~

The default bootloader in link:#fedora3[Fedora] is the
http://www.gnu.org/software/grub/[Grand Unified Bootloader (GRUB)]. It
has to run at the end of the first stage, or you won't be able to boot
thereafter. To preserve it for first stage restoration, make the
following changes:

* Edit the penultimate stanza of
link:#restore.metadata[`restore.metadata`]:
+
....
....
* Add the following stanza to link:#save.metadata[`save.metadata`]:
+
....
....

[[tripwire]]
Tripwire
~~~~~~~~

If you run Tripwire or any other application that maintains a database
of file metadata, rebuild that database immediately after restoring.

[[Squid]]
Squid
~~~~~

Squid is a HTTP proxy and cache. As such it keeps a lot of temporary
data on the hard drive. There is no point in backing that up. Insert
``--exclude /var/spool/squid'' into the appropriate tar command in your
second stage backup script. Then, get squid to rebuild its directory
structure for you. Tack onto the tail end of the second stage restore
script a command for squid to initialize itself. Here is how I did it
over SSH in link:#restore[`restore`]:

....
....

The last command creates a file of length 0 called .OPB_NOBACKUP. This
is for the benefit of link:#arkeia[Arkeia], and tells Arkeia not to back
up below this directory

[[Arkeia]]
Arkeia
~~~~~~

These notes are based on testing with Arkeia 4.2.

http://www.arkeia.com/[Arkeia] is a backup and restore program that runs
on a wide variety of platforms. You can use Arkeia as part of a bare
metal restoration scheme, but there are two caveats.

The first is probably the most problematic, as absent any more elegant
solution you have to hand select the directories to restore in the
navigator at restoration time. The reason is that, apparently, Arkeia
has no mechanism for not restoring files already present on the disk,
nothing analogous to `tar`'s -p option. If you simply allow a full
restore, the restore will crash as Arkeia over-writes a library which is
in use at restore time, e.g. `lib/libc-2.1.1.so`. Hand selection of
directories to restore is at best dicey, so I recommend against it.

The second caveat is that you have to back up the Arkeia data dictionary
and/or programs. To do that, modify the `save.metatdata` script by
adding Arkeia to the list of directories to save:

....
 $zip/arkeia.tar.gz]]>
....

You _must_ back up the data dictionary this way because Arkeia does not
back up the data dictionary. This is one of my complaints about Arkeia,
and I have solved it in the past by saving the data dictionary to tape
with http://www.estinc.com/[The TOLIS Group's BRU].

The data dictionary will be restored in the script `restore.metadata`
automatically.

[[amanda]]
Amanda
~~~~~~

http://www.amanda.org/[Amanda] (The Advanced Maryland Automatic Network
Disk Archiver) works quite well with this set of scripts. Use the normal
Amanda back-up process, and build your first stage data as usual. Amanda
stores the data on tape in GNU tar or cpio format, and you can recover
from individual files to entire backup images. The nice thing about
recovering entire images is that you can then use variants on the
scripts in this HOWTO to restore from the images, or direct from tape. I
was able to restore my test machine with the directions from W. Curtis
Preston's http://www.oreilly.com/catalog/unixbr/[Unix Backup &
Recovery]. For more information on it, see the
link:#resources[Resources]. The Amanda chapter from the book is
http://www.backupcentral.com/amanda.html[on line].

I made two changes to the script link:#restore[`restore`]. First, I
changed it to accept a file name as an argument. Then, since Amanda's
`amrestore` decompresses the data as it restores it, I rewrote it to cat
the file into the pipe instead of decompressing it.

The resulting line looks like this:

....
cat $file | ssh $target "umask 000 ; cd / ; tar -xpkf - "
....

where `$file` is the script's argument, the image recovered from the
tape by `amrestore`.

Since the command line arguments to `tar` prohibit over-writing, restore
from images in the _reverse_ of the order in which they were made.
Restore most recent first.

Amanda does require setting ownership by hand if you back up the amanda
data directory with link:#save.metadata[`save.metadata`]. Something
like:

....
bash# chown -R amanda:disk /var/lib/amanda
....

You can also add that line to your scripts for second state restoration,
such as link:#restore[`restore`].

[[ntfs]]
NTFS
~~~~

OK, NTFS isn't an application. It is a file system used by Microsoft
operating system Windows NT and its descendents, including Windows 2000
and Windows XP. You can back it up and restore to it from Linux with
`ntfsclone`, one of the NTFS utilities in the ntfsprogs suite, available
from http://www.linux-ntfs.org/.

These scripts will create NTFS partitions, but will not put a file
system on them. It is not clear from the docs whether `ntfsclone` will
lay down a file system on a virgin partition or not.

[[SomeAdviceforDisasterRecovery]]
Some Advice for Disaster Recovery
---------------------------------

You should take your ZIP disk for each computer and the printouts you
made, and place them in a secure location in your shop. You should store
copies of these in your off-site backup storage location. The major
purpose of off-site backup storage is to enable disaster recovery, and
restoring each host onto replacement hardware is a part of disaster
recovery.

You should also have several restoration Linux floppies or CD-ROMS, and
possibly some ZIP drives in your off-site storage as well. Also, have
copies of the rescue linux distribution on several of your computers so
that they back each other up.

You should probably have copies of this HOWTO, with your site-specific
annotations on it, with your backups and in your off-site backup
storage.

[[WhatNow]]
What Now?
---------

This HOWTO results from experiments on one computer. No doubt you will
find some directories or files you need to back up in your first stage
backup. I have not dealt with saving and restoring X on the first stage,
nor have I touched at all on processors other than AMD or Intel.

I would appreciate your feedback as you test and improve these scripts
on your own computers. I also encourage vendors of backup software to
document how to do a minimal backup of their products. I'd like to see
the whole Linux community sleep just a little better at night.

[[todo]]
To Do
~~~~~

Volunteers are most welcome. Check with me before you start on one of
these in case someone else is working on it already.

* We have no way to determine the label of a swap partition. This means
that there is no way to provide the swap partition's label when
restoring. We could assume that a system with a single swap partition
(as indicated by fdisk) has the label used in the swap partition line in
`/etc/fstab`, but that only works on single hard drive systems, and
could produce subtle errors in systems with multiple swap partitions.
+
The work-around is to add the label by hand by re-running `mkswap` with
the -L option on it. Sigh.
* A partition editor to adjust partition boundaries in the `dev.hdx`
file. This will let users adjust partitions for a different hard drive,
or the same one with different geometry, or to adjust partition sizes
within the same hard drive. A GUI would probably be a good idea here. On
the other tentacle, the FSF's
http://www.gnu.org/software/parted[`parted`] looks like it will fill
part of the bill. It does re-size existing partitions, but with
restrictions.
* link:#make.fdisk[`make.fdisk`] currently only recognizes some FAT
partitions, not all. Add code to link:#make.fdisk[`make.fdisk`] to
recognize others and make appropriate instructions to rebuild them in
the output files.
* For FAT12 or FAT16 partitions we do not format, write zeros into the
partition so that Mess-DOS 6.x does not get confused. See the notes on
`fdisk` for an explanation of the problem.
* Translations into other (human) languages.
* I've referred to Red Hat Package Manager (rpm) from time to time. What
are the equivalent deb commands?
* Modify the first stage backup code to only save the current kernel.

[[TheScripts]]
The Scripts
-----------

See the notes in the beginning of each script for a summary of what it
does.

[[FirstStage]]
First Stage
~~~~~~~~~~~

[[make.fdisk]]
`make.fdisk`
^^^^^^^^^^^^

This script, run at backup time, creates scripts similar to
link:#make.dev.hda[`make.dev.hda`] and
link:#mount.dev.hda[`mount.dev.x`], below, for you to run at restore
time. It also produces data files similar to link:#dev.hda[`dev.hda`]
and link:#dev.hda.sfd[`dev.hda.sfd`], below. The names of the scripts
and data files produced depend on the device given this script as a a
parameter. Those script, run at restore time, build and mount the
partitions on the hard drive. `make.fdisk` is called from
link:#save.metadata[`save.metadata`], below.

....
....

[[make.dev.hda]]
`make.dev.hda`
^^^^^^^^^^^^^^

This script is a sample of the sort produced by
link:#make.fdisk[`make.fdisk`], above. It uses data files like
link:#dev.hda[`dev.hda`], below. It builds partitions and puts file
systems on some of them. This is the first script run at restore time.

If you are brave enough to edit link:#dev.hda[`dev.hda`] or
link:#dev.hda.sfd[`dev.hda.sfd`] (q.v.), say, to add a new partition,
you may need to edit this script as well.

If you want make.dev.hda to check for bad blocks when it puts a file
system on the partitions, use a "-c" command line option.

....
....

[[make.lvs]]
`make.lvs`
^^^^^^^^^^

`make.lvs` is generated by link:#make.fdisk[`make.fdisk`], but only if
logical volumes are present. As the name suggests, it builds the logical
volumes and makes file systems on them.

....
....

[[mount.dev.hda]]
`mount.dev.hda`
^^^^^^^^^^^^^^^

This script is a sample of the sort produced by
link:#make.fdisk[`make.fdisk`], above. It builds mount points and mounts
partitions on them, making the target file system ready for restoring
files. This is the second script run at restore time.

If you are brave enough to edit link:#dev.hda[`dev.hda`] (q.v.), say, to
add a new partition, you may need to edit this script as well.

....
....

[[mount.lvs]]
`mount.lvs`
^^^^^^^^^^^

`mount.lvs` is generated by link:#make.fdisk[`make.fdisk`], but only if
logical volumes are present. As the name suggests, it mounts the logical
volumes ready for restoration.

....
....

[[dev.hda]]
`dev.hda`
^^^^^^^^^

This data file is used at restore time if `sfdisk` is not present on the
restoration Linux. It is fed to `fdisk` by the script
link:#make.dev.hda[`make.dev.hda`]. It is produced at backup time by
link:#make.fdisk[`make.fdisk`]. Those familiar with `fdisk` will
recognize that each line is an `fdisk` command or value, such as a
cylinder number. Thus, it is possible to change the partition sizes and
add new partitions by editing this file. That's why the penultimate
command is `v`, to verify the partition table before it is written.

....
....

[[dev.hda.sfd]]
`dev.hda.sfd`
^^^^^^^^^^^^^

This data file is used at restore time if `sfdisk` is present on the
restoration Linux system. It is fed to `sfdisk` by the script
link:#make.dev.hda[`make.dev.hda`]. It is produced at backup time by
link:#make.fdisk[`make.fdisk`]. Each line represents a partition. Thus,
it is possible to change the partition sizes and add new partitions by
editing this file.

....
....

[[save.metadata]]
`save.metadata`
^^^^^^^^^^^^^^^

This is the first script to run as part of the backup process. It calls
link:#make.fdisk[`make.fdisk`], above. If you have a SCSI hard drive or
multiple hard drives to back up, edit the call to
link:#make.fdisk[`make.fdisk`] appropriately.

 
*Note*

Recent kernels have incorporated a new ATA (IDE) hard drive driver,
libata. Because of this, parallel ATA (PATA) drives now show up as SCSI
drives, as serial ATA (SATA) have always done. However, not all rescue
distributions (e.g. Finnix) use this new driver. There is a line toward
the bottom of link:#save.metadata[`save.metadata`] wich very carefully
replaces "/dev/sda" with "/dev/hda". Use this as a template if you have
multiple IDE hard drives. Comment it out or delete it if this is not an
issue for you.

Note that there is no guaranteed mapping! Systems with multiple hard
drives may have confusing mappings. Be sure to edit this line carefully.
Check it if you add or remove a hard drive of any interface type to or
from your system!

N.B: if you have libata IDE drive issues, the grub-install line at the
end of link:#restore.metadata[`restore.metadata`] won't work. If it
doesn't, use your rescue disk to do the same. Or burn and boot to the
boot image that is made as part of this script. Boot to it and do the
second state restore as usual. The second state restore should re-run
`grub-install`.
 

....
....

[[restore.metadata]]
`restore.metadata`
^^^^^^^^^^^^^^^^^^

This script restores metadata from the ZIP disk as a first stage
restore.

N.B: if you have libata IDE drive issues, the grub-install line at the
end of this script won't work. If it doesn't, use your rescue disk to do
the same.

....
....

[[first.stage]]
`first.stage`
^^^^^^^^^^^^^

This script runs the entire first stage restore with no operator
intervention.

If you want to check for bad blocks when it puts a file system on the
partitions, use a "-c" command line option.

....
....

[[SecondStage]]
Second Stage
~~~~~~~~~~~~

These scripts run on the computer being backed up or restored.

[[back.up.all]]
`back.up.all`
^^^^^^^^^^^^^

This script saves to another computer via an NFS mount. You can adapt it
to save to tape drives or other media.

....
....

[[back.up.all.ssh]]
`back.up.all.ssh`
^^^^^^^^^^^^^^^^^

This script does exactly what link:#back.up.all[`back.up.all`] does, but
it uses SSH instead of NFS.

....
....

[[restore.all]]
`restore.all`
^^^^^^^^^^^^^

This is the restore script to use if you backed up using
link:#back.up.all[`back.up.all`].

....
....

[[restore.all.ssh]]
`restore.all.ssh`
^^^^^^^^^^^^^^^^^

This is the restoration script to use if you used
link:#back.up.all.ssh[`back.up.all.ssh`] to back up.

....
....

[[BackupServerScripts]]
Backup Server Scripts
~~~~~~~~~~~~~~~~~~~~~

The SSH scripts above have a possible security problem. If you run them
on a firewall, the firewall has to have access via SSH to the backup
server. In that case, a clever cracker might also be able to crack the
backup server. It would be more secure to run backup and restore scripts
on the backup server, and let the backup server have access to the
firewall. That is what these scripts are for.

These scripts backup and restore the target completely, not just the
stage one backup and restore. `get` backs up the bare metal archive
separately so that you can make a CD-ROM ir NFS mount from it.

I use these scripts routinely.

[[get]]
`get`
^^^^^

....
....

[[restore]]
`restore`
^^^^^^^^^

....
....

[[get.target]]
`get.target`
^^^^^^^^^^^^

....
....

[[misc.files]]
Miscellaneous Files
~~~~~~~~~~~~~~~~~~~

[[install]]
`install`
^^^^^^^^^

This little script just installs things and sets up a few directories.
It would be a useful basis for an RPM or deb package. The placement of
files is based on the __http://www.pathname.com/fhs/[Filesystem
Hierarchy Standard]__, version 2.3, announced on January 29, 2004.

....
....

[[Resources]]
Resources
---------

In no particular order. These are things you might want to investigate
for yourself. A listing here should not be taken as an endorsement. In
fact, in many case I have not used the product and cannot comment on it.

* http://osdev.berlios.de/netboot.html[Network-booting Your Operating
System] describes several techniques for booting across a network, using
http://www.gnu.org/software/grub/[grub] and some other tricks. I haven't
tried it, but I have a sneaky suspicion that with an especially trained
floppy diskette, you could get your entire first stage image onto the
computer to be restored.
* ``http://btmgr.webframe.org/[Smart Boot Manager (SBM)] is an OS
independent and full-featured boot manager with an easy-to-use user
interface. There are some screen shots available.'' It is essential if
your BIOS will not allow you to boot to CD-ROM and you want to use a
CD-ROM based Linux for Stage 1 recovery.
* http://www.oreilly.com/catalog/unixbr/author.html[W. Curtis Preston]'s
excellent http://www.oreilly.com/catalog/unixbr/[Unix Backup &
Recovery]. This is the book that got me started on this bare metal
recovery stuff. I highly recommend it;
http://www2.linuxjournal.com/lj-issues/issue78/3839.html[read my
review].
* An old (2000) list of
http://www.fokus.gmd.de/linux/linux-distrib-small.html[small Linux
disties.]
* http://www.toms.net/rb[tomsrtbt], ``The most Linux on 1 floppy disk.''
Tom also has links to other small disties.
* The http://www.tldp.org/[Linux Documentation Project]. See
particularly the ``LILO, Linux Crash Rescue HOW-TO.''
* The Free Software Foundation's
http://www.gnu.org/software/parted[`parted`] for editing (enlarging,
shrinking, moving) partitions.
* http://qtparted.sourceforge.net/[QtParted] looks to do the same thing
with a GUI front end.
* http://www.partimage.org/[Partition Image] for backing up partitions.
+
From the web page: ``Partition Image is a Linux/UNIX utility which saves
partitions in many formats (see below) to an image file. The image file
can be compressed in the GZIP/BZIP2 formats to save disk space, and
split into multiple files to be copied on removable floppies (ZIP for
example), .... The partition can be saved across the network since
version 0.6.0.''
* http://sourceforge.net/projects/bacula[Bacula] is a GLPled backup
product which has bare metal recovery code inspired in part by this
HOWTO.
* ``http://www.feyrer.de/g4u/[g4u ('ghost for unix')] is a NetBSD-based
bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a
common setup on a number of PCs using FTP. The floppy/CD offers two
functions. First is to upload the compressed image of a local harddisk
to a FTP server. Other is to restore that image via FTP, uncompress it
and write it back to disk; network configuration is fetched via DHCP. As
the harddisk is processed as a image, any filesystem and operating
system can be deployed using g4u.''
* ``We present
http://www.cs.utah.edu/flux/papers/frisbee-usenix03-base.html[Frisbee],
a system for saving, transferring, and installing entire disk images,
whose goals are speed and scalability in a LAN environment. Among the
techniques Frisbee uses are an appropriately-adapted method of
filesystem-aware compression, a custom application-level reliable
multicast protocol, and flexible application-level framing. This design
results in a system which can rapidly and reliably distribute a disk
image to many clients simultaneously. For example, Frisbee can write a
total of 50 gigabytes of data to 80 disks in 34 seconds on commodity PC
hardware. We describe Frisbee's design and implementation, review
important design decisions, and evaluate its performance.''
* There are a number of USB key disties available. Check
http://www.distrowatch.com/[DistroWatch] for details.
* CD-ROM based rescue kits. This is not intended to be an exhaustive
list. If you know of one (or even something that pretends to be one),
please link:&myemail;[let me know]. You may find more recent information
at http://www.distrowatch.com/[DistroWatch].
** Hugo Rabson's http://www.microwerks.net/~hugo/[Mondo] ``... creates
one or more bootable Rescue CD's (or tape+floppies) containing some or
all of your filesystem. In the event of catastrophic data loss, you will
be able to restore from bare metal.''
** The http://crashrecovery.org/[Crash Recovery Kit for Linux]
** http://www-106.ibm.com/developerworks/linux/library/l-knopx.html?ca=dgr-lnxw04Knoppix[``System
recovery with Knoppix''] is a good introduction to system recovery in
general, and has some useful http://www.knoppix.org/[Knoppix] links.
** ``http://emergencycd2.sourceforge.net/[Cool Linux CD] is live CD with
Linux system. This used 2.4 kernel and some free and demo soft.''
** http://www.sysresccd.org/index.en.php[SystemRescueCd]``is a linux
system on a bootable cdrom for repairing your system and your data after
a crash. It also aims to provide an easy way to carry out admin tasks on
your computer, such as creating and editing the partitions of the hard
disk. It contains a lot of system utilities (parted, partimage, fstools,
...) and basic ones (editors, midnight commander, network tools). It
aims to be very easy to use: just boot from the cdrom, and you can do
everything. The kernel of the system supports most important file
systems (ext2/ext3, reiserfs, xfs, jfs, vfat, ntfs, iso9660), and
network ones (samba and NFS).''
** http://syslinux.zytor.com/[Syslinux] builds boot code for floppy
diskettes, CD-ROMs and Intel PXE (Pre-Execution Environment) images. It
is not dependent on a floppy diskette image. You can build your own CDs
with a number of tools, such as http://www.toms.net/rb[tomsrtbt], on it.
** In case you'd like to roll your own:
``http://www.linux-live.org/[Linux Live] is a set of bash scripts which
allows you to create [your] own LiveCD from every Linux distribution.
Just install your favourite distro, remove all unnecessary files (for
example man pages and all other files which are not important for you)
and then download and run these scripts.''
** ``The http://www.linbox.com/en/ppart.html[PPART CD] allows you to
generate system recovery bootable CD of previously saved hard disks.''
** http://rescuecd.sourceforge.net/[Timo's Rescue CD Set]: ``This set is
my approach for an easy way to generate a rescue system on a bootable
cd, which can easily be adapted to the own needs. The project evolves
more and more into a 'debian on cd' project, so it's not only possible
to use the system as a rescuecd, it is also possible to install a whole
debian system on cd.''
** The http://www.frozentech.com/content/livecd.php[List of Live CDs]
has more CD based disties.

[[appendix1gfdl]]
GNU Free Documentation License
------------------------------

Version 1.1, March 2000

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Copyright (C) 2000 Free Software Foundation, Inc. 51 Franklin Street,
Fifth Floor, Boston, MA 02110-1301, USA Everyone is permitted to copy
and distribute verbatim copies of this license document, but changing it
is not allowed.
____________________________________________________________________________________________________________________________________________________________________________________________________________________________________

[[gfdl02]]
PREAMBLE
--------

The purpose of this License is to make a manual, textbook, or other
written document "free" in the sense of freedom: to assure everyone the
effective freedom to copy and redistribute it, with or without modifying
it, either commercially or noncommercially. Secondarily, this License
preserves for the author and publisher a way to get credit for their
work, while not being considered responsible for modifications made by
others.

This License is a kind of "copyleft", which means that derivative works
of the document must themselves be free in the same sense. It
complements the GNU General Public License, which is a copyleft license
designed for free software.

We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free program
should come with manuals providing the same freedoms that the software
does. But this License is not limited to software manuals; it can be
used for any textual work, regardless of subject matter or whether it is
published as a printed book. We recommend this License principally for
works whose purpose is instruction or reference.

[[gfdl03]]
APPLICABILITY AND DEFINITIONS
-----------------------------

This License applies to any manual or other work that contains a notice
placed by the copyright holder saying it can be distributed under the
terms of this License. The "Document", below, refers to any such manual
or work. Any member of the public is a licensee, and is addressed as
"you".

A "Modified Version" of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of
the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document's overall subject
(or to related matters) and contains nothing that could fall directly
within that overall subject. (For example, if the Document is in part a
textbook of mathematics, a Secondary Section may not explain any
mathematics.) The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are
designated, as being those of Invariant Sections, in the notice that
says that the Document is released under this License.

The "Cover Texts" are certain short passages of text that are listed, as
Front-Cover Texts or Back-Cover Texts, in the notice that says that the
Document is released under this License.

A "Transparent" copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the general
public, whose contents can be viewed and edited directly and
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or for
automatic translation to a variety of formats suitable for input to text
formatters. A copy made in an otherwise Transparent file format whose
markup has been designed to thwart or discourage subsequent modification
by readers is not Transparent. A copy that is not "Transparent" is
called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII
without markup, Texinfo input format, LaTeX input format, SGML or XML
using a publicly available DTD, and standard-conforming simple HTML
designed for human modification. Opaque formats include PostScript, PDF,
proprietary formats that can be read and edited only by proprietary word
processors, SGML or XML for which the DTD and/or processing tools are
not generally available, and the machine-generated HTML produced by some
word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus
such following pages as are needed to hold, legibly, the material this
License requires to appear in the title page. For works in formats which
do not have any title page as such, "Title Page" means the text near the
most prominent appearance of the work's title, preceding the beginning
of the body of the text.

[[gfdl04]]
VERBATIM COPYING
----------------

You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License applies to
the Document are reproduced in all copies, and that you add no other
conditions whatsoever to those of this License. You may not use
technical measures to obstruct or control the reading or further copying
of the copies you make or distribute. However, you may accept
compensation in exchange for copies. If you distribute a large enough
number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and
you may publicly display copies.

[[gfdl05]]
COPYING IN QUANTITY
-------------------

If you publish printed copies of the Document numbering more than 100,
and the Document's license notice requires Cover Texts, you must enclose
the copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the
back cover. Both covers must also clearly and legibly identify you as
the publisher of these copies. The front cover must present the full
title with all words of the title equally prominent and visible. You may
add other material on the covers in addition. Copying with changes
limited to the covers, as long as they preserve the title of the
Document and satisfy these conditions, can be treated as verbatim
copying in other respects.

If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.

If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable Transparent
copy along with each Opaque copy, or state in or with each Opaque copy a
publicly-accessible computer-network location containing a complete
Transparent copy of the Document, free of added material, which the
general network-using public has access to download anonymously at no
charge using public-standard network protocols. If you use the latter
option, you must take reasonably prudent steps, when you begin
distribution of Opaque copies in quantity, to ensure that this
Transparent copy will remain thus accessible at the stated location
until at least one year after the last time you distribute an Opaque
copy (directly or through your agents or retailers) of that edition to
the public.

It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to give
them a chance to provide you with an updated version of the Document.

[[gfdl06]]
MODIFICATIONS
-------------

You may copy and distribute a Modified Version of the Document under the
conditions of sections 2 and 3 above, provided that you release the
Modified Version under precisely this License, with the Modified Version
filling the role of the Document, thus licensing distribution and
modification of the Modified Version to whoever possesses a copy of it.
In addition, you must do these things in the Modified Version:

A.  Use in the Title Page (and on the covers, if any) a title distinct
from that of the Document, and from those of previous versions (which
should, if there were any, be listed in the History section of the
Document). You may use the same title as a previous version if the
original publisher of that version gives permission.
B.  List on the Title Page, as authors, one or more persons or entities
responsible for authorship of the modifications in the Modified Version,
together with at least five of the principal authors of the Document
(all of its principal authors, if it has less than five).
C.  State on the Title page the name of the publisher of the Modified
Version, as the publisher.
D.  Preserve all the copyright notices of the Document.
E.  Add an appropriate copyright notice for your modifications adjacent
to the other copyright notices.
F.  Include, immediately after the copyright notices, a license notice
giving the public permission to use the Modified Version under the terms
of this License, in the form shown in the Addendum below.
G.  Preserve in that license notice the full lists of Invariant Sections
and required Cover Texts given in the Document's license notice.
H.  Include an unaltered copy of this License.
I.  Preserve the section entitled "History", and its title, and add to
it an item stating at least the title, year, new authors, and publisher
of the Modified Version as given on the Title Page. If there is no
section entitled "History" in the Document, create one stating the
title, year, authors, and publisher of the Document as given on its
Title Page, then add an item describing the Modified Version as stated
in the previous sentence.
J.  Preserve the network location, if any, given in the Document for
public access to a Transparent copy of the Document, and likewise the
network locations given in the Document for previous versions it was
based on. These may be placed in the "History" section. You may omit a
network location for a work that was published at least four years
before the Document itself, or if the original publisher of the version
it refers to gives permission.
K.  In any section entitled "Acknowledgements" or "Dedications",
preserve the section's title, and preserve in the section all the
substance and tone of each of the contributor acknowledgements and/or
dedications given therein.
L.  Preserve all the Invariant Sections of the Document, unaltered in
their text and in their titles. Section numbers or the equivalent are
not considered part of the section titles.
M.  Delete any section entitled "Endorsements". Such a section may not
be included in the Modified Version.
N.  Do not retitle any existing section as "Endorsements" or to conflict
in title with any Invariant Section.

If the Modified Version includes new front-matter sections or appendices
that qualify as Secondary Sections and contain no material copied from
the Document, you may at your option designate some or all of these
sections as invariant. To do this, add their titles to the list of
Invariant Sections in the Modified Version's license notice. These
titles must be distinct from any other section titles.

You may add a section entitled "Endorsements", provided it contains
nothing but endorsements of your Modified Version by various
parties--for example, statements of peer review or that the text has
been approved by an organization as the authoritative definition of a
standard.

You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list
of Cover Texts in the Modified Version. Only one passage of Front-Cover
Text and one of Back-Cover Text may be added by (or through arrangements
made by) any one entity. If the Document already includes a cover text
for the same cover, previously added by you or by arrangement made by
the same entity you are acting on behalf of, you may not add another;
but you may replace the old one, on explicit permission from the
previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License
give permission to use their names for publicity for or to assert or
imply endorsement of any Modified Version.

[[gfdl07]]
COMBINING DOCUMENTS
-------------------

You may combine the Document with other documents released under this
License, under the terms defined in section 4 above for modified
versions, provided that you include in the combination all of the
Invariant Sections of all of the original documents, unmodified, and
list them all as Invariant Sections of your combined work in its license
notice.

The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy. If there are multiple Invariant Sections with the same name but
different contents, make the title of each such section unique by adding
at the end of it, in parentheses, the name of the original author or
publisher of that section if known, or else a unique number. Make the
same adjustment to the section titles in the list of Invariant Sections
in the license notice of the combined work.

In the combination, you must combine any sections entitled "History" in
the various original documents, forming one section entitled "History";
likewise combine any sections entitled "Acknowledgements", and any
sections entitled "Dedications". You must delete all sections entitled
"Endorsements."

[[gfdl08]]
COLLECTIONS OF DOCUMENTS
------------------------

You may make a collection consisting of the Document and other documents
released under this License, and replace the individual copies of this
License in the various documents with a single copy that is included in
the collection, provided that you follow the rules of this License for
verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute
it individually under this License, provided you insert a copy of this
License into the extracted document, and follow this License in all
other respects regarding verbatim copying of that document.

[[gfdl09]]
AGGREGATION WITH INDEPENDENT WORKS
----------------------------------

A compilation of the Document or its derivatives with other separate and
independent documents or works, in or on a volume of a storage or
distribution medium, does not as a whole count as a Modified Version of
the Document, provided no compilation copyright is claimed for the
compilation. Such a compilation is called an "aggregate", and this
License does not apply to the other self-contained works thus compiled
with the Document, on account of their being thus compiled, if they are
not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies
of the Document, then if the Document is less than one quarter of the
entire aggregate, the Document's Cover Texts may be placed on covers
that surround only the Document within the aggregate. Otherwise they
must appear on covers around the whole aggregate.

[[gfdl10]]
TRANSLATION
-----------

Translation is considered a kind of modification, so you may distribute
translations of the Document under the terms of section 4. Replacing
Invariant Sections with translations requires special permission from
their copyright holders, but you may include translations of some or all
Invariant Sections in addition to the original versions of these
Invariant Sections. You may include a translation of this License
provided that you also include the original English version of this
License. In case of a disagreement between the translation and the
original English version of this License, the original English version
will prevail.

[[gfdl11]]
TERMINATION
-----------

You may not copy, modify, sublicense, or distribute the Document except
as expressly provided for under this License. Any other attempt to copy,
modify, sublicense or distribute the Document is void, and will
automatically terminate your rights under this License. However, parties
who have received copies, or rights, from you under this License will
not have their licenses terminated so long as such parties remain in
full compliance.

[[gfdl12]]
FUTURE REVISIONS OF THIS LICENSE
--------------------------------

The Free Software Foundation may publish new, revised versions of the
GNU Free Documentation License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If
the Document specifies that a particular numbered version of this
License "or any later version" applies to it, you have the option of
following the terms and conditions either of that specified version or
of any later version that has been published (not as a draft) by the
Free Software Foundation. If the Document does not specify a version
number of this License, you may choose any version ever published (not
as a draft) by the Free Software Foundation.

[[gfdl13]]
How to use this License for your documents
------------------------------------------

To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and license
notices just after the title page:

________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute
and/or modify this document under the terms of the GNU Free
Documentation License, Version 1.1 or any later version published by the
Free Software Foundation; with the Invariant Sections being LIST THEIR
TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover
Texts being LIST. A copy of the license is included in the section
entitled "GNU Free Documentation License".
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

If you have no Invariant Sections, write "with no Invariant Sections"
instead of saying which ones are invariant. If you have no Front-Cover
Texts, write "no Front-Cover Texts" instead of "Front-Cover Texts being
LIST"; likewise for Back-Cover Texts.

If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of free
software license, such as the GNU General Public License, to permit
their use in free software.
