# Fundamental Backup Techniques

Accidents will happen sooner or later. In this chapter, we'll discuss how to get data to a safe place using other hosts, floppy disks, CD-ROMs and tapes. We will also discuss the most popular compressing and archiving commands.

Upon completion of this chapter, you will know how to:

- Make, query and unpack file archives

- Handle floppy disks and make a boot disk for your system

- Write CD-ROMs

- Make incremental backups

- Create Java archives

- Find documentation to use other backup devices and programs

- Encrypt your data

## Introduction

Although Linux is one of the safest operating systems in existence, and even if it is designed to keep on going, data can get lost. Data loss is most often the consequence of user errors, but occasionally a system fault, such as a power or disk failure, is the cause, so it's always a good idea to keep an extra copy of sensitive and/or important data.

### Preparing your data

#### Archiving with tar

In most cases, we will first collect all the data to back up in a single archive file, which we will compress later on. The process of archiving
involves concatenating all listed files and taking out unnecessary blanks. In Linux, this is commonly done with the `tar
` command. `tar` was originally designed to archive
data on tapes, but it can also make archives, known as tarballs
.

`tar` has many options
, the most important ones are cited below:

- `-v`: verbose

- `-t`: test, shows content of a tarball

- `-x`: extract archive

- `-c`: create archive

- `-f` `archivedevice`: use `archivedevice` as source/destination for the tarball, the device defaults to the first tape device (usually `/dev/st0` or something similar)

- `-j`: filter through `bzip2`, see

It is common to leave out the dash-prefix with `tar` options, as you can see from the examples below.

##### Use GNU tar for compatibility

The archives
made with a proprietary `tar` version on one system, may be incompatible with `tar` on another proprietary system. This may cause much headaches, such as if the archive needs to be recovered on a system that doesn't exist anymore. Use the GNU `tar` version on all systems to prevent your system admin from bursting into tears. Linux always uses GNU tar. When working on other UNIX machines, enter `tar `--help`` to find out which version you are using. Contact your system admin if you don't see the word GNU somewhere.

In the example below, an archive is created
and unpacked.

    gaby:~> ls images/
    me+tux.jpg  nimf.jpg

    gaby:~> tar cvf images-in-a-dir.tar images/
    images/
    images/nimf.jpg
    images/me+tux.jpg

    gaby:~> cd images

    gaby:~/images> tar cvf images-without-a-dir.tar *.jpg
    me+tux.jpg
    nimf.jpg

    gaby:~/images> cd

    gaby:~> ls */*.tar
    images/images-without-a-dir.tar

    gaby:~> ls *.tar
    images-in-a-dir.tar

    gaby:~> tar xvf images-in-a-dir.tar
    images/
    images/nimf.jpg
    images/me+tux.jpg

    gaby:~> tar tvf images/images-without-dir.tar
    -rw-r--r-- gaby/gaby  42888 1999-06-30 20:52:25 me+tux.jpg
    -rw-r--r-- gaby/gaby   7578 2000-01-26 12:58:46 nimf.jpg

    gaby:~> tar xvf images/images-without-a-dir.tar
    me+tux.jpg
    nimf.jpg

    gaby:~> ls *.jpg
    me+tux.jpg  nimf.jpg

This example also illustrates the difference between a tarred directory and a bunch of tarred files. It is advisable to only compress directories, so files don't get spread all over when unpacking the tarball (which may be on another system, where you may not know which files were already there and which are the ones from the archive).

When a tape drive is connected to your machine and configured by your system administrator, the file names ending in `.tar` are replaced with the tape
device name, for example:

`tar `cvf` `/dev/tape` `mail/``

The directory `mail` and all the files it contains are compressed into a file that is written on the tape immediately. A content listing is displayed because we used the verbose option.

#### Incremental backups with tar

The `tar` tool supports the creation of incremental
backups, using the `-N` option. With this option, you can specify a date, and `tar` will check modification time of all specified files against this date. If files are changed more recent than date, they will be included in the backup. The example below uses the timestamp on a previous archive as the date value. First, the initial archive is created and the timestamp on the initial backup file is shown. Then a new file is created, upon which we take a new backup, containing only this new file:

    jimmy:~> tar cvpf /var/tmp/javaproggies.tar java/*.java
    java/btw.java
    java/error.java
    java/hello.java
    java/income2.java
    java/income.java
    java/inputdevice.java
    java/input.java
    java/master.java
    java/method1.java
    java/mood.java
    java/moodywaitress.java
    java/test3.java
    java/TestOne.java
    java/TestTwo.java
    java/Vehicle.java

    jimmy:~> ls -l /var/tmp/javaproggies.tar
    -rw-rw-r-- 1 jimmy   jimmy   10240 Jan 21 11:58 /var/tmp/javaproggies.tar

    jimmy:~> touch java/newprog.java

    jimmy:~> tar -N /var/tmp/javaproggies.tar \
    -cvp /var/tmp/incremental1-javaproggies.tar java/*.java 2> /dev/null
    java/newprog.java

    jimmy:~> cd /var/tmp/

    jimmy:~> tar xvf incremental1-javaproggies.tar
    java/newprog.java

Standard errors are redirected to `/dev/null`. If you don't do this, `tar` will print a message for each unchanged file, telling you it won't be dumped.

This way of working has the disadvantage that it looks at timestamps on files. Say that you download an archive into the directory containing your backups, and the archive contains files that have been created two years ago. When checking the timestamps of those files against the timestamp on the initial archive, the new files will actually seem old to `tar`, and will not be included in an incremental backup made using the `-N` option.

A better choice would be the `-g` option, which will create a list of files to backup. When making incremental backups, files are checked against this list. This is how it works:

    jimmy:~> tar cvpf work-20030121.tar -g snapshot-20030121 work/
    work/
    work/file1
    work/file2
    work/file3

    jimmy:~> file snapshot-20030121
    snapshot-20030121: ASCII text

The next day, user jimmy works on `file3` a bit more, and creates `file4`. At the end of the day, he makes a new backup:

    jimmy:~> tar cvpf work-20030122.tar -g snapshot-20030121 work/
    work/
    work/file3
    work/file4

These are some very simple examples, but you could also use this kind of command in a cronjob (see ), which specifies for instance a snapshot file for the weekly backup and one for the daily backup. Snapshot files should be replaced when taking full backups, in that case.

More information can be found in the `tar` documentation.

##### The real stuff

As you could probably notice, `tar` is OK when we are talking about a simple directory, a set of files that belongs together. There are tools that are easier to manage, however, when you want to archive entire partitions or disks or larger projects. We just explain about `tar` here because it is a very popular tool for distributing archives. It will happen quite often that you need to install a software that comes in a so-called compressed tarball. See for an easier way to perform regular backups.

#### Compressing and unpacking with `gzip` or `bzip2`

Data, including tarballs, can be compressed
using zip tools. The `gzip
` command will add the suffix .gz to the file name and remove the original file.

    jimmy:~> ls -la | grep tar
    -rw-rw-r-- 1 jimmy  jimmy    61440 Jun  6 14:08 images-without-dir.tar

    jimmy:~> gzip images-without-dir.tar

    jimmy:~> ls -la images-without-dir.tar.gz
    -rw-rw-r-- 1 jimmy  jimmy    50562 Jun  6 14:08 images-without-dir.tar.gz

Uncompress gzipped files with the `-d` option
.

`bzip2
` works in a similar way, but uses an improved compression algorithm, thus creating smaller files. See the `bzip2` info pages for more.

Linux software packages are often distributed in a gzipped tarball. The sensible thing to do after unpacking that kind of archives is find the `README
` and read it. It will generally contain guidelines to installing the package.

The GNU `tar` command is aware of gzipped
files. Use the command

    tar zxvf file.tar.gz

for unzipping and untarring `.tar.gz` or `.tgz` files. Use

    tar jxvf file.tar.bz2

for unpacking `tar` archives that were compressed with `bzip2`.

#### Java archives

The GNU project provides us with the `jar
` tool for creating Java archives. It is a Java application that combines multiple files into a single JAR
archive file. While also being a general purpose archiving and compression tool, based on ZIP and the ZLIB compression format, `jar` was mainly designed to facilitate the packing of Java code, applets and/or applications in a single file. When combined in a single archive, the components of a Java application, can be downloaded much faster.

Unlike `tar`, `jar` compresses by default, independent from other tools - because it is basically the Java version of `zip`. In addition, it allows individual entries in an archive to be signed by the author, so that origins can be authenticated.

The syntax is almost identical as for the `tar` command, we refer to `info `jar`` for specific differences.

##### tar, jar and symbolic links

One noteworthy feature not really mentioned in the standard documentation is that `jar` will follow symbolic links. Data to which these links are pointing will be included in the archive. The default in `tar` is to only backup the symbolic link, but this behavior can be changed using the `-h` to `tar`.

#### Transporting your data

Saving copies of your data on another host is a simple but accurate way of making backups. See for more information on `scp`, `ftp` and more.

In the next section we'll discuss local backup devices.

## Moving your data to a backup device

### Making a copy on a floppy disk

#### Formatting the floppy

On most Linux systems, users have access to the floppy disk
device. The name of the device may vary depending on the size and number of floppy drives, contact your system admin if you are unsure. On some systems, there will likely be a link `/dev/floppy` pointing to the right device, probably `/dev/fd0` (the auto-detecting floppy device) or `/dev/fd0H1440` (set for 1,44MB floppies).

`fdformat
` is the low-level floppy disk formatting tool. It has the device name of the floppy disk as an option. `fdformat` will display an error when the floppy is write-protected.

    emma:~> fdformat /dev/fd0H1440
    Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
    Formatting ... done
    Verifying ... done
    emma:~>

The `mformat
` command (from the mtools package) is used to create DOS-compatible floppies which can then be accessed using the `mcopy`, `mdir` and other m-commands.

Graphical tools are also available.

##### Floppy formatter

This GUI allows for chosing the FS type and density.

After the floppy is formatted, it can be mounted into the file system and accessed as a normal, be it small, directory, usually via the `/mnt/floppy` entry.

Should you need it, install the `mkbootdisk
` utility, which makes a floppy from which the current system can boot.

#### Using the `dd` command to dump data

The `dd
` command can be used to put data on a disk, or get it off again, depending on the given input and output devices. An example
:

    gaby:~> dd if=images-without-dir.tar.gz of=/dev/fd0H1440
    98+1 records in
    98+1 records out

    gaby~> dd if=/dev/fd0H1440 of=/var/tmp/images.tar.gz
    2880+0 records in
    2880+0 records out

    gaby:~> ls /var/tmp/images*
    /var/tmp/images.tar.gz

Note that the dumping is done on an unmounted device. Floppies created using this method will not be mountable in the file system, but it is of course the way to go for creating boot or rescue disks. For more information on the possibilities of `dd`, read the man pages.

This tool is part of the GNU coreutils package.

##### Dumping disks

The `dd` command can also be used to make a raw dump of an entire hard disk.

### Making a copy with a CD-writer

On some systems users are allowed to use the CD-writer
device. Your data will need to be formatted first. Use the `mkisofs
` command to do this in the directory containing the files you want to backup. Check with `df
` that enough disk space is available, because a new file
about the same size as the entire current directory will be created:

    [rose@blob recordables] df -h .
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/hde5              19G   15G  3.2G  82% /home

    [rose@blob recordables] du -h -s .
    325M    .

    [rose@blob recordables] mkisofs -J -r -o cd.iso .
    <--snap-->
    making a lot of conversions
    <--/snap-->
    98.95% done, estimate finish Fri Apr  5 13:54:25 2002
    Total translation table size: 0
    Total rockridge attributes bytes: 35971
    Total directory bytes: 94208
    Path table size(bytes): 452
    Max brk space used 37e84
    166768 extents written (325 Mb)

The `-J` and `-r` options are used to make the CD-ROM mountable on different systems, see the man pages for more. After
that, the CD
can be created using the `cdrecord
` tool with appropriate options:

    [rose@blob recordables] cdrecord -dev 0,0,0 -speed=8 cd.iso
    Cdrecord 1.10 (i686-pc-linux-gnu) (C) 1995-2001 Joerg Schilling
    scsidev: '0,0,0'
    scsibus: 0 target: 0 lun: 0
    Linux sg driver version: 3.1.20
    Using libscg version 'schily-0.5'
    Device type    : Removable CD-ROM
    Version        : 0
    Response Format: 1
    Vendor_info    : 'HP      '
    Identification : 'CD-Writer+ 8100 '
    Revision       : '1.0g'
    Device seems to be: Generic mmc CD-RW.
    Using generic SCSI-3/mmc CD-R driver (mmc_cdr).
    Driver flags   : SWABAUDIO
    Starting to write CD/DVD at speed 4 in write mode for single session.
    Last chance to quit, starting real write in 0 seconds.
    Operation starts.

Depending on your CD-writer, you now have the time to smoke^H^H^H^H^H eat a healthy piece of fruit and/or get a cup of coffee. Upon finishing the job, you will get a confirmation message:

    Track 01: Total bytes read/written: 341540864/341540864
              (166768 sectors).

There are some graphical tools available to make it easier on you. One of the popular ones is `xcdroast
`, which is freely available from the X-CD-Roast web site
and is included on most systems and in the GNU directory. Both the KDE and Gnome desktop managers have facilities to make your own CDs.

### Backups on/from jazz drives, USB devices and other removables

These devices are usually mounted into the file system. After the mount procedure, they are accessed as normal directories, so you can use the standard commands for manipulating files.

In the example below, images are copied from a USB
camera to the hard disk:

    robin:~> mount /mnt/camera

    robin:~> mount | grep camera
    /dev/sda1 on /mnt/camera type vfat (rw,nosuid,nodev)

If the camera
is the only USB storage device that you ever connect to your system, this is safe. But keep in mind that USB devices are assigned entries in `/dev` as they are connected to the system. Thus, if you first connect a USB stick to your system, it will be on the `/dev/sda` entry, and if you connect your camera after that, it will be assigned to `/dev/sdb` - provided that you do not have any SCSI disks, which are also on `/dev/sd*`. On newer systems, since kernel 2.6, a hotplug system called HAL (Hardware Abstraction Layer) ensures that users don't have to deal with this burden. If you want to check where your device is, type `dmesg
` after inserting
it.

You can now copy the files:

    robin:~> cp -R /mnt/camera/* images/

    robin:~> umount /mnt/camera

Likewise, a jazz drive may be mounted on `/mnt/jazz`.

Appropriate lines should be added in `/etc/modules.conf
` and `/etc/fstab
` to make this work. Refer to specific hardware HOWTOs for more information. On systems with a 2.6.x kernel or higher, you may also want to check the man pages for `modprobe` and `modprobe.conf`.

### Backing up data using a tape device

This is done using `tar` (see above). The `mt
` tool is used for controlling the magnetic tape
device, like `/dev/st0`. Entire books have been written about tape backup, therefore, refer to our reading-list in for more information. Keep in mind that databases might need other backup procedures because of their architecture.

The appropriate backup commands are usually put in one of the cron directories in order to have them executed on a regular basis. In larger environments, the freely available Amanda backup suite or a commercial solution may be implemented to back up multiple machines. Working with tapes, however, is a system administration task beyond the scope of this document.

### Tools from your distribution

Most Linux distributions offer their own tools
for making your life easy. A short overview:

- SuSE: YaST now includes expanded backup and restore modules.

- RedHat: the File Roller tool provides visual management of (compressed) archives. They seem to be in favour of the X-CD-Roast tool for moving backups to an external device.

- Mandrake: X-CD-Roast.

- Most distributions come with the BSD `dump
  ` and `restore
  ` utilities for making backups of ext2 and ext3 file systems. This tool can write to a variety of devices and literally dumps the file(s) or file system bit per bit onto the specified device. Like `dd`, this allows for backing up special file types such as the ones in `/dev`.

## Using rsync

### Introduction

The `rsync
` program is a fast and flexible tool for remote backup
. It is common on UNIX and UNIX-like systems, easy to configure and use in scripts. While the r in `rsync` stands for remote, you do not need to take this all too literally. Your remote device might just as well be a USB storage device or another partition on your hard disk, you do not need to have two separated machines.

### An example: rsync to a USB storage device

As discussed in , we will first have to mount
the device. Possibly, this should be done as root:

    root@theserver# mkdir /mnt/usbstore

    root@theserver# mount -t vfat /dev/sda1 /mnt/usbstore

#### Userfriendly

More and more distributions give access to removable devices for non-prilileged users and mount USB devices, CD-ROMs and other removable devices automatically.

Note that this guideline requires USB support to be installed on your system. See the USB Guide for help if this does not work. Check with `dmesg` that `/dev/sda1` is indeed the device to mount.

Then you can start the actual backup, for instance of the `/home/karl` directory:

    karl@theserver:~> rsync -avz /home/karl/ /mnt/usbstore

As usual, refer to the man pages for more.

## Encryption

### General remarks

#### Why should you encrypt data?

Encryption
is synonym to secrecy. In the context of backups, encryption can be very useful, for instance if you need to leave your backed up data in a place where you can not control access, such as the server of your provider.

Apart from that, encryption can be applied to E-mails as well: normally, mail is not encrypted and it is often sent in the open over the netwerk or the Internet. If your message contains sensitive information, better encrypt it.

#### GNU Privacy Guard

On Linux systems you will find GnuPG, the GNU Privacy Guard
, which is a suite of programs that are compatible with the PGP (Pretty Good Privacy) tools that are commercially available.

In this guide we will only discuss the very simple usage of the encryption tools and show what you will need in order to generate an encryption key and use it to encrypt data for yourself, which you can then safely store in a public place. More advanced usage directions can be found in the man pages of the various commands.

### Generate a key

Before you can start encrypting your data, you need to create
a pair of keys. The pair consists of a private and a public key. You can send the public key to correspondents, who can use it to encrypt data for you, which you decrypt with your private key. You always keep the private key, never share it with somebody else, or they will be able to decrypt data that is only destined for you. Just to make sure that no accidents happen, the private key is protected with a password. The key pair is created using this command
:

    willy@ubuntu:~$ gpg --key-gen
    gpg (GnuPG) 1.4.2.2; Copyright (C) 2005 Free Software Foundation, Inc.
    This program comes with ABSOLUTELY NO WARRANTY.
    This is free software, and you are welcome to redistribute it
    under certain conditions.  See the file COPYING for details.

    gpg: directory `/home/willy.gnupg' created
    gpg: new configuration file `/home/willy/.gnupg/gpg.conf' created
    gpg: WARNING: options in `/home/willy/.gnupg/gpg.conf' are not yet
     active during this run
    gpg: keyring `/home/willy/.gnupg/secring.gpg' created
    gpg: keyring `/home/willy/.gnupg/pubring.gpg' created
    Please select what kind of key you want:
        (1) DSA and Elgamal (default)
        (2) DSA (sign only)
        (5) RSA (sign only)
    Your selection? 1
    DSA keypair will have 1024 bits.
    ELG-E keys may be between 1024 and 4096 bits long.
    What keysize do you want? (2048) 4096
    Requested keysize is 4096 bits
    Please specify how long the key should be valid.
             0 = key does not expire
          <n>  = key expires in n days
          <n>w = key expires in n weeks
          <n>m = key expires in n month
          <n>y = key expires in n years
    Key is valid for? (0) 0
    Key does not expire at all
    Is this correct? (y/N) y

    You need a user ID to identify your key; the software constructs the
    user ID from the Real Name, Comment and Email Address in this form:
        "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

    Real name: Willy De Wandel
    Email address: wdw@mvg.vl
    Comment: Willem
    You selected this USER-ID:
        "Willy De Wandel (Willem) <wdw@mvg.vl>"

    Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
    You need a Passphrase to protect your secret key.

    Passphrase:

Now enetr your password. This can be a phrase, the longer, the better, the only condition is that you should be able to remember it at all times. For verification, you need to enter the same phrase again.

Now the key pair is generated by a program that spawns random numbers and that is, among other factors, fed with the activity data of the system. So it is a good idea to start some programs now, to move the mouse cursor or to type some random characters in a terminal window. That way, the chances to generate a number that contains lots of different digits will be much bigger and the key will be more difficult to crack.

### About your key

When your key has been created, you will get a message about the fingerprint
. This is a sequence of 40 hexadecimal
numbers, which is so long that it is very, very hard to generate the same key twice, on any computer. You can be rather sure that this is a unique sequence. The short form of this key consists of your name, followed by the last 8 hexadecimal numbers.

You can get information about your key as follows
:

    willy@ubuntu:~$ gpg --list-keys
    /home/willy/.gnupg/pubring.gpg
    ------------------------------
    pub     1024D/BF5C3DBB 2006-08-08
    uid                    Willy De Wandel (Willem) <wdw@mvg.vl>
    sub     4096g/A3449CF7 2006-08-08

The key ID of this key
is BF5C3DBB. You can send your key ID and your name to a key server, so that other people can get this info about you and use it to encrypt data for you. Alternatively, you can send your public key directly to the people who need it. The public part of your key is the long series of numbers that you see when using the `--export` option
to the `gpg` command:

    gpg --export -a

However, as far is this guide is concerned, we assume that you only need your key in order to encrypt and decrypt data for yourself. Read the `gpg` man pages if you want to know more.

### Encrypt data

Now you can encrypt a `.tar` archive
or a compressed archive, prior to saving it to a backup medium or transporting it to the backup server. Use the `gpg` command
like this:

    gpg -e -r (part of) uid archive

The `-e` option tells `gpg` to encrypt, the `-r` option indicates who to encrypt for. Keep in mind that only only the user name(s) following this `-r` option will be able to decrypt the data again. An example:

    willy@ubuntu:~$ gpg -e -r Willy /var/tmp/home-willy-20060808.tar

### Decrypting files

Using the `-d` option, you can decrypt
files that have been encrypted
for you. The data will scroll over your screen, but an encrypted copy will remain on disk. So for file formats other than plain text, you will want to save the decrypted data, so that you can view them with the appropriate program. This is done using the `-o` option to the `gpg` command:

    willy@ubuntu:~$ gpg -d -o /var/tmp/home-willy-decrypt.tar /var/tmp/home-willy-20060808.tar.gpg

    You need a passphrase to unlock the secret key for
    user: "Willy De Wandel (Willem) <wdw@mvg.vl>"
    4096 ELG-E key, ID A3449CF7, created 2006-08-08 (main key ID BF5C3DBB)

    gpg: encrypted with 4096-bit ELG-E key, ID A3449CF7, created 2006-08-08
            "Willy De Wandel (Willem) <wdw@mvg.vl>"

#### No password = no data

If you can not remember your password, the data is lost. Not even the system administrator will be able to decrypt the data. That is why a copy of important keys is sometimes kept in a sealed vault in a bank.

## Summary

Here's a list of the commands involving file backup:

### New commands in chapter 9: Backup

Command | Meaning
--- | ---
`bzip2` | A block-sorting file compressor.
`cdrecord` | Record audio or data Compact Disks from a master.
`dd` | Convert and copy a file
`fdformat` | Low-level formats a floppy disk.
`gpg` | Encrypt and decrypt data.
`gzip` | Compress or expand files.
`mcopy` | Copy MSDOS files to/from UNIX.
`mdir` | Display an MSDOS directory.
`mformat` | Add an MSDOS file system to a low-level formatted floppy disk.
`mkbootdisk` | Creates a stand-alone boot floppy for the running system.
`mount` | Mount a file system (integrate it with the current file system by connecting it to a mount point).
`rsync` | Synchronize directories.
`tar` | Tape archiving utility, also used for making archives on disk instead of on tape.
`umount` | Unmount file systems.

## Exercises

- Make a backup copy of your home directory in `/var/tmp` using the `tar` command. Then further compress the file using `gzip` or `bzip2`. Make it a clean tarred file, one that doesn't make a mess when unpacking.

- Format a floppy and put some files from your home directory on it. Switch floppies with another trainee and recover his/her floppy in your home directory.

- DOS format the floppy. Use the mtools to put and delete files on it.

- What happens to an unformatted floppy when you want to mount it into the file system?

- If you have any USB storage, try to put a file on it.

- Using `rsync`, make a copy of your home directory to another local or remote file system.

- When leaving files on a network server, it's best to encrypt them. Make a `tar` archive of your home directory and encrypt it.