# Repetitive tasks

Upon completion of this chapter, you will be able to

- Use `for`, `while` and `until` loops, and decide which loop fits which occasion.

- Use the `break` and `continue` Bash built-ins.

- Write scripts using the `select` statement.

- Write scripts that take a variable number of arguments.

## The for loop

### How does it work?

The `for` loop is the first of the three shell looping constructs. This loop allows for specification of a list of values. A list of commands is executed for each value in the list.

The syntax for this loop is:

    for NAME [in LIST ]; do COMMANDS; done

If `[in LIST]` is not present, it is replaced with `in `$@`` and `for` executes the `COMMANDS` once for each positional parameter that is set (see and ).

The return status is the exit status of the last command that executes. If no commands are executed because `LIST` does not expand to any items, the return status is zero.

`NAME` can be any variable name, although `i` is used very often. `LIST` can be any list of words, strings or numbers, which can be literal or generated by any command. The `COMMANDS` to execute can also be any operating system commands, script, program or shell statement. The first time through the loop, `NAME` is set to the first item in `LIST`. The second time, its value is set to the second item in the list, and so on. The loop terminates when `NAME` has taken on each of the values from `LIST` and no items are left in `LIST`.

### Examples

#### Using command substitution for specifying LIST items

The first is a command line example, demonstrating the use of a `for` loop that makes a backup copy of each `.xml` file. After issuing the command, it is safe to start working on your sources:

    [carol@octarine ~/articles] ls *.xml
    file1.xml  file2.xml  file3.xml

    [carol@octarine ~/articles] ls *.xml > list

    [carol@octarine ~/articles] for i in `cat list`; do cp "$i" "$i".bak ; done

    [carol@octarine ~/articles] ls *.xml*
    file1.xml  file1.xml.bak  file2.xml  file2.xml.bak  file3.xml  file3.xml.bak

This one lists the files in `/sbin` that are just plain text files, and possibly scripts:

    for i in `ls /sbin`; do file /sbin/$i | grep ASCII; done

#### Using the content of a variable to specify LIST items

The following is a specific application script for converting HTML files, compliant with a certain scheme, to PHP files. The conversion is done by taking out the first 25 and the last 21 lines, replacing these with two PHP tags that provide header and footer lines:

    [carol@octarine ~/html] cat html2php.sh
    #!/bin/bash
    # specific conversion script for my html files to php
    LIST="$(ls *.html)"
    for i in "$LIST"; do
         NEWNAME=$(ls "$i" | sed -e 's/html/php/')
         cat beginfile > "$NEWNAME"
         cat "$i" | sed -e '1,25d' | tac | sed -e '1,21d'| tac >> "$NEWNAME"
         cat endfile >> "$NEWNAME"
    done

Since we don't do a line count here, there is no way of knowing the line number from which to start deleting lines until reaching the end. The problem is solved using `tac`, which reverses the lines in a file.

##### The basename command

Instead of using `sed` to replace the `html` suffix with `php`, it would be cleaner to use the `basename` command. Read the man page for more info.

##### Odd characters

You will run into problems if the list expands to file names containing spaces and other irregular characters. A more ideal construct to obtain the list would be to use the shell's globbing feature, like this:

    for i in $PATHNAME/*; do
    	commands
    done

## The while loop

### What is it?

The `while` construct allows for repetitive execution of a list of commands, as long as the command controlling the `while` loop executes successfully (exit status of zero). The syntax is:

    while CONTROL-COMMAND; do CONSEQUENT-COMMANDS; done

`CONTROL-COMMAND` can be any command(s) that can exit with a success or failure status. The `CONSEQUENT-COMMANDS` can be any program, script or shell construct.

As soon as the `CONTROL-COMMAND` fails, the loop exits. In a script, the command following the `done` statement is executed.

The return status is the exit status of the last `CONSEQUENT-COMMANDS` command, or zero if none was executed.

### Examples

#### Simple example using while

Here is an example for the impatient:

    #!/bin/bash

    # This script opens 4 terminal windows.

    i="0"

    while [ $i -lt 4 ]
    do
    xterm &
    i=$[$i+1]
    done

#### Nested while loops

The example below was written to copy pictures that are made with a webcam to a web directory. Every five minutes a picture is taken. Every hour, a new directory is created, holding the images for that hour. Every day, a new directory is created containing 24 subdirectories. The script runs in the background.

    #!/bin/bash

    # This script copies files from my homedirectory into the webserver directory.
    # (use scp and SSH keys for a remote directory)
    # A new directory is created every hour.

    PICSDIR=/home/carol/pics
    WEBDIR=/var/www/carol/webcam

    while true; do
    	DATE=`date +%Y%m%d`
    	HOUR=`date +%H`
    	mkdir $WEBDIR/"$DATE"

    	while [ $HOUR -ne "00" ]; do
    		DESTDIR=$WEBDIR/"$DATE"/"$HOUR"
    		mkdir "$DESTDIR"
    		mv $PICDIR/*.jpg "$DESTDIR"/
    		sleep 3600
    		HOUR=`date +%H`
    	done
    done

Note the use of the `true` statement. This means: continue execution until we are forcibly interrupted (with `kill` or Ctrl+C).

This small script can be used for simulation testing; it generates files:

    #!/bin/bash

    # This generates a file every 5 minutes

    while true; do
    touch pic-`date +%s`.jpg
    sleep 300
    done

Note the use of the `date` command to generate all kinds of file and directory names. See the man page for more.

##### Use the system

The previous example is for the sake of demonstration. Regular checks can easily be achieved using the system's cron facility. Do not forget to redirect output and errors when using scripts that are executed from your crontab!

#### Using keyboard input to control the while loop

This script can be interrupted by the user when a Ctrl+C sequence is entered:

    #!/bin/bash

    # This script provides wisdom

    FORTUNE=/usr/games/fortune

    while true; do
    echo "On which topic do you want advice?"
    cat << topics
    politics
    startrek
    kernelnewbies
    sports
    bofh-excuses
    magic
    love
    literature
    drugs
    education
    topics

    echo
    echo -n "Make your choice: "
    read topic
    echo
    echo "Free advice on the topic of $topic: "
    echo
    $FORTUNE $topic
    echo

    done

A here document is used to present the user with possible choices. And again, the `true` test repeats the commands from the `CONSEQUENT-COMMANDS` list over and over again.

#### Calculating an average

This script calculates the average of user input, which is tested before it is processed: if input is not within range, a message is printed. If q is pressed, the loop exits:

    #!/bin/bash

    # Calculate the average of a series of numbers.

    SCORE="0"
    AVERAGE="0"
    SUM="0"
    NUM="0"

    while true; do

      echo -n "Enter your score [0-100%] ('q' for quit): "; read SCORE;

      if (("$SCORE" < "0"))  || (("$SCORE" > "100")); then
        echo "Be serious.  Common, try again: "
      elif [ "$SCORE" == "q" ]; then
        echo "Average rating: $AVERAGE%."
        break
      else
        SUM=$[$SUM + $SCORE]
        NUM=$[$NUM + 1]
        AVERAGE=$[$SUM / $NUM]
      fi

    done

    echo "Exiting."

Note how the variables in the last lines are left unquoted in order to do arithmetic.

## The until loop

### What is it?

The `until` loop is very similar to the `while` loop, except that the loop executes until the `TEST-COMMAND` executes successfully. As long as this command fails, the loop continues. The syntax is the same as for the `while` loop:

    until TEST-COMMAND; do CONSEQUENT-COMMANDS; done

The return status is the exit status of the last command executed in the `CONSEQUENT-COMMANDS` list, or zero if none was executed. `TEST-COMMAND` can, again, be any command that can exit with a success or failure status, and `CONSEQUENT-COMMANDS` can be any UNIX command, script or shell construct.

As we already explained previously, the ; may be replaced with one or more newlines wherever it appears.

### Example

An improved `picturesort.sh` script (see ), which tests for available disk space. If not enough disk space is available, remove pictures from the previous months:

    #!/bin/bash

    # This script copies files from my homedirectory into the webserver directory.
    # A new directory is created every hour.
    # If the pics are taking up too much space, the oldest are removed.

    while true; do
    	DISKFUL=$(df -h $WEBDIR | grep -v File | awk '{print $5 }' | cut -d "%" -f1 -)

    	until [ $DISKFUL -ge "90" ]; do

            	DATE=`date +%Y%m%d`
            	HOUR=`date +%H`
            	mkdir $WEBDIR/"$DATE"

            	while [ $HOUR -ne "00" ]; do
                    	DESTDIR=$WEBDIR/"$DATE"/"$HOUR"
                    	mkdir "$DESTDIR"
                    	mv $PICDIR/*.jpg "$DESTDIR"/
                    	sleep 3600
                    	HOUR=`date +%H`
            	done

    	DISKFULL=$(df -h $WEBDIR | grep -v File | awk '{ print $5 }' | cut -d "%" -f1 -)
    	done

    	TOREMOVE=$(find $WEBDIR -type d -a -mtime +30)
    	for i in $TOREMOVE; do
    		rm -rf "$i";
    	done

    done

Note the initialization of the `HOUR` and `DISKFULL` variables and the use of options with `ls` and `date` in order to obtain a correct listing for `TOREMOVE`.

## I/O redirection and loops

### Input redirection

Instead of controlling a loop by testing the result of a command or by user input, you can specify a file from which to read input that controls the loop. In such cases, `read` is often the controlling command. As long as input lines are fed into the loop, execution of the loop commands continues. As soon as all the input lines are read the loop exits.

Since the loop construct is considered to be one command structure (such as `while TEST-COMMAND; do CONSEQUENT-COMMANDS; done`), the redirection should occur after the `done` statement, so that it complies with the form

    command < file

This kind of redirection also works with other kinds of loops.

### Output redirection

In the example below, output of the `find` command is used as input for the `read` command controlling a `while` loop:

    [carol@octarine ~/testdir] cat archiveoldstuff.sh
    #!/bin/bash

    # This script creates a subdirectory in the current directory, to which old
    # files are moved.
    # Might be something for cron (if slightly adapted) to execute weekly or
    # monthly.

    ARCHIVENR=`date +%Y%m%d`
    DESTDIR="$PWD/archive-$ARCHIVENR"

    mkdir "$DESTDIR"

    # using quotes to catch file names containing spaces, using read -d for more
    # fool-proof usage:
    find "$PWD" -type f -a -mtime +5 | while read -d $'\000' file

    do
    gzip "$file"; mv "$file".gz "$DESTDIR"
    echo "$file archived"
    done

Files are compressed before they are moved into the archive directory.

## Break and continue

### The break built-in

The `break` statement is used to exit the current loop before its normal ending. This is done when you don't know in advance how many times the loop will have to execute, for instance because it is dependent on user input.

The example below demonstrates a `while` loop that can be interrupted. This is a slightly improved version of the `wisdom.sh` script from .

    #!/bin/bash

    # This script provides wisdom
    # You can now exit in a decent way.

    FORTUNE=/usr/games/fortune

    while true; do
    echo "On which topic do you want advice?"
    echo "1.  politics"
    echo "2.  startrek"
    echo "3.  kernelnewbies"
    echo "4.  sports"
    echo "5.  bofh-excuses"
    echo "6.  magic"
    echo "7.  love"
    echo "8.  literature"
    echo "9.  drugs"
    echo "10. education"
    echo

    echo -n "Enter your choice, or 0 for exit: "
    read choice
    echo

    case $choice in
         1)
         $FORTUNE politics
         ;;
         2)
         $FORTUNE startrek
         ;;
         3)
         $FORTUNE kernelnewbies
         ;;
         4)
         echo "Sports are a waste of time, energy and money."
         echo "Go back to your keyboard."
         echo -e "\t\t\t\t -- \"Unhealthy is my middle name\" Soggie."
         ;;
         5)
         $FORTUNE bofh-excuses
         ;;
         6)
         $FORTUNE magic
         ;;
         7)
         $FORTUNE love
         ;;
         8)
         $FORTUNE literature
         ;;
         9)
         $FORTUNE drugs
         ;;
         10)
         $FORTUNE education
         ;;
         0)
         echo "OK, see you!"
         break
         ;;
         *)
         echo "That is not a valid choice, try a number from 0 to 10."
         ;;
    esac
    done

Mind that `break` exits the loop, not the script. This can be demonstrated by adding an `echo` command at the end of the script. This `echo` will also be executed upon input that causes `break` to be executed (when the user types 0).

In nested loops, `break` allows for specification of which loop to exit. See the Bash `info` pages for more.

### The continue built-in

The `continue` statement resumes iteration of an enclosing `for`, `while`, `until` or `select` loop.

When used in a `for` loop, the controlling variable takes on the value of the next element in the list. When used in a `while` or `until` construct, on the other hand, execution resumes with `TEST-COMMAND` at the top of the loop.

### Examples

In the following example, file names are converted to lower case. If no conversion needs to be done, a `continue` statement restarts execution of the loop. These commands don't eat much system resources, and most likely, similar problems can be solved using `sed` and `awk`. However, it is useful to know about this kind of construction when executing heavy jobs, that might not even be necessary when tests are inserted at the correct locations in a script, sparing system resources.

    [carol@octarine ~/test] cat tolower.sh
    #!/bin/bash

    # This script converts all file names containing upper case characters into file# names containing only lower cases.

    LIST="$(ls)"

    for name in "$LIST"; do

    if [[ "$name" != *[[:upper:]]* ]]; then
    continue
    fi

    ORIG="$name"
    NEW=`echo $name | tr 'A-Z' 'a-z'`

    mv "$ORIG" "$NEW"
    echo "new name for $ORIG is $NEW"
    done

This script has at least one disadvantage: it overwrites existing files. The `noclobber` option to Bash is only useful when redirection occurs. The `-b` option to the `mv` command provides more security, but is only safe in case of one accidental overwrite, as is demonstrated in this test:

    [carol@octarine ~/test] rm *

    [carol@octarine ~/test] touch test Test TEST

    [carol@octarine ~/test] bash -x tolower.sh
    ++ ls
    + LIST=test
    Test
    TEST
    + [[ test != *[[:upper:]]* ]]
    + continue
    + [[ Test != *[[:upper:]]* ]]
    + ORIG=Test
    ++ echo Test
    ++ tr A-Z a-z
    + NEW=test
    + mv -b Test test
    + echo 'new name for Test is test'
    new name for Test is test
    + [[ TEST != *[[:upper:]]* ]]
    + ORIG=TEST
    ++ echo TEST
    ++ tr A-Z a-z
    + NEW=test
    + mv -b TEST test
    + echo 'new name for TEST is test'
    new name for TEST is test

    [carol@octarine ~/test] ls -a
    ./  ../  test  test~

The `tr` is part of the textutils package; it can perform all kinds of character transformations.

## Making menus with the select built-in

### General

#### Use of select

The `select` construct allows easy menu generation. The syntax is quite similar to that of the `for` loop:

    select WORD [in LIST]; do RESPECTIVE-COMMANDS; done

`LIST` is expanded, generating a list of items. The expansion is printed to standard error; each item is preceded by a number. If `in `LIST`` is not present, the positional parameters are printed, as if `in `$@`` would have been specified. `LIST` is only printed once.

Upon printing all the items, the `PS3` prompt is printed and one line from standard input is read. If this line consists of a number corresponding to one of the items, the value of `WORD` is set to the name of that item. If the line is empty, the items and the `PS3` prompt are displayed again. If an EOF (End Of File) character is read, the loop exits. Since most users don't have a clue which key combination is used for the EOF sequence, it is more user-friendly to have a `break` command as one of the items. Any other value of the read line will set `WORD` to be a null string.

The read line is saved in the `REPLY` variable.

The `RESPECTIVE-COMMANDS` are executed after each selection until the number representing the `break` is read. This exits the loop.

#### Examples

This is a very simple example, but as you can see, it is not very user-friendly:

    [carol@octarine testdir] cat private.sh
    #!/bin/bash

    echo "This script can make any of the files in this directory private."
    echo "Enter the number of the file you want to protect:"

    select FILENAME in *;
    do
         echo "You picked $FILENAME ($REPLY), it is now only accessible to you."
         chmod go-rwx "$FILENAME"
    done

    [carol@octarine testdir] ./private.sh
    This script can make any of the files in this directory private.
    Enter the number of the file you want to protect:
    1) archive-20030129
    2) bash
    3) private.sh
    #? 1
    You picked archive-20030129 (1)
    #?

Setting the `PS3` prompt and adding a possibility to quit makes it better:

    #!/bin/bash

    echo "This script can make any of the files in this directory private."
    echo "Enter the number of the file you want to protect:"

    PS3="Your choice: "
    QUIT="QUIT THIS PROGRAM - I feel safe now."
    touch "$QUIT"

    select FILENAME in *;
    do
      case $FILENAME in
            "$QUIT")
              echo "Exiting."
              break
              ;;
            *)
              echo "You picked $FILENAME ($REPLY)"
              chmod go-rwx "$FILENAME"
              ;;
      esac
    done
    rm "$QUIT"

### Submenus

Any statement within a `select` construct can be another `select` loop, enabling (a) submenu(s) within a menu.

By default, the `PS3` variable is not changed when entering a nested `select` loop. If you want a different prompt in the submenu, be sure to set it at the appropriate time(s).

## The shift built-in

### What does it do?

The `shift` command is one of the Bourne shell built-ins that comes with Bash. This command takes one argument, a number. The positional parameters are shifted to the left by this number, N. The positional parameters from `N+1` to `$#` are renamed to variable names from `$1` to `$# - N+1`.

Say you have a command that takes 10 arguments, and N is 4, then `$4` becomes `$1`, `$5` becomes `$2` and so on. `$10` becomes `$7` and the original `$1`, `$2` and `$3` are thrown away.

If N is zero or greater than `$#`, the positional parameters are not changed (the total number of arguments, see ) and the command has no effect. If N is not present, it is assumed to be 1. The return status is zero unless N is greater than `$#` or less than zero; otherwise it is non-zero.

### Examples

A shift statement is typically used when the number of arguments to a command is not known in advance, for instance when users can give as many arguments as they like. In such cases, the arguments are usually processed in a `while` loop with a test condition of `(( $# ))`. This condition is true as long as the number of arguments is greater than zero. The `$1` variable and the `shift` statement process each argument. The number of arguments is reduced each time `shift` is executed and eventually becomes zero, upon which the `while` loop exits.

The example below, `cleanup.sh`, uses `shift` statements to process each file in the list generated by `find`:

    #!/bin/bash

    # This script can clean up files that were last accessed over 365 days ago.

    USAGE="Usage: $0 dir1 dir2 dir3 ... dirN"

    if [ "$#" == "0" ]; then
    	echo "$USAGE"
    	exit 1
    fi

    while (( "$#" )); do

    if [[ $(ls "$1") == "" ]]; then
    	echo "Empty directory, nothing to be done."
      else
    	find "$1" -type f -a -atime +365 -exec rm -i {} \;
    fi

    shift

    done

#### -exec vs. xargs

The above `find` command can be replaced with the following:

    find options | xargs [commands_to_execute_on_found_files]

The `xargs` command builds and executes command lines from standard input. This has the advantage that the command line is filled until the system limit is reached. Only then will the command to execute be called, in the above example this would be `rm`. If there are more arguments, a new command line will be used, until that one is full or until there are no more arguments. The same thing using `find `-exec`` calls on the command to execute on the found files every time a file is found. Thus, using `xargs` greatly speeds up your scripts and the performance of your machine.

In the next example, we modified the script from so that it accepts multiple packages to install at once:

    #!/bin/bash
    if [ $# -lt 1 ]; then
            echo "Usage: $0 package(s)"
            exit 1
    fi
    while (($#)); do
    	yum install "$1" << CONFIRM
    y
    CONFIRM
    	shift
    done

## Summary

In this chapter, we discussed how repetitive commands can be incorporated in loop constructs. Most common loops are built using the `for`, `while` or `until` statements, or a combination of these commands. The `for` loop executes a task a defined number of times. If you don't know how many times a command should execute, use either `until` or `while` to specify when the loop should end.

Loops can be interrupted or reiterated using the `break` and `continue` statements.

A file can be used as input for a loop using the input redirection operator, loops can also read output from commands that is fed into the loop using a pipe.

The `select` construct is used for printing menus in interactive scripts. Looping through the command line arguments to a script can be done using the `shift` statement.

## Exercises

Remember: when building scripts, work in steps and test each step before incorporating it in your script.

1. Create a script that will take a (recursive) copy of files in `/etc` so that a beginning system administrator can edit files without fear.

2. Write a script that takes exactly one argument, a directory name. If the number of arguments is more or less than one, print a usage message. If the argument is not a directory, print another message. For the given directory, print the five biggest files and the five files that were most recently modified.

3. Can you explain why it is so important to put the variables in between double quotes in the example from ?

4. Write a script similar to the one in , but think of a way of quitting after the user has executed 3 loops.

5. Think of a better solution than `move `-b`` for the script from to prevent overwriting of existing files. For instance, test whether or not a file exists. Don't do unnecessary work!

6. Rewrite the `whichdaemon.sh` script from , so that it:

   - Prints a list of servers to check, such as Apache, the SSH server, the NTP daemon, a name daemon, a power management daemon, and so on.

   - For each choice the user can make, print some sensible information, like the name of the web server, NTP trace information, and so on.

   - Optionally, build in a possibility for users to check other servers than the ones listed. For such cases, check that at least the given process is running.

   - Review the script from . Note how character input other than q is processed. Rebuild this script so that it prints a message if characters are given as input.