From michael.cox1 at imperial.ac.uk  Fri Oct  1 10:02:09 2010
From: michael.cox1 at imperial.ac.uk (Cox, Michael J)
Date: Fri, 1 Oct 2010 15:02:09 +0100
Subject: [Bio-Linux] Cron and Qiime
Message-ID: <E11406C7CCAD0847AFC5C6100B004FDB2270AFA1DA@ICEXM5.ic.ac.uk>

Hi Tony,

That was the case, I've edited:

backup_filesystems=(${FILESYSTEMS:="/home /var /usr"})

to read:

backup_filesystems=(${FILESYSTEMS:="/home" "/var" "/usr"})

Running as root:

./backup: line 119: backups[@]: unbound variable

which seems to refer to the following section of backup

# dump each directory, one at a time
for item in "${backups[@]}"; do
    basename=`basename $item`

    #sync or swim
    sync
    /sbin/dump -"$level" -"$compression_mode" -n -u -f "$backup_destination/$basename.bak.$level" "$item"
done

so I seem to be creeping closer to a conclusion?

Thanks

Mike


________________________________________


Message: 2
Date: Thu, 30 Sep 2010 15:12:31 +0100
From: Anthony Pemberton <A.J.Pemberton at bham.ac.uk>
Subject: Re: [Bio-Linux] Cron and Qiime
To: Bio-Linux help and discussion <bio-linux at nebclists.nerc.ac.uk>
Message-ID:
        <3A5B0BBDAF00724AB5F10155650102300216806C1D at LESMBX1.adf.bham.ac.uk>
Content-Type: text/plain; charset="us-ascii"

Mike,

OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})

Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.

The way to quickly test the script after editing is to run it from the command line eg.

% sudo -i
# cd /etc/cron.daily
# ./backup

and see what happens. Use of echo statements in shell scripts like this helps to debug,

Regards,

Tony P.


-----Original Message-----
From: bio-linux-bounces at nebclists.nerc.ac.uk [mailto:bio-linux-bounces at nebclists.nerc.ac.uk] On Behalf Of Cox, Michael J
Sent: 30 September 2010 13:03
To: bio-linux at nebclists.nerc.ac.uk
Subject: Re: [Bio-Linux] Cron and Qiime

Thanks so much for your help Anthony,

cron is running:

root      1267     1  0 Sep28 ?        00:00:00 cron

and sudo start cron gave: cron is already running.


Checking cron.daily/backup the backup destination is backup_destination=${DESTINATION:="/backups"}

and checking fstab, this does seem to exist and to be mounted:

/dev/sdb1       /backups        ext4    defaults,noauto 0       2

However cron.daily/backup also lists a config file:

# Source configuration file
CONFIG=/etc/default/backup
if [ -r $CONFIG ]; then
    . $CONFIG
fi

Which doesn't exist in /etc/default/backup - could this be the source of the problem?  mount /backups gives the same empty Lost+found directory.

Thanks to you too Bela,

Helpdesk it is, non-virtualbox qiime looks to be a slippery customer :)

Cheers

Mike


From A.J.Pemberton at bham.ac.uk  Fri Oct  1 11:40:19 2010
From: A.J.Pemberton at bham.ac.uk (Anthony Pemberton)
Date: Fri, 1 Oct 2010 16:40:19 +0100
Subject: [Bio-Linux] Cron and Qiime
In-Reply-To: <E11406C7CCAD0847AFC5C6100B004FDB2270AFA1DA@ICEXM5.ic.ac.uk>
References: <E11406C7CCAD0847AFC5C6100B004FDB2270AFA1DA@ICEXM5.ic.ac.uk>
Message-ID: <3A5B0BBDAF00724AB5F10155650102300216806C7C@LESMBX1.adf.bham.ac.uk>

Mike,

Yes - now there should be some lines above the "# dump each directory, one at a time" section starting with

#DEBUG

Uncomment the echo and exit lines - this should tell you what variables have been set by the script. You should get something like:

/dev/sda3 /dev/sda4 /dev/sda5 9 z

- the numbers after the /dev/sd might be different, but you should see the device names for the 3 filesystems you intend to backup (dump). The next variable is the dump level (e.g. 9) then the compression variable which should be set to z (use zlib)

What output do you get from the script after these edits?

Regards,

Tony P.



-----Original Message-----
From: bio-linux-bounces at nebclists.nerc.ac.uk [mailto:bio-linux-bounces at nebclists.nerc.ac.uk] On Behalf Of Cox, Michael J
Sent: 01 October 2010 15:02
To: bio-linux at nebclists.nerc.ac.uk
Subject: Re: [Bio-Linux] Cron and Qiime

Hi Tony,

That was the case, I've edited:

backup_filesystems=(${FILESYSTEMS:="/home /var /usr"})

to read:

backup_filesystems=(${FILESYSTEMS:="/home" "/var" "/usr"})

Running as root:

./backup: line 119: backups[@]: unbound variable

which seems to refer to the following section of backup

# dump each directory, one at a time
for item in "${backups[@]}"; do
    basename=`basename $item`

    #sync or swim
    sync
    /sbin/dump -"$level" -"$compression_mode" -n -u -f "$backup_destination/$basename.bak.$level" "$item"
done

so I seem to be creeping closer to a conclusion?

Thanks

Mike


________________________________________


Message: 2
Date: Thu, 30 Sep 2010 15:12:31 +0100
From: Anthony Pemberton <A.J.Pemberton at bham.ac.uk>
Subject: Re: [Bio-Linux] Cron and Qiime
To: Bio-Linux help and discussion <bio-linux at nebclists.nerc.ac.uk>
Message-ID:
        <3A5B0BBDAF00724AB5F10155650102300216806C1D at LESMBX1.adf.bham.ac.uk>
Content-Type: text/plain; charset="us-ascii"

Mike,

OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})

Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.

The way to quickly test the script after editing is to run it from the command line eg.

% sudo -i
# cd /etc/cron.daily
# ./backup

and see what happens. Use of echo statements in shell scripts like this helps to debug,

Regards,

Tony P.


-----Original Message-----
From: bio-linux-bounces at nebclists.nerc.ac.uk [mailto:bio-linux-bounces at nebclists.nerc.ac.uk] On Behalf Of Cox, Michael J
Sent: 30 September 2010 13:03
To: bio-linux at nebclists.nerc.ac.uk
Subject: Re: [Bio-Linux] Cron and Qiime

Thanks so much for your help Anthony,

cron is running:

root      1267     1  0 Sep28 ?        00:00:00 cron

and sudo start cron gave: cron is already running.


Checking cron.daily/backup the backup destination is backup_destination=${DESTINATION:="/backups"}

and checking fstab, this does seem to exist and to be mounted:

/dev/sdb1       /backups        ext4    defaults,noauto 0       2

However cron.daily/backup also lists a config file:

# Source configuration file
CONFIG=/etc/default/backup
if [ -r $CONFIG ]; then
    . $CONFIG
fi

Which doesn't exist in /etc/default/backup - could this be the source of the problem?  mount /backups gives the same empty Lost+found directory.

Thanks to you too Bela,

Helpdesk it is, non-virtualbox qiime looks to be a slippery customer :)

Cheers

Mike
_______________________________________________
Bio-Linux mailing list
Bio-Linux at nebclists.nerc.ac.uk
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux


From a.travis at abdn.ac.uk  Sun Oct  3 11:31:49 2010
From: a.travis at abdn.ac.uk (Tony Travis)
Date: Sun, 03 Oct 2010 16:31:49 +0100
Subject: [Bio-Linux] Cron and Qiime
In-Reply-To: <3A5B0BBDAF00724AB5F10155650102300216806C1D@LESMBX1.adf.bham.ac.uk>
References: <E11406C7CCAD0847AFC5C6100B004FDB2270AFA1CB@ICEXM5.ic.ac.uk>
	<3A5B0BBDAF00724AB5F10155650102300216806C1D@LESMBX1.adf.bham.ac.uk>
Message-ID: <4CA8A1E5.6040006@abdn.ac.uk>

On 30/09/10 15:12, Anthony Pemberton wrote:
> Mike,
>
> OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
> #What to backup
> backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})
>
> Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.
>
> The way to quickly test the script after editing is to run it from the command line eg.
>
> % sudo -i
> # cd /etc/cron.daily
> # ./backup
>
> and see what happens. Use of echo statements in shell scripts like this helps to debug,

Hello, Tony.

Tim added my changes to bio-linux-backups, but I have also noticed 
problems if an "/etc/default/backup" file is absent. Sorry, I should 
have posted something to the list.

I use these defaults (from Bio-Linux 5) for our NBX's:

> # @(#)/etc/default/backup  2009-07-17  A.J.Travis
>
> #
> # Bio-Linux 5.0 backup configuration
> #
>
> # Where to backup
> #DESTINATION="/backups"
>
> # What to backup
> #FILESYSTEMS="/ /home /var /usr"
> FILESYSTEMS="/ /home"
>
> # Compression scheme.  For large drives bzip2 is probably too slow.
> #COMPRESSION="z"

HTH,

   Tony.
-- 
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt


From michael.cox1 at imperial.ac.uk  Mon Oct  4 09:34:46 2010
From: michael.cox1 at imperial.ac.uk (Cox, Michael J)
Date: Mon, 4 Oct 2010 14:34:46 +0100
Subject: [Bio-Linux] Cron and Qiime
In-Reply-To: <mailman.1.1286190002.14964.bio-linux@nebclists.nerc.ac.uk>
References: <mailman.1.1286190002.14964.bio-linux@nebclists.nerc.ac.uk>
Message-ID: <1C3F9504-E9B5-45A8-A8AE-036B63C32C2E@imperial.ac.uk>

Hi Tony P,

Sorry, I did uncomment those lines and rerun backup, but it didn't return anything different from before.  I'm pretty sure that I've uncommented the right part of the script (there's only one debug line), but here is a full copy of my backup script as it now stands.  

Tony T, is the answer to create a file at /etc/default/backup with the config details that you list?

Thanks Mike

#!/bin/bash

#Fail on errors!
set -e
set -u

#######################################################################
# /etc/cron.daily/backup - a script to backup one hd to               # 
# another on the same machine, using dump                             #
# (see the dump man page for details, and                             #
# http://envgen.nox.ac.uk/envgen/software/archives/000501.html#auto   #
#                                                                     #
# Written by MIT, 20/6/2002                                           #
# Modified by Dan Swan for Bio-Linux 2.0 17/3/2003                    #
# Updated for Bio-Linux 5 by Tim Booth 1/8/2008                       #
# Updated to perform Quarterly ETOH backups by Tony Travis 17/7/2009  #
# Updated to dump ext4 fileystems by Tony Travis 19/5/2010            #
#######################################################################

# Source configuration file
CONFIG=/etc/default/backup
if [ -r $CONFIG ]; then
    . $CONFIG
fi

#Use basic or ETOH mode?
backup_mode=${MODE:="etoh"}

#Where to backup
backup_destination=${DESTINATION:="/backups"}

#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/var" "/usr"})

#Compression scheme.  For large drives bzip2 is probably too slow.
compression_mode=${COMPRESSION:="z"}

# get the date
day=`date +%a`
month=`date +%b`
dom=`date +%d`

#Dump level can be set on command line or by today's date.
if [ -n "$*" ]; then
    level="$1"
elif [ "$backup_mode" == "etoh" ] ; then

    # Quarterly Enhanced Towers of Hanoi
    if [ $day == "Sun" ]; then
        if [ $dom -le 7 ]; then
            case $month in
            "Jan" | "Apr" | "Jul" | "Oct") level=0 ;;
            *) level=1 ;;
        esac
        elif [ $dom -le 14 ]; then
            level=3
        elif [ $dom -le 21 ]; then
            level=2
        elif [ $dom -le 28 ]; then
            level=4
        elif [ $dom -le 31 ]; then
            level=3
        fi
    else
        case $day in
        "Mon") level=6 ;;
        "Tue") level=5 ;;
        "Wed") level=8 ;;
        "Thu") level=7 ;;
        "Fri") level=9 ;;
        "Sat") level=8 ;;
        esac
    fi
elif [ "$backup_mode" == "basic" ] ; then
    # dump is 9 any day but Sunday
    if [[ $day == "Sun" ]]; then
            level=0
    else
            level=9
    fi
else
    echo "ERROR: Unknown backup mode and no level specified"
    exit 1
fi

# dirs to back up - used to be:
#backups=(hda1 hda2 hda5 hda6)
#But now I need to be a bit more sophisticated.
#List filesystems to back up and use mount to find where they are.
backups=()

for fs in "${backup_filesystems[@]}" ; do
    device=`mount | egrep "[^[:space:]]+ on $fs type ext[234]" | awk '{print $1}'`

    if [ "$device" != "" ]; then
	backups[${#backups[@]}]=$device
    fi
done

#DEBUG
echo ${backups[@]} $level $compression_mode
exit

# if backup drive is already mounted then try to unmount it first.              
# if this is in use the step will fail and exit                                 
/bin/mount | grep -q "/backups" && /bin/umount /backups 2>/dev/null

# mount necessary drive - must be listed
# in /etc/fstab in order to be mounted
/bin/mount "$backup_destination" 2>/dev/null || exit 0

# dump each directory, one at a time
for item in "${backups[@]}"; do
    basename=`basename $item`

    #sync or swim
    sync
    /sbin/dump -"$level" -"$compression_mode" -n -u -f "$backup_destination/$basename.bak.$level" "$item"
done

# umount drive
/bin/umount "$backup_destination" 
 

On 4 Oct 2010, at 12:00, <bio-linux-request at nebclists.nerc.ac.uk> wrote:

Send Bio-Linux mailing list submissions to
bio-linux at nebclists.nerc.ac.uk

To subscribe or unsubscribe via the World Wide Web, visit
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux
or, via email, send a message with subject or body 'help' to
bio-linux-request at nebclists.nerc.ac.uk

You can reach the person managing the list at
bio-linux-owner at nebclists.nerc.ac.uk

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Bio-Linux digest..."


Today's Topics:

  1. Re: Cron and Qiime (Tony Travis)


----------------------------------------------------------------------

Message: 1
Date: Sun, 03 Oct 2010 16:31:49 +0100
From: Tony Travis <a.travis at abdn.ac.uk>
Subject: Re: [Bio-Linux] Cron and Qiime
To: bio-linux at nebclists.nerc.ac.uk
Message-ID: <4CA8A1E5.6040006 at abdn.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 30/09/10 15:12, Anthony Pemberton wrote:
Mike,

OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})

Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.

The way to quickly test the script after editing is to run it from the command line eg.

% sudo -i
# cd /etc/cron.daily
# ./backup

and see what happens. Use of echo statements in shell scripts like this helps to debug,

Hello, Tony.

Tim added my changes to bio-linux-backups, but I have also noticed
problems if an "/etc/default/backup" file is absent. Sorry, I should
have posted something to the list.

I use these defaults (from Bio-Linux 5) for our NBX's:

# @(#)/etc/default/backup  2009-07-17  A.J.Travis

#
# Bio-Linux 5.0 backup configuration
#

# Where to backup
#DESTINATION="/backups"

# What to backup
#FILESYSTEMS="/ /home /var /usr"
FILESYSTEMS="/ /home"

# Compression scheme.  For large drives bzip2 is probably too slow.
#COMPRESSION="z"

HTH,

  Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt


------------------------------

_______________________________________________
Bio-Linux mailing list
Bio-Linux at nebclists.nerc.ac.uk
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux


End of Bio-Linux Digest, Vol 36, Issue 3
****************************************

Mike Cox

Research Associate
Centre for Respiratory Infection
National Heart Lung Institute
Royal Brompton Campus
Guy Scadding Building
London SW3 6LY

michael.cox1 at imperial.ac.uk
http://www1.imperial.ac.uk/medicine/people/michael.cox1/






From A.J.Pemberton at bham.ac.uk  Mon Oct  4 09:42:06 2010
From: A.J.Pemberton at bham.ac.uk (Anthony Pemberton)
Date: Mon, 4 Oct 2010 14:42:06 +0100
Subject: [Bio-Linux] Cron and Qiime
In-Reply-To: <1C3F9504-E9B5-45A8-A8AE-036B63C32C2E@imperial.ac.uk>
References: <mailman.1.1286190002.14964.bio-linux@nebclists.nerc.ac.uk>
	<1C3F9504-E9B5-45A8-A8AE-036B63C32C2E@imperial.ac.uk>
Message-ID: <3A5B0BBDAF00724AB5F101556501023002192C3698@LESMBX1.adf.bham.ac.uk>

Mike,

Yes, I think the answer will be to put in the defaults file that Tony Travis has provided. I know I did get this working without a defaults file but I probably did some other edits that took-away the need for the defaults file. Put the defaults file as Tony T. has suggested on your machine and give it a try. You can leave the DEBUG echo for now to see what happens,

Regards,

Tony P.
 

-----Original Message-----
From: bio-linux-bounces at nebclists.nerc.ac.uk [mailto:bio-linux-bounces at nebclists.nerc.ac.uk] On Behalf Of Cox, Michael J
Sent: 04 October 2010 14:35
To: bio-linux at nebclists.nerc.ac.uk
Subject: Re: [Bio-Linux] Cron and Qiime

Hi Tony P,

Sorry, I did uncomment those lines and rerun backup, but it didn't return anything different from before.  I'm pretty sure that I've uncommented the right part of the script (there's only one debug line), but here is a full copy of my backup script as it now stands.  

Tony T, is the answer to create a file at /etc/default/backup with the config details that you list?

Thanks Mike

#!/bin/bash

#Fail on errors!
set -e
set -u

#######################################################################
# /etc/cron.daily/backup - a script to backup one hd to               # 
# another on the same machine, using dump                             #
# (see the dump man page for details, and                             #
# http://envgen.nox.ac.uk/envgen/software/archives/000501.html#auto   #
#                                                                     #
# Written by MIT, 20/6/2002                                           #
# Modified by Dan Swan for Bio-Linux 2.0 17/3/2003                    #
# Updated for Bio-Linux 5 by Tim Booth 1/8/2008                       #
# Updated to perform Quarterly ETOH backups by Tony Travis 17/7/2009  #
# Updated to dump ext4 fileystems by Tony Travis 19/5/2010            #
#######################################################################

# Source configuration file
CONFIG=/etc/default/backup
if [ -r $CONFIG ]; then
    . $CONFIG
fi

#Use basic or ETOH mode?
backup_mode=${MODE:="etoh"}

#Where to backup
backup_destination=${DESTINATION:="/backups"}

#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/var" "/usr"})

#Compression scheme.  For large drives bzip2 is probably too slow.
compression_mode=${COMPRESSION:="z"}

# get the date
day=`date +%a`
month=`date +%b`
dom=`date +%d`

#Dump level can be set on command line or by today's date.
if [ -n "$*" ]; then
    level="$1"
elif [ "$backup_mode" == "etoh" ] ; then

    # Quarterly Enhanced Towers of Hanoi
    if [ $day == "Sun" ]; then
        if [ $dom -le 7 ]; then
            case $month in
            "Jan" | "Apr" | "Jul" | "Oct") level=0 ;;
            *) level=1 ;;
        esac
        elif [ $dom -le 14 ]; then
            level=3
        elif [ $dom -le 21 ]; then
            level=2
        elif [ $dom -le 28 ]; then
            level=4
        elif [ $dom -le 31 ]; then
            level=3
        fi
    else
        case $day in
        "Mon") level=6 ;;
        "Tue") level=5 ;;
        "Wed") level=8 ;;
        "Thu") level=7 ;;
        "Fri") level=9 ;;
        "Sat") level=8 ;;
        esac
    fi
elif [ "$backup_mode" == "basic" ] ; then
    # dump is 9 any day but Sunday
    if [[ $day == "Sun" ]]; then
            level=0
    else
            level=9
    fi
else
    echo "ERROR: Unknown backup mode and no level specified"
    exit 1
fi

# dirs to back up - used to be:
#backups=(hda1 hda2 hda5 hda6)
#But now I need to be a bit more sophisticated.
#List filesystems to back up and use mount to find where they are.
backups=()

for fs in "${backup_filesystems[@]}" ; do
    device=`mount | egrep "[^[:space:]]+ on $fs type ext[234]" | awk '{print $1}'`

    if [ "$device" != "" ]; then
	backups[${#backups[@]}]=$device
    fi
done

#DEBUG
echo ${backups[@]} $level $compression_mode
exit

# if backup drive is already mounted then try to unmount it first.              
# if this is in use the step will fail and exit                                 
/bin/mount | grep -q "/backups" && /bin/umount /backups 2>/dev/null

# mount necessary drive - must be listed
# in /etc/fstab in order to be mounted
/bin/mount "$backup_destination" 2>/dev/null || exit 0

# dump each directory, one at a time
for item in "${backups[@]}"; do
    basename=`basename $item`

    #sync or swim
    sync
    /sbin/dump -"$level" -"$compression_mode" -n -u -f "$backup_destination/$basename.bak.$level" "$item"
done

# umount drive
/bin/umount "$backup_destination" 
 

On 4 Oct 2010, at 12:00, <bio-linux-request at nebclists.nerc.ac.uk> wrote:

Send Bio-Linux mailing list submissions to
bio-linux at nebclists.nerc.ac.uk

To subscribe or unsubscribe via the World Wide Web, visit
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux
or, via email, send a message with subject or body 'help' to
bio-linux-request at nebclists.nerc.ac.uk

You can reach the person managing the list at
bio-linux-owner at nebclists.nerc.ac.uk

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Bio-Linux digest..."


Today's Topics:

  1. Re: Cron and Qiime (Tony Travis)


----------------------------------------------------------------------

Message: 1
Date: Sun, 03 Oct 2010 16:31:49 +0100
From: Tony Travis <a.travis at abdn.ac.uk>
Subject: Re: [Bio-Linux] Cron and Qiime
To: bio-linux at nebclists.nerc.ac.uk
Message-ID: <4CA8A1E5.6040006 at abdn.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 30/09/10 15:12, Anthony Pemberton wrote:
Mike,

OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
#What to backup
backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})

Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.

The way to quickly test the script after editing is to run it from the command line eg.

% sudo -i
# cd /etc/cron.daily
# ./backup

and see what happens. Use of echo statements in shell scripts like this helps to debug,

Hello, Tony.

Tim added my changes to bio-linux-backups, but I have also noticed
problems if an "/etc/default/backup" file is absent. Sorry, I should
have posted something to the list.

I use these defaults (from Bio-Linux 5) for our NBX's:

# @(#)/etc/default/backup  2009-07-17  A.J.Travis

#
# Bio-Linux 5.0 backup configuration
#

# Where to backup
#DESTINATION="/backups"

# What to backup
#FILESYSTEMS="/ /home /var /usr"
FILESYSTEMS="/ /home"

# Compression scheme.  For large drives bzip2 is probably too slow.
#COMPRESSION="z"

HTH,

  Tony.
--
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt


------------------------------

_______________________________________________
Bio-Linux mailing list
Bio-Linux at nebclists.nerc.ac.uk
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux


End of Bio-Linux Digest, Vol 36, Issue 3
****************************************

Mike Cox

Research Associate
Centre for Respiratory Infection
National Heart Lung Institute
Royal Brompton Campus
Guy Scadding Building
London SW3 6LY

michael.cox1 at imperial.ac.uk
http://www1.imperial.ac.uk/medicine/people/michael.cox1/




_______________________________________________
Bio-Linux mailing list
Bio-Linux at nebclists.nerc.ac.uk
http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux


From tbooth at ceh.ac.uk  Mon Oct  4 09:53:42 2010
From: tbooth at ceh.ac.uk (Tim Booth)
Date: Mon, 04 Oct 2010 14:53:42 +0100
Subject: [Bio-Linux] Cron and Qiime
In-Reply-To: <1C3F9504-E9B5-45A8-A8AE-036B63C32C2E@imperial.ac.uk>
References: <mailman.1.1286190002.14964.bio-linux@nebclists.nerc.ac.uk>
	<1C3F9504-E9B5-45A8-A8AE-036B63C32C2E@imperial.ac.uk>
Message-ID: <1286200422.20632.16.camel@barsukas>

Hi All,

I'm back from hols and have taken a look.  Problems like this remind me
why I should always write scripts like this in Perl!

Mike - can you try going back to the original script and simply changing
the troublesome line 33 as follows:

snip>>>

#What to backup
backup_filesystems=(`echo ${FILESYSTEMS:="/ /home /var /usr"}`)

<<<snip

You can also uncomment the DEBUG bit on line 103 if you want to see what
it is doing.

If this works I'll update the package with the change (unless someone
can suggest a neater way than the use of echo + backticks).

Cheers,

TIM

On Mon, 2010-10-04 at 14:34 +0100, Cox, Michael J wrote:
> Hi Tony P,
> 
> Sorry, I did uncomment those lines and rerun backup, but it didn't
> return anything different from before.  I'm pretty sure that I've
> uncommented the right part of the script (there's only one debug
> line), but here is a full copy of my backup script as it now stands.  
> 
> Tony T, is the answer to create a file at /etc/default/backup with the
> config details that you list?
> 
> Thanks Mike
> 
> #!/bin/bash
> 
> #Fail on errors!
> set -e
> set -u
> 
> #######################################################################
> # /etc/cron.daily/backup - a script to backup one hd to               # 
> # another on the same machine, using dump                             #
> # (see the dump man page for details, and                             #
> # http://envgen.nox.ac.uk/envgen/software/archives/000501.html#auto   #
> #                                                                     #
> # Written by MIT, 20/6/2002                                           #
> # Modified by Dan Swan for Bio-Linux 2.0 17/3/2003                    #
> # Updated for Bio-Linux 5 by Tim Booth 1/8/2008                       #
> # Updated to perform Quarterly ETOH backups by Tony Travis 17/7/2009  #
> # Updated to dump ext4 fileystems by Tony Travis 19/5/2010            #
> #######################################################################
> 
> # Source configuration file
> CONFIG=/etc/default/backup
> if [ -r $CONFIG ]; then
>     . $CONFIG
> fi
> 
> #Use basic or ETOH mode?
> backup_mode=${MODE:="etoh"}
> 
> #Where to backup
> backup_destination=${DESTINATION:="/backups"}
> 
> #What to backup
> backup_filesystems=(${FILESYSTEMS:="/home" "/var" "/usr"})
> 
> #Compression scheme.  For large drives bzip2 is probably too slow.
> compression_mode=${COMPRESSION:="z"}
> 
> # get the date
> day=`date +%a`
> month=`date +%b`
> dom=`date +%d`
> 
> #Dump level can be set on command line or by today's date.
> if [ -n "$*" ]; then
>     level="$1"
> elif [ "$backup_mode" == "etoh" ] ; then
> 
>     # Quarterly Enhanced Towers of Hanoi
>     if [ $day == "Sun" ]; then
>         if [ $dom -le 7 ]; then
>             case $month in
>             "Jan" | "Apr" | "Jul" | "Oct") level=0 ;;
>             *) level=1 ;;
>         esac
>         elif [ $dom -le 14 ]; then
>             level=3
>         elif [ $dom -le 21 ]; then
>             level=2
>         elif [ $dom -le 28 ]; then
>             level=4
>         elif [ $dom -le 31 ]; then
>             level=3
>         fi
>     else
>         case $day in
>         "Mon") level=6 ;;
>         "Tue") level=5 ;;
>         "Wed") level=8 ;;
>         "Thu") level=7 ;;
>         "Fri") level=9 ;;
>         "Sat") level=8 ;;
>         esac
>     fi
> elif [ "$backup_mode" == "basic" ] ; then
>     # dump is 9 any day but Sunday
>     if [[ $day == "Sun" ]]; then
>             level=0
>     else
>             level=9
>     fi
> else
>     echo "ERROR: Unknown backup mode and no level specified"
>     exit 1
> fi
> 
> # dirs to back up - used to be:
> #backups=(hda1 hda2 hda5 hda6)
> #But now I need to be a bit more sophisticated.
> #List filesystems to back up and use mount to find where they are.
> backups=()
> 
> for fs in "${backup_filesystems[@]}" ; do
>     device=`mount | egrep "[^[:space:]]+ on $fs type ext[234]" | awk '{print $1}'`
> 
>     if [ "$device" != "" ]; then
> 	backups[${#backups[@]}]=$device
>     fi
> done
> 
> #DEBUG
> echo ${backups[@]} $level $compression_mode
> exit
> 
> # if backup drive is already mounted then try to unmount it first.              
> # if this is in use the step will fail and exit                                 
> /bin/mount | grep -q "/backups" && /bin/umount /backups 2>/dev/null
> 
> # mount necessary drive - must be listed
> # in /etc/fstab in order to be mounted
> /bin/mount "$backup_destination" 2>/dev/null || exit 0
> 
> # dump each directory, one at a time
> for item in "${backups[@]}"; do
>     basename=`basename $item`
> 
>     #sync or swim
>     sync
>     /sbin/dump -"$level" -"$compression_mode" -n -u -f "$backup_destination/$basename.bak.$level" "$item"
> done
> 
> # umount drive
> /bin/umount "$backup_destination" 
>  
> 
> On 4 Oct 2010, at 12:00, <bio-linux-request at nebclists.nerc.ac.uk> wrote:
> 
> Send Bio-Linux mailing list submissions to
> bio-linux at nebclists.nerc.ac.uk
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux
> or, via email, send a message with subject or body 'help' to
> bio-linux-request at nebclists.nerc.ac.uk
> 
> You can reach the person managing the list at
> bio-linux-owner at nebclists.nerc.ac.uk
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bio-Linux digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Cron and Qiime (Tony Travis)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Sun, 03 Oct 2010 16:31:49 +0100
> From: Tony Travis <a.travis at abdn.ac.uk>
> Subject: Re: [Bio-Linux] Cron and Qiime
> To: bio-linux at nebclists.nerc.ac.uk
> Message-ID: <4CA8A1E5.6040006 at abdn.ac.uk>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> On 30/09/10 15:12, Anthony Pemberton wrote:
> Mike,
> 
> OK cron is running. This looks like the problem I encountered with the script /etc/cron.daily/backup - under
> #What to backup
> backup_filesystems=(${FILESYSTEMS:="/home" "/home1"})
> 
> Note the use of the double quotes around the partitions you want to backup (these are the mount points). If I remember correctly, the quotes were missing and are now required. I think that is why the script is failing. The default config is a "red-herring" because what the script is saying is: if the file exists read it, if not continue.
> 
> The way to quickly test the script after editing is to run it from the command line eg.
> 
> % sudo -i
> # cd /etc/cron.daily
> # ./backup
> 
> and see what happens. Use of echo statements in shell scripts like this helps to debug,
> 
> Hello, Tony.
> 
> Tim added my changes to bio-linux-backups, but I have also noticed
> problems if an "/etc/default/backup" file is absent. Sorry, I should
> have posted something to the list.
> 
> I use these defaults (from Bio-Linux 5) for our NBX's:
> 
> # @(#)/etc/default/backup  2009-07-17  A.J.Travis
> 
> #
> # Bio-Linux 5.0 backup configuration
> #
> 
> # Where to backup
> #DESTINATION="/backups"
> 
> # What to backup
> #FILESYSTEMS="/ /home /var /usr"
> FILESYSTEMS="/ /home"
> 
> # Compression scheme.  For large drives bzip2 is probably too slow.
> #COMPRESSION="z"
> 
> HTH,
> 
>   Tony.
> --
> Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
> and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
> tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
> mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt
> 
> 
> ------------------------------
> 
> _______________________________________________
> Bio-Linux mailing list
> Bio-Linux at nebclists.nerc.ac.uk
> http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux
> 
> 
> End of Bio-Linux Digest, Vol 36, Issue 3
> ****************************************
> 
> Mike Cox
> 
> Research Associate
> Centre for Respiratory Infection
> National Heart Lung Institute
> Royal Brompton Campus
> Guy Scadding Building
> London SW3 6LY
> 
> michael.cox1 at imperial.ac.uk
> http://www1.imperial.ac.uk/medicine/people/michael.cox1/
> 
> 
> 
> 
> _______________________________________________
> Bio-Linux mailing list
> Bio-Linux at nebclists.nerc.ac.uk
> http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux

-- 
Tim Booth <tbooth at ceh.ac.uk>
NERC Environmental Bioinformatics Centre 

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB 

+44 1491 69 2705


-- 
This message (and any attachments) is for the recipient only. NERC
is subject to the Freedom of Information Act 2000 and the contents
of this email and any reply you make may be disclosed by NERC unless
it is exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.



From michael.cox1 at imperial.ac.uk  Mon Oct  4 10:20:13 2010
From: michael.cox1 at imperial.ac.uk (Cox, Michael J)
Date: Mon, 4 Oct 2010 15:20:13 +0100
Subject: [Bio-Linux] Cron
In-Reply-To: <mailman.4.1286200509.2178.bio-linux@nebclists.nerc.ac.uk>
References: <mailman.4.1286200509.2178.bio-linux@nebclists.nerc.ac.uk>
Message-ID: <447C4DF4-1B35-49A1-9711-B7EE94E39F4E@imperial.ac.uk>

Hi Tim,

welcome back!

Correcting line 33 as your suggestion, saving and running gives the same message as previously:

./backup: line 101: backups[@]: unbound variable

I also tried the Tonys' suggestion (thanks!) of creating a /etc/default/backup file, but sadly, no joy with running backup after this either.  

It looks like this:

#Where to backup
DESTINATION="/backups"

#What to backup
FILESYSTEMS="/ /home /var /usr"

#Compression scheme. For large drives bzip2 is probably too slow.
#COMPRESSION="z"

Cheers

Mike









From a.travis at abdn.ac.uk  Mon Oct  4 10:32:41 2010
From: a.travis at abdn.ac.uk (Tony Travis)
Date: Mon, 04 Oct 2010 15:32:41 +0100
Subject: [Bio-Linux] Cron
In-Reply-To: <447C4DF4-1B35-49A1-9711-B7EE94E39F4E@imperial.ac.uk>
References: <mailman.4.1286200509.2178.bio-linux@nebclists.nerc.ac.uk>
	<447C4DF4-1B35-49A1-9711-B7EE94E39F4E@imperial.ac.uk>
Message-ID: <4CA9E589.6020105@abdn.ac.uk>

On 04/10/10 15:20, Cox, Michael J wrote:
> Hi Tim,
>
> welcome back!
>
> Correcting line 33 as your suggestion, saving and running gives the
> same message as previously:
>
> ./backup: line 101: backups[@]: unbound variable

Hello, Mike.

Line 101 is normally commented out:

#echo ${backups[@]} $level $compression_mode

I suspect that you've uncommented it...

> I also tried the Tonys' suggestion (thanks!) of creating a
> /etc/default/backup file, but sadly, no joy with running backup after
> this either.

The "backup" script will just exit if it encounters and error, for 
safety, just in case it accidentally zaps your precious backups ;-)

HTH,

   Tony.
-- 
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt


From tbooth at ceh.ac.uk  Tue Oct  5 09:14:10 2010
From: tbooth at ceh.ac.uk (Tim Booth)
Date: Tue, 05 Oct 2010 14:14:10 +0100
Subject: [Bio-Linux] Cron - backups fix
In-Reply-To: <447C4DF4-1B35-49A1-9711-B7EE94E39F4E@imperial.ac.uk>
References: <mailman.4.1286200509.2178.bio-linux@nebclists.nerc.ac.uk>
	<447C4DF4-1B35-49A1-9711-B7EE94E39F4E@imperial.ac.uk>
Message-ID: <1286284450.20632.157.camel@barsukas>

Hi All,

For anyone following the discussions on the backup script, I've tweaked
the package slightly to overcome the quoting issue found by Mike.  I've
also added a better error message when things go wrong.  The updated
package is bio-linux-backups_0.3-6

In general, if you are relying on any backup system to preserve
important data then you can't be sure the backup is good unless you have
actually restored a system from it.  At the very least you need to keep
an eye on things to make sure everything is working as expected.

Cheers,

TIM




-- 
This message (and any attachments) is for the recipient only. NERC
is subject to the Freedom of Information Act 2000 and the contents
of this email and any reply you make may be disclosed by NERC unless
it is exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.