# a great backup system for gentoo workstations

## razamatan

ok... i started looking for simple backup solutions for just the bare essentials (mostly config and irreplaceable files) and i came to this script.  

people have extended this script by adding functionality, and i've linked the msgs with their mods at the bottom of this first post....

key points:

meant for personal workstations, not servers

allows for scalability by using user definable lists

extremely small and imho, has a very simple configuration

basically a fancy and automatic tar based backup solution

i call it backup.cron and put it in /etc/cron.weekly...  :Wink:  :

```

#!/bin/sh

# for FULL backups

# this backs up the important stuffs listed in ${lists} to ${BKPDIR}

# the lists *should* be in ${BKPDIR} and named <whatever>.lst

# the resulting backups will be <whatever>.<timestamp>.tgz

#

# notes:

# variables in CAPS are ok for you to set

# change the other variables if you know what you're doing

# you can get fancy in the lists... (think xargs *wink*)

# assumes /boot is usually unmounted, and has an /etc/fstab entry

# follow your security policy when setting perms on ${BKPDIR}

#

# written by razamatan

#

# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go

BOOT=sys                        # list that has /boot

NUMBKPS=3                       # how many backups to keep

                                                                                

if [ ! -d ${BKPDIR} ] ; then

        echo ${BKPDIR} is not a valid directory or is non-existant

fi

                                                                                

lists=${BKPDIR}/*.lst

ext=tgz

                                                                                

for list in `ls ${lists}`; do

        type=`basename ${list} .lst`

                                                                                

        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi

        cat ${list} | xargs tar zlcf \

                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1

        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}

        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

                if [ ${num} -le 0 ] ; then rm -f ${evict}

                else num=$((${num}-1)) ; fi

        done

done

```

you can download the latest version of the script from here: http://www-personal.umich.edu/~jyi/backup.cron

i actually only have one list, the /home/.backup/sys.lst:

```

/boot

/etc

/root

/usr/local/portage

--exclude=/usr/local/portage/distfiles

/usr/src/linux/.config

/var/lib

--exclude=/var/lib/texmf

```

EXTENSIONS & COMMENTS WELCOME!!!

here's some contributed extensions

iso images

iso images, tar splitting, pre & post-backup command processing, command line option to backup specific lists

specification of lists in a file (so you can have lists at various places

**EDIT** fixed numbkps issue

**EDIT** links to messages of extended versions

----------

## razamatan

oh... and if you should ever need to recover from said backups, make sure you use the --same-owner and --preserve flags w/ the tar command...

heh...

----------

## gatiba

Great script !

Would be nice if it could split the backup file in more parts if it's > 700mb for example (for burning  :Wink:  )

----------

## vulcan_

note that GNU tar has this option

 *Quote:*   

> 
> 
> -L --tape-length N
> 
>            change tapes after writing N*1024 bytes
> ...

 

but of course, this works to any file system that tar can output to

----------

## razamatan

yeah... you can do that on a per list basis.  so say your <username>.lst would be huge cus of all those multimedia files, then you just need something like this to split stuff.

```

/home/raza

--exclude=/home/raza/*cache

--exclude=/home/raza/*Cache

-L 7000  # ~700 MB chunks

```

 *vulcan_ wrote:*   

> note that GNU tar has this option
> 
>  *Quote:*   
> 
> -L --tape-length N
> ...

 

----------

## gatiba

Mmmhhh i have problems.

This is my sys.lst:

```
/boot

/etc

/home/httpd

/windows/D/____progetti

/windows/D/____progetti/progetti_Gabriele/Alberone/vari

--exclude=/windows/D/____progetti/progetti_Gabriele/Alberone/vari/foto

/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce

--exclude=/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce/foto

-L 6500
```

When i start the script i got this error:

```
ls: /windows/D/backups/sys.*.tgz: No such file or directory
```

If i delete the "-L 6500" line it works ...

----------

## sessionID

There should be an "umount /boot" line somewhere...   :Wink: 

----------

## razamatan

there is... look right below the crazy tar line..  :Smile: 

 *sessionID wrote:*   

> There should be an "umount /boot" line somewhere...  

 

----------

## razamatan

hmmm...

this script doesn't handle archive spanning.. :/ ... spanning requires interaction.  i really don't do archive spanning so i didn't test for that functionality.  i though the -L would do it, but you'd have to modify the script a bit.

to span archives, you can't compress it.  this would require the script to handle compression on a list by list basis, and that would add complexity.  i'll see what i can whip up  :Wink: .

 *gatiba wrote:*   

> Mmmhhh i have problems.
> 
> This is my sys.lst:
> 
> ```
> ...

 

----------

## razamatan

gah... tar's -L option is weird...

this'll be a bit more research than i planned... in the meantime... treat this backup system for compressed archives... and DOESN'T SUPPORT SPANNING!!

sigh..

sorry spanning folks... i'll program my way outta this.

----------

## gatiba

 :Wink: 

----------

## sessionID

 *razamatan wrote:*   

> there is... look right below the crazy tar line.. 
> 
>  *sessionID wrote:*   There should be an "umount /boot" line somewhere...   

 

oops  :Embarassed:  sorry

----------

## S_aIN_t

well done. i have been considering something similar.. althought i need a script that will backup linux and (gasp) windows.. but it would run from a linux server. i think there is something that samba has that would allow me to tar over an smb share.

more research is required for that. i'll be sure to post it here once i have something working.

----------

## danhan79

odd, i get a bus error when i run your script. queer no?

----------

## razamatan

lol... so you've managed to switch from rh to gen2?   :Smile:   hehehe...

the moderators on these forums are pretty tuff dan...  just im me later.

 *danhan79 wrote:*   

> odd, i get a bus error when i run your script. queer no?

 

----------

## Elm0

Great little script, just what I've been looking for  :Smile:  Thanks!

----------

## Elm0

I don't know if the script is designed to this but...

The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.

I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this...

----------

## razamatan

ahh!!

stupid global/local variable issue.. thanks for spelunking!  the code is fixed and reposted.

 *Elm0 wrote:*   

> I don't know if the script is designed to this but...
> 
> The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.
> 
> I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this...

 

----------

## Zeitgeist

Awsome script.

I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.

I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.

It's great  :Smile:  Thanks

----------

## razamatan

 *Zeitgeist wrote:*   

> Awsome script.
> 
> I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.
> 
> I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.
> ...

 

i feel all squishy and warm...  thanks man.     :Very Happy: 

----------

## Jeedo

Your posted script did not work for me, howver this one did: http://www-personal.umich.edu/~jyi/backup.cron

I've been looking for something like this, i even was going to write one myself but didnt know how (suck at bash)..

Anyway i intend to to what Zeitgeist did and back up my files on a regularly..

However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.

Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..

Another idea is to add this to the comments:

```

# DISCLAIMER: razamatan didn't write this if something goes wrong

#

# The latest version can always be found at : http://www-personal.umich.edu/~jyi/backup.cron

```

Anyway thanks alot man.

----------

## razamatan

 *Jeedo wrote:*   

> 
> 
> However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.
> 
> Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..
> ...

 

you can remove the "x" option from tar to remove default compression... you can then put -x or -j in your lst files to do a case by case compression.... but then, you'd have to deal with naming the files with the proper extension for every respective file... so that's why i went with default gzipping... since tar and gzip is usually around, i don't mind compressing even my most important files.

 *Jeedo wrote:*   

> 
> 
> Another idea is to add this to the comments:
> 
> ```
> ...

 

no problem...  i should put the url for this forum thread instead.  more support.. heh.

----------

## nat

Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.

```
--- backup.cron.orig    2003-10-07 15:25:16.282737029 +0200

+++ backup.cron 2003-10-07 15:41:29.382176277 +0200

@@ -16,7 +16,6 @@

 # DISCLAIMER: razamatan didn't write this if something goes wrong

 

 BKPDIR=/home/.backup           # where the backups go

-BOOT=sys                       # list that has /boot

 NUMBKPS=3                      # how many backups to keep

 

 if [ ! -d ${BKPDIR} ] ; then

@@ -29,10 +28,20 @@

 for list in `ls ${lists}`; do

        type=`basename ${list} .lst`

 

-       if [ ${type} = ${BOOT} ] ; then mount /boot ; fi

+# if the .lst file contains '/boot' lets mount /boot before backing up

+# unless its already mounted.

+

+       if grep '/boot' ${list} > /dev/null && \

+           ! mount | grep /boot > /dev/null ; then 

+           BOOT=1 

+       else

+           BOOT=""

+       fi

+

+       [ ${BOOT} ] && mount /boot

        cat ${list} | xargs tar zlcf \

                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1

-       if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

+       [ ${BOOT} ] && umount /boot

 

        num=${NUMBKPS}

        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

```

----------

## razamatan

 *nat wrote:*   

> Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.
> 
> 

 

i thougth about this... but then i realized.... there's no need to have boot in more than 1 archive.  so let's just specify it.  even in my incarnation, you can stick /boot in any list.. just name that list in the env. variable.

your code actually has a slight problem.  say that i have a situation where i have boot as a subdir of someplace that i wanted (/this/boot)..  or even that i don't wanna include boot (--exclude=/boot)... it greps for any instance of /boot... thereby, it does unnecessary (u)mounts.  just a slight problem.

----------

## Jeedo

 *razamatan wrote:*   

>  *Jeedo wrote:*   
> 
> However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.
> 
> Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..
> ...

 

I know, and that is what i did, but what i meant is that it would be nice to add it as a variable, like

compression=tbz2 

or tgz og just tar and the script would then add xj, xz or just nothing respectively

----------

## Elm0

I added this to the end of my script to allow automatic creation of an iso image of the backups to burn straight onto CD. I included the lst files as well because they are useful for determing whats in each final tar..

```

#!/bin/sh

# for FULL backups

# this backs up the important stuffs listed in ${lists} to ${BKPDIR}

# the lists *should* be in ${BKPDIR} and named <whatever>.lst

# the resulting backups will be <whatever>.<timestamp>.tgz

#

# notes:

# variables in CAPS are ok for you to set

# change the other variables if you know what you're doing

# you can get fancy in the lists... (think xargs *wink*)

# assumes /boot is usually unmounted, and has an /etc/fstab entry

# follow your security policy when setting perms on ${BKPDIR}

#

# written by razamatan

# ISO image modification by Elm0

# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go

BOOT=sys                        # list that has /boot

NUMBKPS=1                       # how many backups to keep

ISOUSER=jack                    # who you want to own the final ISO image

if [ ! -d ${BKPDIR} ] ; then

        echo ${BKPDIR} is not a valid directory or is non-existant

fi

lists=${BKPDIR}/*.lst

ext=tgz

for list in `ls ${lists}`; do

        type=`basename ${list} .lst`

        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi

        cat ${list} | xargs tar zlcf \

                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1

        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}

        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

                if [ ${num} -le 0 ] ; then rm -f ${evict}

                else num=$((${num}-1)) ; fi

        done

done

mkisofs -o /home/${ISOUSER}/backup.iso -iso-level 2 \

        -max-iso9660-filenames ${BKPDIR} > /dev/null 2>&1

chown ${ISOUSER} /home/${ISOUSER}/backup.iso

chgrp users /home/${ISOUSER}/backup.iso

chmod 0700 /home/${ISOUSER}/backup.iso

```

Just change the ISOUSER variable to reflect the name of the user who will own the iso file. It will also be moved to the root of their home directory.

----------

## Starfox

 *Jeedo wrote:*   

> 
> 
> ...
> 
> compression=tbz2 
> ...

 

Using bzip2 doesn't make much sense for backupping as your backup size will decrease by ~10-15% while your backup time will increase by ~100-150%!!!

decide yourself which one is more valueable to you, i'll keep gzip!

bye fox

----------

## Starfox

Hi folks,

what dirs do i have to put in sys.lst to backup my system, so i will be able to reinstall it by just taking a stage-X-tarball, and running emerge world???

I used:

 *Quote:*   

> 
> 
> /boot
> 
> /etc
> ...

 

Do I need any more???

thanks, fox

----------

## Jeedo

why are you backing up portage and libs?

----------

## Starfox

hmm, /usr/locale/portage because of my personal changed ebuilds, and i took /var/lib because it was mentioned just some posts above, but i'm not really sure if i'll need it for a reinstall!?

maybe i'd just take /var/lib/mysql !?

----------

## Starfox

hmm, i just reallized that 

```
/lib/modules
```

 is important for system backup , too. If you won't include it, you won't be able to start your old kernel from /boot corectly.

----------

## meowsqueak

Splitting into separate archives with tar -L requires all archives to be present to restore, correct? In that case, why not just create one big tarball and use 'split' to split it into 700MB chunks. You need them all to restore (cat them together and pipe into tar) so you don't lose anything (except you have to have them all available at the same time).

----------

## cripwalk

razamatan-

Thanks for the script.  This is EXACTLY what I was looking for!  It works perfect for me. Thanks again

----------

## Starfox

 *meowsqueak wrote:*   

> Splitting into separate archives with tar -L requires all archives to be present to restore, correct? In that case, why not just create one big tarball and use 'split' to split it into 700MB chunks. You need them all to restore (cat them together and pipe into tar) so you don't lose anything (except you have to have them all available at the same time).

 

This only works if you have enough free disk space, but anyway a nice idea!

----------

## meowsqueak

Indeed - I implied that, but you are right to clarify. Cheers.

----------

## razamatan

yeah... split'ing into 700mb chunks and later cat'ing them together would do it.    but i accomplish the same thing by having various lst files where each resultant tarball is smaller that 700mb...  usually a lot smaller.  so if i do decide to archive my backups to cdrom, i just burn the various tarballs individually until i hit 700mb.

i'm glad ppl are making their own solutions off of this.  that's what open source is all about.   :Smile: 

----------

## Souperman

 *Starfox wrote:*   

> maybe i'd just take /var/lib/mysql !?

 

It's safer to use the mysqldump program to create a plain text "dump" of your mysql database(s).  See 'man mysqldump' for details.

----------

## meowsqueak

This looks interesting and relevant:

http://freshmeat.net/branches/29090/

 *Quote:*   

> About: hdup is used to back up a filesystem. Features include encryption of the archive (with mcrypt), compression of the archive (bzip/gzip/none), the ability to transfer the archive to a remote host or restoring from a remote host (with ssh), the ability to split up archives, and no obscure archive format (it is a normal compressed tar file).

 

----------

## tomapd

Thaks for a very nice script!!

I posted just to say thank you.   :Very Happy: 

----------

## odegard

Great script!!

Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?

Thanks Gents

----------

## Souperman

 *odegard wrote:*   

> Great script!!
> 
> Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?
> 
> Thanks Gents

 

I have the following in my backup script:

```
BACKUP_ITEMS='/root /home /etc /var/log \

/var/spool/cron/crontabs /usr/local /var/cache/edb/world'

EXCLUDE_ITEMS='/root/tmp /home/httpd/htdocs/admin/backup/backup.tar.bz2 \

/home/httpd/htdocs/phpmyadmin /home/httpd/htdocs/admin/phpsysinfo'

```

Your requirements may differ.  :Wink: 

----------

## Starfox

 *odegard wrote:*   

> Great script!!
> 
> Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?
> 
> 

 

For a (without warranty) "full" system backup take this as sys.lst:

 *Quote:*   

> 
> 
> /boot
> 
> /etc
> ...

 

To restore, do

untar gentoo-statge-3-tarball to /

untar sys.backup to / 

emerge sync

emerge -eb world

after that you should have exactly the same system as before.

bye fox

PS: perhaps you should do some testing before to verfiy completness of the backup

----------

## meowsqueak

Another relevant backup script that creates uncompressed iso's for burning straight to CD, amongst others:

http://hoopajoo.net/quick_scripts.html

Specifically, take a look at the mkcdbackup script:

http://hoopajoo.net/static/quick_scripts/mkcdbackup

----------

## odegard

Hm, I'm no wizard and I can't sort out this problem.

I made /home/.backup where I put my backup.lst with the following content

```

/boot

/etc

/root

/usr/local/portage

--exclude=/usr/local/portage/distfiles

/usr/src/linux/.config

/var/lib

--exclude=/var/lib/texmf

/var/cache/edb/world

/lib/modules

/home/odegard

```

I copied the backup.cron file to cron.daily, made it executable and ran it manually. It executed it and went to make a cup of tee. When I came back it had stopped, but only because I was out of diskspace!! The file was several gigabytes!! Obviously it got looped. I did some small changes to the file:

```

#!/bin/sh

# for FULL backups

# this backs up the important stuffs listed in ${lists} to ${BKPDIR}

# the lists *should* be in ${BKPDIR} and named <whatever>.lst

# the resulting backups will be <whatever>.<timestamp>.tgz

#

# notes:

# - assumes ${BKPDIR} is unmounted and has an /etc/fstab entry

# - assumes /boot is unmounted and has an /etc/fstab entry

# - variables in CAPS are ok for you to set... change the other

#   vars if you know what you're doing

# - you can get fancy in the lists... think xargs *wink*, but

#   you can't use thes spanning feature to break up an

#   archive to smaller pieces of arbitrary size

# - follow your security policy when setting perms on ${BKPDIR}

#

# written by razamatan

#

# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go

BOOT=sys                        # list that has /boot

NUMBKPS=4                       # how many backups to keep

if [ ! -d ${BKPDIR} ] ; then

        echo ${BKPDIR} is not a valid directory or does not exist

fi

#mount ${BKPDIR}                # i have my backup directory on a seperate partition

lists=${BKPDIR}/*.lst

ext=tgz

for list in `ls ${lists}`; do

        type=`basename ${list} .lst`

        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi

        cat ${list} | xargs tar zlcf \

                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1

        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}

        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

                if [ ${num} -le 0 ] ; then rm -f ${evict}

                else num=$((${num}-1)) ; fi

        done

done

#umount ${BKPDIR}       # and i like to keep it unmounted

```

Since my backup goes to /home/.backup I commented out the mount/umount. I must confess that I don't quite understand the BOOT=sys thing so I just left it.

So, can anyone spot what's wrong?

EDIT: The only changes to the script is the mentioned commenting of mount/umount and setting BKPDIR to the correct dir.

----------

## hadfield

I can think of only one thing that might be happening, you have several gigabytes of files in one or more than one of the folders you're trying to backup. Use 'du -sk' on every folder listed in all of your lists. When I tested this script the other day I ran the test on folders that I thought only had about 20 Megs in, but when it got to 40 I knew there was a problem. It turned out that my .ccache folder in my /root directory had 150MB's in it.

If this doesn't help, you could narrow down the problem by taking out some folders from your list files until you've found the exact folder you're having a problem with.

I would bet that the problem is with your home folder run du -sk /home/odegard to see how many KB's are in that folder.

Scott

----------

## hadfield

first off, razamatan thanks for the script! I've been planning on writing a script to do just this for the past couple months, but I've just been too lazy to write it. So when I found your script yesterday, I just decided I had to finally do it. I've taken your script and a lot of the suggestions here and made the script a little more powerfull. Of course it added a lot of complexity to your script, which is too bad, becaue the simplicity of it is what got me interested in it in the first place.

Anyway, I've made the following additions to the script:

1. I Cleaned up the code a little just to be a bit more to my liking.  :Smile: 

2. Modified the ISO creation section, posted by Elm0 to make ISO's no larger than MAXISOSIZE. So multiple ISO's will be created.

3. If a single tgz file is larger than MAXISOSIZE, it splits the tgz file, using the split command into a bunch of seperate tgz files and creates an ISO for each.

4. I added pre and post command options on a per list basis. This allows you to run commands both before and after a list is compressed. I needed this for my ccache problem in the root folder. Just add %pre and %post to the list if you want to perform any additional operations.

Format: %pre command cmd_options ;command2 ;command3 cmd3_options

5. You can call this script with each list you want it to backup. So,

backup.cron  (with no options means backup all of the lists in the bkpdir)

backup.cron sys.lst music.lst  (backup only sys.list and music.lst)

Here's my example sys.lst with %pre commands:

```
%pre ccache -C

/boot

/etc

/root

/usr/local/portage

--exclude=/usr/local/portage/distfiles

/usr/src/linux/.config

--exclude=/var/lib/texmf

/var/cache/edb/world

/lib/modules
```

Here's the new script, mostly untested with larger backups, so be warned! Once again, sorry for the added complexity, but if you step through it line by line it's not actually all that difficult to understand.

```
#!/bin/sh

BKPDIR=/pub/backup              # where the backups go

BOOT=sys                        # list that has /boot

NUMBKPS=3                       # how many backups to keep

ISODIR=${BKPDIR}/iso            # where the iso's go

ISOUSER=root                    # who you want to own the final ISO image

MAXISOSIZE=600000               # the max size of iso files created

MAKEISO=Y                       # Should the script make an ISO

lists=${BKPDIR}/*.lst

ext=tgz

checkdir() {

    if [ -e $1 ] && [ ! -d $1 ]; then

   echo "$1 is not a valid directory!"

   exit 0

    elif [ ! -e $1 ]; then

   mkdir -p $1

    fi

}

makeISO() {

    # Remove any existing iso

    if [ -e $2 ]; then

   rm $2

    fi

    mkisofs -o $2 -iso-level 2 -max-iso9660-filenames $1 > /dev/null 2>&1

    chown ${ISOUSER} $2

    chgrp users $2

    chmod 0700 $2

}

checkdir ${BKPDIR}/tmp

checkdir ${ISODIR}/tmp

origdir=`pwd`  # Save the current directory

cd $BKPDIR

if [ $# -eq 0 ]; then

    lists=`ls ${lists}`

else

    lists="$*"

fi

bkpfiles=""

for list in $lists; do

    type=`basename ${list} .lst`

    bkpname=${type}.$(date +%Y-%m-%d-%H%M).${ext}

    

    if [ ${type} = ${BOOT} ]; then

   mount /boot

    fi

    # Execute pre-tar commands

    grep "%pre" ${list} | awk -F "%pre" {'print $2'} &> /dev/null

    echo "Compressing $bkpname..."

    grep -vE "^%pre|^%post" ${list} | xargs tar zlcf $bkpname > /dev/null 2>&1

    if [ ${type} = ${BOOT} ]; then

   umount /boot

    fi

    # Execute post-tar commands

    grep "%post" ${list} | awk -F "%post" {'print $2'} &> /dev/null

    

    # Add the file to our list of current backup items

    bkpfiles="$bkpfiles $bkpname"

    

    num=${NUMBKPS}

    for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

   if [ ${num} -le 0 ]; then

       rm -f ${evict}

   else

       num=$((${num}-1))

   fi

    done

done 

if [ "$MAKEISO" = "Y" ]; then

    # Create ISO's

    echo "Creating ISO's..."

    isosize=0

    isolist=""

    isocount=1

    for bkp in $bkpfiles; do

   bkpsize=$(du -sk $bkp | sed 's/^\([0-9]\+\).*/\1/')

        # Make sure the single tgz file isn't bigger than MAXISOSIZE

   if [ "$bkpsize" -gt "$MAXISOSIZE" ]; then

       split -C ${MAXISOSIZE}k $bkp ${BKPDIR}/tmp/${bkp}.

       partcount=1

       for broken in `ls -t ${BKPDIR}/tmp`; do

      mv ${BKPDIR}/tmp/${broken} ${ISODIR}/tmp

      makeISO ${ISODIR}/tmp \

          ${ISODIR}/backup.${isocount}-${partcount}.iso

      rm ${ISODIR}/tmp/${broken}

      let partcount++

       done

       let isocount++

       continue

   fi

   if [ "$(echo ${isosize}+${bkpsize} | bc)" -ge "$MAXISOSIZE" ]; then

       mv $isolist ${ISODIR}/tmp

       makeISO ${ISODIR}/tmp ${ISODIR}/backup.$isocount.iso

       mv ${ISODIR}/tmp/* ${BKPDIR}

       let isocount++

       isolist=""

       isosize=0

   else

       isolist="$isolist $bkp"

       isosize=$(echo "${isosize}+${bkpsize}" | bc)

   fi

    done

    # Create the last iso

    mv $isolist ${ISODIR}/tmp

    makeISO ${ISODIR}/tmp ${ISODIR}/backup.$isocount.iso

    mv ${ISODIR}/tmp/* ${BKPDIR}

fi

cd $origdir
```

----------

## razamatan

it's cool man..  i intended for this script to be used however ppl needed to use it.  i wanted to implement the "lower level" part of getting stuff into tarballs using lists.  that was enough for my backup needs.

in terms of the iso creating stuff... it might be wiser to seperate that into a different script... and call that script from the backup script...  iso creating isn't dependent on intermediary steps of this backup creation process...  it can do its thing after the backups are generated...

generally... i write scripts on the idea of just doing a small unit of work.. it fits better with the idea of scripts as commands... commands aren't applications.  the power of the command line lies in the idea that a command does one thing very well.

----------

## odegard

Hi, new problem.

I just reinstalled my system. I downloaded the script, commented out the mount commands since I don't back up to a mounted volume (yet), made it executable but when I try to execute it I get:

```

01:53:40 root /etc/cron.daily # ./backup.cron 

ls: /home/.backup/backup.*.tgz: No such file or directory

```

the list, backup.lst, is in /home/.backup

What is wrong?

----------

## hadfield

The line that is causing this problem is:

```
for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
```

It's looking for files to delete, but none exist yet. However, at least one file should exist, the file that just got backed up.

Are you sure that the backup file is being created? Is it being saved to the proper directory? First make sure that the tar command is running properly, I'm assuming this is where the error lies. Perhaps there's a typo in your .lst or something.

----------

## odegard

I'm running the script for the first time so there are no other files besides the backup.lst file there yet.

My backup.lst is

```
/boot

/etc

/root

/usr/local/portage

--exclude=/usr/local/portage/distfiles

/usr/src/linux/.config

/var/lib

--exclude=/var/lib/texmf

/var/cache/edb/world

/lib/modules

/home/odegard

--exclude/home/odegard/download

```

I'm at loss as to why this happens...

----------

## hadfield

Right, but by the time the script gets to the line that does the 'ls' on your backup directory (i.e. the line that's generating the error) there SHOULD be at least one file there. This is because the line of code that creates the tar file in that directory has already been executed. This is why I think there's something wrong with the generation of the tar file. Have any tgz files been created in your backup directory? Your CURRENT backup should be there (i.e. the backup you're trying to create).

----------

## odegard

Maybe I don't understand what you're saying but there are only one file in the backup dir. manual tar works just fine, this is how I tested it:

```

touch list

echo file > list

cat list | xargs tar zlcf archive

```

and I got an archive containing the file "file".

Could it be something with permissions? Can you give me the exact syntax I need to make the backup.cron executable?

Thanks for your time!

----------

## hadfield

Yeah, I often have trouble explaining things. :/ Anyway, If you put a file of the form:

backup.2003-10-30-2339.tgz

into your backup folder, do you still get the error message? Does your backup file get successfully created?

----------

## odegard

If I put an empty file with that name in my .backup directory and run the script nothing happens. No error message and the the file stays with the name I gave it and with 0 bytes.

The weird thing is that the script *worked* before I reinstalled.

To make the script executable, I do

```

chmod u+x backup.cron

```

Correct?

----------

## razamatan

hmm... first off... i'd

```
chmod 700 backup.cron
```

just so root is the only one who can run it...

now... as for the problems...

you should have these things existing:

```
- the directory /home/.backup

- a file /home/.backup/sys.lst

```

that's about it... and that lst file should list your directories and such.

----------

## razamatan

in general... if a script is giving you troubles, run it line by line (cut and paste)... see where it borks..

the thing is... this script should work...  try renaming your list to sys.lst since it contains /boot (and assuming that you need to mount it)...

----------

## odegard

Ahh...that was the error, my list file was called backup.lst and not sys.lst. It works now. Thank you very much!

----------

## razamatan

 *odegard wrote:*   

> Ahh...that was the error, my list file was called backup.lst and not sys.lst. It works now. Thank you very much!

 

honestly... it shouldn't matter...

try naming your list back to backup.lst (still under /home/.backup)... then in the backup.cron script, do this:

```
BOOT=backup
```

essentially, you should set BOOT to be the list that contains the /boot directory, so the script can properly (u)mount the thing.

but still... even if it wasn't properly mounting boot... it would archive the directory...

to really test to see what was wrong, revert everything back to when you were getting errors and

change 

```
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1 
```

to

```
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} 
```

(this just removes the redirection of errors and warnings for tar)

----------

## odegard

 *razamatan wrote:*   

> 
> 
> honestly... it shouldn't matter...
> 
> try naming your list back to backup.lst (still under /home/.backup)... then in the backup.cron script, do this:
> ...

 

Thats what I did. 

 *razamatan wrote:*   

> 
> 
> to really test to see what was wrong, revert everything back to when you were getting errors and
> 
> change 
> ...

 

Ok, I left the script in working order, that is, I did not change back the name of the list to 'sys' in the script, and did this:

```

#               ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1

                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext}

```

And I get this error msg:

```

06:49:44 root /etc/cron.daily # ./backup.cron 

tar: Removing leading `/' from member names

tar (child): /boot: Cannot open: Is a directory

tar (child): Error is not recoverable: exiting now

xargs: tar: terminated by signal 13

./backup.cron: line 40: /home/.backup/backup.2003-11-07-0649.tgz: No such file or directory

```

Is there something wrong with my tar-tools? Another thing thats funny is that I can't enter tar archives when using krusader...it says protocol not supported but I can pack/unpack just fine...just not browse the archive...

----------

## razamatan

how does /boot look on your machine...

is it a separate partition that is usually left unmounted?

----------

## odegard

Yes.

----------

## odegard

Yes.

----------

## odegard

Yes.

----------

## speedster

I know this is much more elementary; but for backing up my Documents (which change daily) I use the following in cron (I use fcron to run it):

15,45 * * * *  cp -ruf /home/SECRET/Documents /mnt/DESTINATION

every 15 and 45 past the hour, documents that have changed are automatically copied to my destination partition.  Of course, no compression is used, but who cares with a 100GB hard drive?  This hardly uses any system resources as well, except the first time it runs.  The same thing could be used for any system files or whatever.

----------

## my_nick

One tip from me:

Exclude mldonkey/xmule/whatever temp files from the backup because you can't extract the archive containing these files.

Been there, done that, had to reinstall   :Crying or Very sad: 

----------

## mkamonster

Are incremental backups possible with this script? I like the incremental backup's a lot.

----------

## razamatan

 *mkamonster wrote:*   

> Are incremental backups possible with this script? I like the incremental backup's a lot.

 

never really looked into that... i personally just wanted weekly full backups.

i'm sure you can add the incremental stuff, but it'll add more lines of code.   :Smile: 

----------

## Starfox

HI folks,

as i wanted to process only some backups every day i just wrote a small patch to razas script to take the first command line argument (if there is one!) as backuplist.

Just patch the original script with the following lines:

```
26,35c26

< # command line patch added by starfox

< # usage (e.g. if you have test.lst in your $BKPDIR):

< #     ./backup.sh test

< # now only the "test.lst" file is processed for backup

< if [ -f ${BKPDIR}/${1}.lst ]; then

<       lists=${BKPDIR}/${1}.lst

< else

<       lists=${BKPDIR}/*.lst

< fi

< # end command line patch

---

> lists=${BKPDIR}/*.lst

```

enjoy

----------

## timaios

 *gatiba wrote:*   

> Great script !
> 
> Would be nice if it could split the backup file in more parts if it's > 700mb for example (for burning  )

 

if you haven't still found a way to split your backup-files

in order to fit the in cds, you can also check split

"split -b 700m <tarfile> <prefix>"

----------

## Guglie

I've modified the script from razamatan with some new features:

- I use separate directories for the lists (~/.backup) and for the backups (NFS directory of my server) 

- I make backups every week and every month: I will also have every time backups of the lasts 6 months and backups of the lasts 4 weeks (1 month)

this is possible running the script with

```
backup_script -t [monthly/weekly]
```

I've also added the  -s list_name option, to select the list and the -l option to show a list of the availables lists

I copy here the full script and not a patch, because my code is much different than the code of razamatan

```
#!/bin/bash

# file: backup_script.sh

# date: september 2004

# original author: razamatan

# version: 0.2

# license: GPL-2

BKPDIR=/mnt/backup/galvorn     # directory where the backups go

LISTDIR=/home/evellon/.backup  # directory where the lists are

BOOT=sys                       # list that has /boot

NUMBKPS_weekly=4               # how many backups to keep in weekly-dir

NUMBKPS_monthly=6              # how many backups to keep in monthly-dir

EXT=tar.gz                     # extension of backups

function whelp

{

   echo "Usage: $(basename ${0})"

   echo

   echo "Options:"

   echo " -s list   backup only the directories of the specified list"

   echo " -t time   weekly or monthly"

   echo " -l        show the list of the backup-modes availables"

   echo " -h        watch this help and exit"

   exit 0

}

function wlist

{

   echo "Selectable lists:"

   for file in `ls ${LISTDIR}/*.lst`; do

      echo $(basename $file .lst)

   done

   exit 0

}

function make_backup

{

   if [ -z "$lists" ] ; then lists=${LISTDIR}/*.lst; fi

   if [ -z "$time" ] ; then time="weekly"; fi

   

   BKPDIR=$BKPDIR/$time

   if [ ! -d "$BKPDIR" ] ; then mkdir $BKPDIR; fi

   eval num=\$"NUMBKPS_"$time

   

   for list in `ls ${lists}`; do

      type=`basename ${list} .lst`

      

      if [ ${type} = ${BOOT} ] ; then mount /boot; fi

      

      echo "backupping "$type" ..."

      cat ${list} | xargs tar zlcf ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${EXT} > /dev/null 2>&1      

      

      if [ ${type} = ${BOOT} ] ; then umount /boot; fi

      

      for evict in `ls -t ${BKPDIR}/${type}.*.${EXT}`; do

         if [ ${num} -le 0 ] ; then

            rm -f ${evict}

         else

            num=$((${num}-1))

         fi

      done

   done

   exit 0

}

if [ ! -d ${LISTDIR} ] ; then

   echo "The LISTDIR ${LISTDIR} is not a valid directory or is non-existant"

elif [ ! -d ${BKDIR} ] ; then

   echo "The BKDIR ${BKDIR} is not a valid directory or is non-existant"

fi

while getopts ":hlt:s:" OPTION; do

   case $OPTION in

      h) whelp ;;

      l) wlist ;;

      t)   if [ "$OPTARG" = "weekly" ] || [ "$OPTARG" = "monthly" ] ; then

            time=$OPTARG

         else

            echo "Time mus be weekly or monthly"

            exit 0

         fi

         ;;

      s)

         if [ -f ${LISTDIR}/$OPTARG.lst ]; then

            lists=${LISTDIR}/$OPTARG.lst

         else

            echo "The list $OPTARG is not available"

            echo

            wlist

         fi

         ;;

      *) whelp ;;

   esac

done

make_backup

```

EDIT: I have patched the original code with the suggestions of burndive

----------

## burndive

Change BKPDIR to LISTDIR in the first line of this function (as shown below):

Also, ${BKPDIR}/weekly and ${BKPDIR}/monthly must already exist for the script to work.

```

function make_backup

{

   if [ -z "$lists" ] ; then lists=${LISTDIR}/*.lst; fi

   if [ -z "$time" ] ; then time="weekly"; fi

   ...

}

```

----------

## Guglie

 *burndive wrote:*   

> Change BKPDIR to LISTDIR in the first line of this function (as shown below):
> 
> Also, ${BKPDIR}/weekly and ${BKPDIR}/monthly must already exist for the script to work.

 

thanks for the testing: I have corrected this mistakes, this is the patch

```
--- backup_script-0.1.sh   2005-09-25 16:29:55.397696250 +0200

+++ backup_script-0.2.sh   2005-09-25 16:31:00.089739250 +0200

@@ -2,9 +2,10 @@

 # file: backup_script.sh

 # date: september 2004

 # original author: razamatan

+# version: 0.2

 # license: GPL-2

 

-BKPDIR=/mnt/backup             # directory where the backups go

+BKPDIR=/mnt/backup/galvorn     # directory where the backups go

 LISTDIR=/home/evellon/.backup  # directory where the lists are

 BOOT=sys                       # list that has /boot

 NUMBKPS_weekly=4               # how many backups to keep in weekly-dir

@@ -27,7 +28,6 @@

 {

    echo "Selectable lists:"

    for file in `ls ${LISTDIR}/*.lst`; do

-      #echo $file | sed -e 's/'$LISTDIR'/ /g' 

       echo $(basename $file .lst)

    done

    exit 0

@@ -35,11 +35,13 @@

 

 function make_backup

 {

-   if [ -z "$lists" ] ; then lists=${BKPDIR}/*.lst; fi

+   if [ -z "$lists" ] ; then lists=${LISTDIR}/*.lst; fi

    if [ -z "$time" ] ; then time="weekly"; fi

    

    BKPDIR=$BKPDIR/$time

-   num=$( eval echo "\$NUMBKPS_${time}" )

+   if [ ! -d "$BKPDIR" ] ; then mkdir $BKPDIR; fi

+

+   eval num=\$"NUMBKPS_"$time

    

    for list in `ls ${lists}`; do

       type=`basename ${list} .lst`

@@ -68,8 +70,7 @@

    echo "The BKDIR ${BKDIR} is not a valid directory or is non-existant"

 fi

 

-while getopts ":hlt:s:" OPTION;

-do

+while getopts ":hlt:s:" OPTION; do

    case $OPTION in

       h) whelp ;;

       l) wlist ;;

@@ -94,3 +95,4 @@

 done

 

 make_backup

+
```

NB: I have modified the code in the previous post with this patch

----------

## spindle

This script is so great. Simple and straight-forward. Just what I was looking for.

I've been using it  for a few weeks now. Works wonderfully.

Thanks!

----------

## Orange Crush

There's a pretty simple way of getting this script to do differential backups (i.e. every updated file since the last full backup so that you can restore with just the most recent full backup and the most recent differential.)

Just have two copies of the script . . .say "full.cron" and "differential.cron"

In full.cron, at some point before it runs tar add the line:

date > /var/lastbak

(that will record the date and time of the full backup)

Then in differential.cron just add "--newer /var/lastbak" to the tar options.

That should do it.  :Wink: 

-Nick

----------

## micr0c0sm

This would make a great wiki entry...

----------

## Hypnos

I have a laptop, and use an external USB drive + docking station.  I run the backup with rsnapshot and anacron.

Daily -- home dir

Weekly -- system files

Monthly -- everything else

----------

## jtp755

 *Quote:*   

> hadfield:
> 
> Here's my example sys.lst with %pre commands: 
> 
> Code: 
> ...

 

you are missing /var/lib in your list but are excluding things from it   :Smile:  Also what kind of pre and post options are there and what to they do?

on to my ??s....i have chosen to try out Guglie's version for now because i like the opions...ill probably go back to add another script to add on the ISO support later on. Is there a way to run this one in a cron? What im thinking is to specify in cron.weekly to run backup_script -t weekly and then in cron.monthly run backup_script -t montly. is that possible? Also has anyone tried saving backups on another system on the network like through Samba or something and if so how and what do you use. Thanks so much

----------

## LAj

Hi all,

I understand this is a great script but my needs are slowly different.

I want to know which dirs I must backup before upgrading to gcc4 if I would came back to gcc3.

I think:

/bin

/emul

/lib32

/lib64

/media

/opt

/sbin

/usr

I don't know:

/sys

Is thit a good backup?

----------

## razamatan

at this point, it seems that people who want a "simple" backup script can go one of two ways.  stick w/ the script i put up which handles most people's needs and can be hacked pretty easily, or if you would rather get all the bells and whistles right away (along w/ the configuration choices) check out http://www.backup-manager.org/ (it's in portage) .  it's a bash + perl solution that pretty robust but isn't really simple, imo.  put in another way, it's what this script would look like to cover all the use cases outlined in the various feature requests of this thread.

----------

