# HOWTO: Easily make a full system backup (stage4)

## BlinkEye

Please go here for the most up-to-date version of this howto. i put this howto into my wiki as it's far easier to maintain and a lot more clearly laid out. 

INDEX

0. What's this all about

1. How it works

2. Features

3. Portability/Prerequisites

4. How to customise

5. The result

6. The script

7. Download

8. Run it

9. Restore

10. Changelog

0. What's this all about

i want to backup my system quickly, easily and with minimum effort. that's why i wrote this script. the resulting backup file (.tar.bz2 or .tar.gz) is called a stage4. you're probably a gentoo user and hence know the three stages gentoo uses. this backup will be a stage4, meaning, it will be a full system backup of your current system. I got inspired by this thread. if you want to know how it works, continue reading, if not, there's no reason to read all through this thread - jump to Section "8. Run it".

1. How it works

it's a bash script which uses only "standard" tools, namely "tar" and if you intend to burn the stage4 on a cd you'll probably use "split" too. this means this script works out of the box. it has been modified a lot and i now tried to detect any error/misconfiguration a user might do (wrong $PATH, wrong commands, wrong exclude list, overwriting existing files ...). 

after checking everything's all set up the script will provide you with four options:

```
Backup script v3.5

==================

What do you want to do? (Use CONTROL-C to abort)

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup

Please enter your option:
```

Minimal backup

this option allows you to quickly build a stage4.tar.bz2/stage4.tar.gz without any further fuss, which means

1. it will exclude (i.e. NOT backup) any files/directories listed in the $custom_list variable

2. result in a minimal stage4.

Interactive backup

this option will ask you for any file/directory listed in the custom_list variable if you want it to backup. this might result in the same stage4.tar.bz2/stage4.tar.gz if you answer all question with "no" or in a considerably bigger stage4. 

Parameters

the script only shows error on stdout per default. if you want to backup in verbose mode and watch the files processed by tar call the script like:

```
./mkstage4.sh --verbose
```

or

```
./mkstage4.sh -v
```

if you intend to backup your script on a cd/dvd with a predefined size call the script like:

```
./mkstage4.sh --split
```

or

```
./mkstage4.sh -s
```

don't forget to adjust the $split_options variable, i.e. to set the desired chunk size. default is 685MB. 

of course you may combine command line parameters in any way.

2. Features

* works right out of the box. 

* checks if all needed tools exist.

* checks that any file/folder listed exists (no spelling errors).

* names backups according to its hostname and the date it was created -> unique, meaningful file name.

* prevents overwriting a backup file created on the same day.

* does an integrity check after the stage4 has been created.

* allows fine-tuning and easy change.

* verbose mode.

* command line option --split for splitting the tarball on the fly.

3. Portability/Prerequisites

Against some misconceptions you may use your stage4 for a box with a different CPU as well. If - and only if - you plan to use your stage4 for another CPU type (not just another box with the same CPU), you must check your /etc/make.conf for the CFLAGS entry:

```
CFLAGS="-Os -mcpu=athlon -funroll-loops -pipe"
```

if your CFLAGS looks like this: 

```
CFLAGS="-Os -march=athlon -funroll-loops -pipe"
```

you won't be able to use your stage4 for a box with a different CPU type. "march" breaks compatibility, "mcpu" doesn't. there's an easy workaround: replace "march" with "mcpu" and recompile your hole system: 

```
emerge -e world
```

i created a stage4 from my laptop (CFLAGS="-Os -mcpu=pentium4 -funroll-loops -pipe"), extracted it on a usb harddisk and am able to boot my amd64 machine. applications work flawlessly.

vmware users, please read this for restore.

4. How to customise

as mentioned above there are several variables to customise. the script itself has a few comments which should get you started. you might want to change the location where the stage4.tar.bz2 is put (currently /mnt/backups/stage4), or you might not like the filename (hostname-stage4-datum.tar.bz2), or you might think that the kernel sources is part of a minimal system backup and hence you won't exclude it (and so on) ...

$default_exclude_list: put any file/directory in here which is never needed nor wanted for a minimal full system backup. 

$default_exclude_pattern: exclude patterns for the $default_include_folders/files.

$default_include_files: files/folder which are needed for a minimal working system. don't add folders which should be backed up recursively. folders added here provide solely the folder structure.

$default_include_folders: folders which need to be backed-up recursively for a minimal working system.

$custom_include_list: directories which aren't needed but which may be desirable to be backed-up too interactively (like /home or /usr/src/). of course opinions about such files/directories differ, so suit yourself.

$custom_exclude_list: files/folders which are subfolders of a folder listed in $custom_include_list which should NOT be backed up.

$custom_exclude_pattern: exclude patterns for the $custom_include_list.

5. The result

the result is IMPRESSIVE. 

stage4.tar.gz: you'll get a stage4.tar.gz which will take about half the time it does for a bzip2 stage4. for my laptop it takes about 20 minutes to create the stage4, my amd64 with a software raid0 needs 8 minutes. unpacking takes about 2/3 of the time it does for a bzip2 stage4. the stage4.tar.gz will be about 10% bigger than a stage4.tar.bzip2. it's your choice!

stage4.tar.bz2: you'll get a stage4.tar.bz2 (for me it's about 800MB from 3.5GB of data) which will be a fully working backup of your system. on my ibm laptop it takes about 50 minutes to create the bz2 stage4.

i recall all those hours i backup-ed my windows installation. the record was 7CD's for a full installation (inkl. some software you need for the daily use) - not bootable  :Twisted Evil: .

6. The Script

```
#!/bin/bash 

# Backup script for Gentoo Linux 

# Copyright Reto Glauser aka Blinkeye 

# Distributed under the terms of the GNU General Public License v2

# Mailto: stage4 at blinkeye dot ch

# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html 

# Date: 2005-06-30

version=v3.5

basename=`basename $0`

find=/usr/bin/find

tar=/bin/tar

# these are the commands we actually need for the backup

command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.

for command in ${command_list[@]}; do

   if [ ! -x "`which $command 2>&1`" ]; then

      echo -e "\nERROR: $command not found! "

      base=`basename $command`

       if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then

          echo -e "ERROR: $base is not in your \$PATH."

       fi

       exit -1 

   fi

done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# Defaults to creating one tarball

tar_output="--file"

# split command

split_options="--suffix-length=1 --bytes=685m"

# options for the tar command 

tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read" 

# where to put the stage4

stage4Location=/mnt/backups/stage4

# name prefix

stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`

# patterns which should not be backed up (like iso files).

# example: default_exclude_pattern="*.iso *.divx"

# These pattern count only for files NOT listed in the $custom_include_list.

default_exclude_pattern=""

# these files/directories are always excluded. don't add trailing slashes.

# don't touch it unless you know what you are doing!

# /var/db and /var/cache/edb are intentionally added here. they are listed

# in $default_include_folders

default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

/var/db

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

# files/devices/folders, which need to be backed up (preserve folder structure). 

# don't touch it unless you know what you are doing! no recursive backup of folders. 

# use $default_include_folders instead.

default_include_files="

/dev/null

/dev/console

/home

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log"

# folders, which need to be backed up recursively on every backup. 

# don't touch it unless you know what you are doing! the reason for this

# variable is that some users add /var to the $default_exclude_list. here 

# we ensure that portage's memory is backed up in any case.

default_include_folders="

/var/db"

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will

# include/exclude these files/folders depening on your answer.

custom_include_list="

/home/*

/usr/src/linux-`uname -r`"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT

# be backed up. eg. 

#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"

custom_exclude_list=""

# Only files/folders within the $custom_include_list are checked against these patterns

# custom_exclude_pattern="*.mp3 *.iso"

custom_exclude_pattern=""

# the find_command

find_command="$find /*"

# don't backup anything which matches pattern listed in $default_exclude_pattern

for pattern in $default_exclude_pattern; do

   find_command="$find_command -not -name $pattern"

done

# assemble the find_command 

function find_files()

{

   for folder in $default_exclude_list; do

      find_command="$find_command -path $folder -prune -o"

   done

   find_command="$find_command -print"

   for i in $default_include_files; do

      find_command="echo $i; $find_command"

   done   

   for i in $default_include_folders; do

      if [ -d $i ]; then 

         find_command="$find $i; $find_command"

      else

         find_command="echo $i; $find_command"

      fi      

   done

}

# check the exclude/include variables for non-existing entries

function verify()

{

   for i in $1; do

      if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then

         echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2

         exit 0

      fi

   done   

}

# check input parameters

while [ $1 ]; do

   case  $1 in

   "-h" | "--help")

      echo -e $help

      exit 0;;

   "-v" | "--verbose")

      verbose=$1;;

   "-s" | "--split")

      tar_output="--split";;

   "");;

   *) 

      echo -e $help

      exit 0;;

   esac

   shift

done

echo ""

# check folder/files listed in $default_exclude_list exist

verify "$default_exclude_list" "\$default_exclude_list"

# check files listed in $default_include_files exist

verify "$default_include_files" "\$default_include_files"

# check folder listed in $default_include_folders exist

verify "$default_include_folders" "\$default_include_folders"

#check folder listed in $custom_include_list exist

verify "$custom_include_list" "\$custom_include_list"

#check folder listed in $custom_exclude_list exist

verify "$custom_exclude_list" "\$custom_exclude_list"

 

# print out the version

 echo -e "\nBackup script $version"

 echo -e "=================="

# how do you want to backup?

echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup\n"

while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do

   echo -en "Please enter your option: "

   read option

done

case $option in

[1,3])

   stage4Name=$stage4Location/$stage4prefix-minimal.tar;;

[2,4])

   stage4Name=$stage4Location/$stage4prefix-custom.tar

   for folder in $custom_include_list; do 

      echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 

      read answer 

      while [ "$answer" != 'y' -a "$answer" != 'n' ]; do 

         echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 

         read answer 

      done 

      if [ "$answer" == 'n' ]; then 

         find_command="$find_command -path $folder -prune -o"

      else

         custom_find="$find $folder"

         for i in $custom_exclude_pattern; do

            custom_find="$custom_find -name $i -o"

         done

         for i in $custom_exclude_list; do

            custom_find="$custom_find -path $i -prune -o"

         done

         find_command="$custom_find -print; $find_command"

      fi

   done ;;

esac

# add $custom_include_list to the $default_exclude_list as we assembled

# $custom_find with $custom_include_list already.

default_exclude_list="$default_exclude_list $custom_include_list"

case $option in

[1,2])

   stage4postfix="gz"

   zip="--gzip";;

   

[3,4])

   stage4postfix="bz2" 

   zip="--bzip2";;

esac

# mount boot

echo -e "\n* mounting boot" 

mount /boot >/dev/null 2>&1

# find the files/folder to backup

find_files

find_command="($find_command)"

# create the final command

if [ "$tar_output" == "--file" ]; then

   tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"

else

   tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"

fi

if [ "$verbose" ]; then

   echo -ne "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command

fi

# everything is set, are you sure to continue?

echo -ne "\nDo you want to continue? (y/n) " 

read answer 

while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do 

   echo -ne "Do you want to continue? (y/n) " 

   read answer 

done 

if [ "$answer" == 'y' ]; then 

   # check whether the file already exists.

   if [ "$tar_output" == "--split" ]; then

      overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"

   else

      overwrite="$stage4Name.$stage4postfix"

   fi

   

   if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then  

      echo -en "\nDo you want to overwrite $overwrite? (y/n) "

      read answer

      while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

         echo -en "Do you want to overwrite $overwrite? (y/n) "

         read answer

      done

      if [ "$answer" == 'n' ]; then

         echo -e "\n* There's nothing to do ... Exiting" 

         exit 0;

      fi

   fi

   

   # if necessary, create the stage4Location 

   if [ ! -d "$stage4Location" ] ; then 

      echo "* creating directory $stage4Location" 

      mkdir -p $stage4Location 

   fi

   echo -e "\n* Please wait while the stage4 is being created.\n"

   # do the backup.   

   sh -c "$tar_command"

   

   # finished, clean up

   echo -e "\n* stage4 is done" 

   echo "* umounting boot" 

   umount /boot >/dev/null 2>&1

   # Integrity check

   echo -e "* Checking integrity"

   if [ "$zip" == "--gzip" ]; then

      zip="gzip"

   else

      zip="bzip2"

   fi   

   

   if [ "$tar_output" == "--split" ]; then

      if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then

         echo -e "* Integrity check failed. Re-run the script and check your hardware."

         exit -1

      fi

   else

      if [ "`$zip --test  $stage4Name.$stage4postfix 2>&1`" != "" ]; then

         echo -e "* Integrity check failed. Re-run the script and check your hardware."

         exit -1

      fi

   fi

   

   # everything went smoothly"

   echo -e "* Everything went smoothly. You successfully created a stage4."

   

else

   echo -e "\n* There's nothing to do ... Exiting" 

fi 

# Split the archive into chunks - uncomment the 3 lines if you want to split the stage4

# echo -e "* split $stage4Name.$stage4postfix" 

# split $split_options $stage4Name.$stage4postfix "$stage4Name.$stage4postfix"_ 

# echo "* splitting is done"
```

7. Download

ftp://blinkeye.ch/gentoo/mkstage4.sh

8. Run it

```
wget ftp://blinkeye.ch/gentoo/mkstage4.sh

chmod +x  mkstage4.sh

./mkstage4.sh
```

this will execute the script (you must be root to succesfully backup all folders). 

for available parameters see section "Parameters" in "1. How it works".

9. Restore

```
 1. boot off a live-cd and repartition and create filesystems as necessary

 2. eventually reboot, using option: gentoo docache 

 3. umount /mnt/cdrom 

 4. remove the live-cd and insert the cd with the stage4

 5. mount /dev/cdrom /mnt/cdrom 

 6. mount /dev/hdaX /mnt/gentoo 

 7. mkdir /mnt/gentoo/boot

 8. mount /dev/hdaX /mnt/gentoo/boot

 9. tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/ 

 or

 9. tar xjvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.bz2 -C /mnt/gentoo/

10. mount -t proc none /mnt/gentoo/proc 

11. mount -o bind /dev /mnt/gentoo/dev

12. chroot /mnt/gentoo /bin/bash

13. env-update

14. source /etc/profile

if in need adjust necessary files (/etc/fstab, /boot/grub/grub.conf) and/or install grub

15. emerge sync      (rebuild portage tree)

16. exit

17. cd /

18. umount /mnt/cdrom 

19. remove backup cd 

20. umount /mnt/gentoo/boot

21. umount /mnt/gentoo/dev

22. umount /mnt/gentoo/proc

23  umount /mnt/gentoo

24. Reboot
```

Re-assemble split-ed stage4

If you split your stage4 either with the command line parameter --split/-s or uncommented the split section within the script you need to re-assemble the split chunks if you intend to restore such a stage4:

```
cat stage4.tar.gz_* > stage4.tar.gz
```

or

```
cat stage4.tar.bz2_* > stage4.tar.bz2
```

substitute stage4.tar.gz/stage4.tar.bz2 with your stage4 name.

10. Changelog

2005-03-23 fixed split command

2005-03-27 added new option for faster backup (gzip)

2005-03-28 as of udev i added /dev/console /dev/null to be backed up - gets rid of those "Unable to open initial console" messages

2005-03-27 added /dev/console, /dev/null to both final_command variables

2005-03-31 fixed issue of included files/folder from excluded parent folder

2005-04-15 fixed typ (zip -> gzip) in command_list

2005-04-20 added new $exclude_pattern variable thanks to a hint by saskatchewan46. not yet checking/verifying valid entries though

2005-04-24 NW RLAS. the stage4 is compressed in-place -> no more temporary additional space is needed. it's faster now.

2005-04-28 Fixed command (bzip2 options now create a bzip2 stage4 again). small change in the code.

2005-04-30 Fixed Restore commands. thanks to Alpo Nestori

2005-05-08 Removed warning about the /boot/boot symlink. no longer necessary

2005-05-09 Added "6. Warnning" section. $default_include_list doesn't recursively back up folders

2005-05-09 Fixed issue with $default_include_list, i.e. added new variable $custom_include_list

2005-05-15 nclude /tmp without its content to preserve folder structure. thanks to Lorijho. Verifying custom_include_list for wrong entries.

2005-06-07 Added integrity check of the stage4. Changed default filename to Year-Month-ay for easier listing.

2005-06-08 Fixed issue about not preserving all permissions.

2005-06-11 

     * Fixed confusing variable names.

     * Removed old debug echo statement.

     * Split $default_include_list in $default_include_files and $default_include_folders.

     * Added portage's memory (/var/db) to $default_include_folders to prevent any mistakes.

     * Added portage's /var/cach/edb to $default_include_folders.

     * Included files/folders now take precedence of excluded files/folders.

     * Not existing files/folders in any list exits the script as it could be ignored too easily.

     * Added ftp link to the script for those having issues copy & pasting it.

     * Updated this Howo - explained some variables.

2005-06-13 Added $custom_include_list, $custom_exclude_list, $custom_exclude_pattern to provide the abilty of backing up directories interactively while excluding files/folders deeper in the tree.

2005-06-14 

* Added /var/db and /var/cache/edb to the $default_exclude_list to prevent backing it up twice.

* Removed udev warning sections as there seem to be no futher issues.

* Added verbose mode to the script. Default mode is now non-verbose, i.e. only errors are shown. For verbose mode use "mkstage4.sh --verbose" or "mkstage4.sh -v".

* Hardcoded tar and find. Added routine for suggestion path if tar and/or find could not be found.

* Updated Howto. New Section "2. Features". New Section "7. Run it".

2005-06-15 Added section "3. Portability/Prerequesites"

2005-06-21 Added new parameter --split and new section "Parameters" in "1. How it works".

2005-06-30 

* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order.

* Fixed integrity check for splitted files - it's now possible

* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up.

* Removed /var/log/portage per a default as new users get confused about adjusting the script.

* Added used commands to be checked at the beginning.

2005-07-25 Added vmware link to dundas mini howto.

----------

## tnt

Very usefull!

Thank you!

 :Wink: 

----------

## jdgill0

You could also backup your MBR (master boot record) with

```
dd if=/dev/hdx of=mbr.img bs=512 count=1
```

where you replace hdx with the dev that contains your MBR for booting, for example hda (not hda1 either).

You could then restore your MBR with

```
dd if=mbr.img of=/dev/hda
```

Although I note that I have not actually ever restored the MBR  :Smile: 

----------

## iverasp

just did a stage4 installation myself, and when rebooting my computer complained about missing /proc folder. so remember to make one with "mkdir -p /proc" before rebooting.

----------

## BlinkEye

 *iverasp wrote:*   

> just did a stage4 installation myself, and when rebooting my computer complained about missing /proc folder. so remember to make one with "mkdir -p /proc" before rebooting.

 

you have to do that only if you did exclude wrongly. use 

```
--exclude=/foldername/*
```

instead of

```
--exclude=/foldername
```

to keep the directory structure. 

note: the -p flag isn't needed anyway, you don't have none existent parent directories of /proc.

----------

## BlinkEye

 *jdgill0 wrote:*   

> You could also backup your MBR (master boot record) with
> 
> ```
> dd if=/dev/hdx of=mbr.img bs=512 count=1
> ```
> ...

 

it's too sensitive.

----------

## tnt

 *jdgill0 wrote:*   

> 
> 
> You could then restore your MBR with
> 
> ```
> ...

 

Well that would overwrite your partition table, too. That's not so good idea if you changed partitions meanwhile or just want to restore it on other hard drive.

As far as I know (so it should be checked), last 2 bytes in the MBR tell BIOS to boot or not to boot from that drive (0xAA55 or something else).

64 bytes before that are partition table: 4 entry with 16 bytes each.

So, boot code (lilo, grub, or whatever) shoud be in the first 446 bytes of MBR.

Maybe we should backup, or at least restore, just boot code and try not to mess partition tables...

```
dd if=/dev/hdx of=mbr.img bs=446 count=1
```

But, anyway, I thing that it should be better to chroot in restored environment, edit fstab & lilo.conf, and run "lilo -v" just to avoid partition location and numbering problems...

----------

## wilho

You're right according to this: http://www.mobiledyne.com/pub/mbrh.html

----------

## BlinkEye

thanks for clarifying. 

notice: new version of the script - it's now possible to choose between a tar.bzip2 or a tar.gz stage4. i had to unpack a stage4 on a 166Mhz box recently where i noticed it might be desirable not to use bzip2 but gzip.

----------

## tnt

Maybe you should change

```
version=v1.2 
```

to

```
version=v1.2.1
```

or something similar...

----------

## tetrahydroc

Been looking for something like this, thanks!

----------

## superfes

I hate to mention this on such a neat script posting and stuff, but you're missing a quote at  echo -e "\nBackup script v.1.3

*innocent smile*

----------

## BlinkEye

i don't know how this happened, i didn't miss that quote in my script even though i do copy the code from my script. nevertheless i found another minor mistake and hence changed the version to 1.3.1. thanks!

----------

## grenouille

nice work, thanks  :Smile: 

----------

## syg00

Had to throw away my / partition today, and this recovered the situation delightfully.

Have some slight issues with the script - hopefully can be dealt with via PM.

----------

## BlinkEye

glad to hear that. i just noticed that /dev/console and /dev/null don't get backed up this way. doesn't work combined with the exclude parameters. hmm, i'm trying to figure out a solution, but at the moment i don't see one without using the -T parameter (which needs an extra file). anyone has got an idea?

----------

## tnt

Why not to make that extra file in /tmp/ right from the script by

```
echo "whatever you need" > /tmp/temp.backup.exclude.file
```

and remove it 

```
rm -f /tmp/temp.backup.exclude.file
```

at the and of the script?

----------

## BlinkEye

a new version: 1.4.0. i finally took the time to look into the issue that you can't backup files/folders under a folder which is excluded. see info page from tar: it's simply not possible, nomatter how you do it. i added a new variable $default_include_list which purpose's to backup files/folder which are deeper in the tree of an excluded folder (like /var/log/messages or /dev/null). it's somehow slower as i must do it in 3 steps now - and you need temporarly more harddisk memory (the compression of the tar file can't be done in-place anymore, but the final stage4 is of the same size of course). nevertheless the script's now even more easily configurable and you won't get those nasty "Could not open inital root= console" errors with udev anymore. if you want to know the running time use it like:

```
time ./mkstage4.sh
```

----------

## mauricev

The backup seems work nicely, but there isn't much information on restore, although I succeeded on a bare metal restore as a test!   :Very Happy: 

Anyway, the wiki  doesn't seem to mention anything about where to put the stage4 tar file for easy access. I used the steps described in 

https://forums.gentoo.org/viewtopic-t-21327-postdays-0-postorder-asc-start-0.html

to build a Live DVD appending the stage4 file I made. This is cool  :Cool: 

Also, regarding the wiki,  where it says:

 *Quote:*   

> Partition and mount filesystem

 

What happened to creating the filesystems? (Here's where I learned that 2005.0 CD for AMD requires ext3 to be mounted as "mount -t ext3...", some bug in the e2fprogs utilities)

For restoration I ran into a number of snags. First, when restoring the partitions, the sequence 

```
sfdisk -d /dev/(your_disk) > partitions.save
```

```
sfdisk /dev/(your_disk) < partitions.save
```

works just great if I try onto a second drive in one computer, but when I bring it to another computer, it doesn't work right. I get errors like 

 *Quote:*   

> Partition 1 does not end on cylinder boundary. 

 

The drive where it worked was an Hitachi 400 GB connected via Silicon Image SATA, but the one where it didn't is a WD 74 GB connected via 3Ware SATA). Both systems are Opterons. I'm not sure that could make a difference. In the first case, I was booted on my live filesystem, on the second I was booted from Stage4 DVD.

Interestingly, restoring the MBR after restoring the partitions eliminates partition funkiness, so for some strange reason sfdisk appears to be doing something wrong when it's run from this other computer from my Stage4 DVD (which is based Gentoo 2005.0 CD for AMD64).

However, the MBR didn't completely restore. First, the bootable flag for /dev/sda1 was not set. Second, GRUB just hangs at boot:

 *Quote:*   

> GRUB

 

When I tried to fix this with grub-install, it complained

 *Quote:*   

> /dev/sda does not have a corresponding BIOS drive

 

I was, however, able to manually run grub and then it just worked.

----------

## Greven

You really have made this a great script.  How about adding Dar to the script, so if it's finds an older tar, it will dar it and save space.

----------

## mudrii

nice script and usefull I will try it 

Thanx

----------

## BlinkEye

thanks for the feedback. 

@mauricev: thanks for the link. i didn't know such a thread existed, have been looking for it for quite some time. 

@greven: seems i'll have to look into dar, looks like a great tool.

----------

## Sade

is there a way to easily copy the script from a program such as lynx or links2??

if not, could u put the script in a tar, and post a link to it so i can download it.

when i select copy and paste, i get a lot of spaces in my file, and i don't know how not to select them, or smartly remove the spaces.

----------

## BlinkEye

do you know w3m? i like it a lot more than lynx - it has even mouse-scroll, mouse-klick (left and right) support and is coloured!

EDIT: oh my dear - lynx is displaying the gentoo forum really badly. you MUST try w3m, w3m is displaying the page similar to the layout with a graphical browser

----------

## Sade

Thx for the tip on w3m, this is a nice browser. I think u should keep in mind that having a download of the script makes your job harder, because u have 2 update both scripts every time something changes. Copying with w3m is peanuts. So u might want to considder removing the link.

[EDIT]

just wondering: is the ccache also stored in the archive? it would be nice if u had the choise.

----------

## BlinkEye

 *Sade wrote:*   

> just wondering: is the ccache also stored in the archive? it would be nice if u had the choise.

 

well, i gave some hints under "2. How to customise". i don't know where you put your ccache. check the variables, if it is a subfolder of a folder listed in the exclude variables add it to $default_include_list. if not put it either in $default_exclude_list or $custom_exclude_list (i suggest in $custom_exclude_list).

----------

## Sade

i will try, i think i can complete that puzzle.

----------

## d_adams

Just tried this one out, and I must say, very nice work indeed. One of my kid's boxes had to be reloaded, so I used my backup to install gentoo on their box, instead of suse or windows, as it had both and was getting pretty messed up. I used the 1.2 version initially and got the console display error. I'm doing the 1.4 version now, and it's a lot better than doing a stage 1 or 3 install on a athlon xp 1900. Just need to create a user account for her and I'll be done. I set the backup file on my internal ftp, and used links from the live cd to get the file, and then untarred it. Very quick way to install gentoo.

Thanks a million for this one.

----------

## DNL

Just want to drop in and add my thanks... Getting ready to try some stupid things for the sake of my own stupidity, and did a quick search for backup and found this. Excellent, I was hoping I wouldn't have to write my own. Thanks a lot for sharing this. Gotta love open source.

----------

## Jerri

 *Sade wrote:*   

> when i select copy and paste, i get a lot of spaces in my file, and i don't know how not to select them, or smartly remove the spaces.

 

cat file | sed '/^$/d'

check this out for more tricks...

----------

## DNL

Well, unless I'm missing something, restoring this backup worked perfectly. Thanks again!

Highly recommend n00bs run this before doing anything they are unsure of. Could have used this before my first emerge --deep --newuse world...

----------

## BlinkEye

thanks for the feedbacks. i'm glad it's working for you guys too ...

----------

## Uncle_Psychosis

Hi

This looks like a great script, thanks! 

Could someone please help me out a little- I'm a bit confused as to how to use the backup once I've got it. What I want to do is use the stage4 to copy my system onto a new HDD. What do I need to do? 

Here is what I *think* needs to be done:

boot from live cd into new hard drive, as if installing from scratch

fdisk accordingly

make filesystems

activate swap

mount /mnt/gentoo and /mnt/gentoo/boot in the right places

cd /mnt/gentoo

extract stage4

install portage snapshot [as per instructions in handbook]??

chroot

emerge sync

configure kernel?? 

configure /etc/fstab accordingly

configure Grub

Install system loggers and tools??

Create passwords and users

Have I missed anything or done anything unnecessary? Or do I just follow the handbook instructions, but replace their stage tarball with my own? Sorry for the silly questions! 

Thanks

Sam

----------

## Jerri

 *Quote:*   

> configure kernel??
> 
> configure /etc/fstab accordingly
> 
> configure Grub
> ...

 

if your backing up a working install, these should already be taken care of.  You might have to update your MBR if it was damaged... but everything else whould have been configured before you backed it up.

Basically, this whole process takes a snapshot of your system, as is (mind you, everything that is created when you boot, like sys/* and proc/ are excluded).

----------

## Uncle_Psychosis

Ok, thanks. I just tried it, and it complained that my .bz2 file was corrupted. Boogar. Will try again tommorrow.

Thanks

Sam

----------

## mascanho

Damn, what a usefull script very good indeed, tanks man it realy saves a lot of work and time  :Razz:  , tested it here and it works great  :Cool: 

----------

## Asmdroid

Hi, I have two systems a laptop and a pentium 4 driven desktop workstation. I'm wondering if the CFLAGs I used on my main Gentoo workstation (march=pentium4) are p4 optimized CFLAGS and if this means they will traumatise (or not) my i686 Celeron laptop system due to both processors being compatible with the same architecture somewhat?Last edited by Asmdroid on Fri Apr 08, 2005 2:05 pm; edited 2 times in total

----------

## Asmdroid

 *Uncle_Psychosis wrote:*   

> Ok, thanks. I just tried it, and it complained that my .bz2 file was corrupted. Boogar. Will try again tommorrow.
> 
> Thanks
> 
> Sam

 

Strange, me too, I might try tar.gz as space isn't an issue on my external hard drive. I'm guessing that bz2 doesn't find large achives attractive? Perhaps tar.gz will perform better. Integrity check failure.

"Unexpected EOF (End Of File)"

----------

## BlinkEye

 *Asmdroid wrote:*   

> Hi, I have two systems a laptop and a pentium 4 driven desktop workstation. I'm wondering if the CFLAGs I used on my main Gentoo workstation (march=pentium4) are p4 optimized CFLAGS and if this means they will triaumatise (or not) my i686 Celeron laptop system due to both processors being compatible with the same architecture somewhat?

 

i'm afraid  but that won't work. 

 */etc/make.conf wrote:*   

> 
> 
> # Host and optimization settings
> 
> # ==============================
> ...

 

----------

## BlinkEye

 *Asmdroid wrote:*   

>  *Uncle_Psychosis wrote:*   Ok, thanks. I just tried it, and it complained that my .bz2 file was corrupted. Boogar. Will try again tommorrow.
> 
> Thanks
> 
> Sam 
> ...

 

could you guys specify your problem? how big is your stage4? i never had a tar.bz2 failing on me, although i've started to use tar.gz because of the time gaining. there is a limit to the hole size of the archive (filesystem dependent) - but the creation for the stage4 would stop while creating the tar, so, this isn't the problem

----------

## Asmdroid

 *BlinkEye wrote:*   

>  stage4? i never had a tar.bz2 failing on me, although i've started to use tar.gz because of the time gaining. there is a limit to the hole size of the archive (filesystem dependent) - but the creation for the stage4 would stop while creating the tar, so, this isn't the problem

 

The file that I made was just over 1GB (1.1). I originally stored it to the default directory in the script then transferred across to my USB hard-drive, this took a fair amount of time after continuously sending parts then caching. This could be a problem with my hardware, I'm not sure. The file did include a few useless directories also, CCACHE for example, which isn't appropriate.

----------

## BlinkEye

do you still have the tar.bz2? could could you experiment further (like extracting ? it doesn't matter if you've inlcuded the ccache folder or not, a tar.bz2 is a tar.bz2 and MUST work. 

EDIT: i'm just remembering that i had a lot of errors lately extracting some zipped files, so, did it just break or was it a warning/error at the end of the extracting? if possible post the error message

----------

## Mr Faber

Thanks, this is a really great and easy to use script. It makes life much easier  :Smile: 

Maybe an option for incrementell backups is usefull for future versions.

cu

Mr Faber

----------

## devils

Thanks alot man for doing this very cool script. 

Exactly what i have been looking for...

/devil

----------

## Uncle_Psychosis

 *BlinkEye wrote:*   

> do you still have the tar.bz2? could could you experiment further (like extracting ? it doesn't matter if you've inlcuded the ccache folder or not, a tar.bz2 is a tar.bz2 and MUST work. 
> 
> EDIT: i'm just remembering that i had a lot of errors lately extracting some zipped files, so, did it just break or was it a warning/error at the end of the extracting? if possible post the error message

 

I'm afraid I binned the file so I can't be much help. It was about 3.5Gb because I'd also backed up my /home. 

I've managed to restore my system onto a new hard drive using this, but I've ran into a few problems. For instance, I was getting a "Error opening terminal: Eterm." error everytime I tried to run nano, so I tried to fix this by re-emerging ncurses (which seems to be a fix for this sort of problem). However, ncurses fails with: 

```
Could not run/locate "g++"

make[1]: *** [../obj_s/cursesf.o] Error 1

make[1]: *** Waiting for unfinished jobs....

Could not run/locate "g++"

make[1]: *** [../obj_s/cursesm.o] Error 1

make[1]: Leaving directory `/var/tmp/portage/ncurses-5.4-r5/work/ncurses-5.4/c++'

make: *** [all] Error 2

+ diefunc src_compile 84 2 'make failed'

+ local funcname=src_compile lineno=84 exitcode=2

+ shift 3

+ echo

+ echo '!!! ERROR: sys-libs/ncurses-5.4-r5 failed.'

!!! ERROR: sys-libs/ncurses-5.4-r5 failed.

+ echo '!!! Function src_compile, Line 84, Exitcode 2'

!!! Function src_compile, Line 84, Exitcode 2

+ echo '!!! make failed'

!!! make failed

```

Any ideas? I saw when I unpacked my stage 4 that it complained about not being able to hardlink a lot of files (don't ask which, it was scrolling way too fast!). Could this be the source of the problem? Do I need to just do an emerge -uD system/world to fix it?

Cheers

Sam [/code]

----------

## feld

thank you SO much for this script. I just backed up, wiped my system, and tested the restore. IT WORKED FLAWLESSLY.

I really REALLY appreciate all the hard work you put into this backup script. Finally I have something easy and reliable enough to make me want to backup nearly every day  :Very Happy:  (ok ok, i'll keep it to once a week!)

thanks again!

-Feld

----------

## BlinkEye

 *Uncle_Psychosis wrote:*   

> 
> 
> I've managed to restore my system onto a new hard drive using this, but I've ran into a few problems. For instance, I was getting a "Error opening terminal: Eterm." error everytime I tried to run nano, so I tried to fix this by re-emerging ncurses (which seems to be a fix for this sort of problem). However, ncurses fails with: 
> 
> ```
> ...

 

ncurses complains about not being able to run/locate the C++ compiler, which is bad. so i guess you can't emerge anything. i don't know how this happened but could it be due to your personal setup? have you had any hardlinks at all? what does 

```
# emerge --info

# gcc -info
```

show?

the only option for hardlinks (if you have any) i saw in tar is

```
--check-links

              warn if number of hard links to the file on the filesystem mismatch the number of links recorded in the archive
```

so, you could change 

```
# options for the tar command 

 tarOptions="--absolute-names --preserve-permissions --totals --ignore-failed-read --verbose --file"
```

to

```
# options for the tar command 

 tarOptions="--absolute-names --preserve-permissions --totals --check-links --ignore-failed-read --verbose --file"
```

but you need to do that only if you're working with hardlinks and i don't know if it will solve your problem or not. you're the first complaining about this.

----------

## Uncle_Psychosis

 *BlinkEye wrote:*   

> 
> 
> but you need to do that only if you're working with hardlinks and i don't know if it will solve your problem or not. you're the first complaining about this.

 

I don't know. I know that I have personally never created any hardlinks...

Of course, the reason I'm doing this restore in this first place is because my old hard drive is on the way out (system just freezes and the logs show a whole load of i/o errors). It might well be that that is the cause of the problem. I did manage to fix this problem by doing a 

#emerge --usepkg gcc

#emerge ncurses

Other than that, everything seems to be in tip-top condition. Thanks very much for the script---you've saved me hours of work!

Sam

----------

## aschueler

How do you recombine the split bzipped tar files?  man split doesn't seem to tell me unless I am missing something.

Thanks for a great program that I hope I will never need!

----------

## BlinkEye

```
cat amd64-stage4-10.04.2005-minimal.tar.gz_* > amd64-stage4-10.04.2005-minimal.tar.gz
```

or

```
cat amd64-stage4-10.04.2005-minimal.tar.bz2_* > amd64-stage4-10.04.2005-minimal.tar.bz2
```

of course you may enumerate your splitted files instead of using an asterix

----------

## jbannon

Nice script, well presented and easy to understand. Works fine for me. I haven't been brave enough to wipe my system and try a full restore but listing it certainly seemed fine (no reports of any corruption). Only mods I made were the placement of the backup file on my external iomega drive and to include /usr/portage/distfiles (I wanted this because some packages I had to manually download). One mod I will do is to create the backup on the fast disk and then move it to my iomega as running bzip on that is mighty slow. I'll also use it as a basis for doing some others, like a daily backup of my data and so forth.

----------

## javock

Hey nice script...

I found it usefull, but, correct me if I am wrong, to make the backup, it first creates the tar, and then compresses it right?

I can't afford to duplicate the space of my instalation, so I made some corrections to the script.

First, as you clearly need some files on the excluded directories, and tar wont accept to exclude the directory and include some file, I create a list with the excluded files, on this list I omit the files to be included on the tar. Then I pass the -X option and this created file to tar and modify the script to pipe the result from tar to gzip/bzip2...

PM me if you are interested on the sugested behavior...

Regards.

----------

## slydini

 *Quote:*   

> Strange, me too, I might try tar.gz as space isn't an issue on my external hard drive. I'm guessing that bz2 doesn't find large achives attractive? Perhaps tar.gz will perform better. Integrity check failure. 
> 
> "Unexpected EOF (End Of File)"

 

I have seen bad media or certain brands of cdr's cause the above "Unexpected EOF" type error. I have also seen the file size not be correct when doing a long listing on a linux machine (shows 20Mb instead of 1G) but read the correct file size on a windows machine. The only difference being a change in media to fix the problem and be able to read the tar.gz files on the linux machine. BTW, I was able to extract the needed files from the seemingly bad media by copying the tar balls to an nfs mount from a windows machine to a linux machine. Then I was able successully extract the files.

What type of cdr's is everyone using? I have had good luck with the sony brand.

----------

## slydini

Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.

----------

## BlinkEye

 *javock wrote:*   

> Hey nice script...
> 
> I found it usefull, but, correct me if I am wrong, to make the backup, it first creates the tar, and then compresses it right?
> 
> I can't afford to duplicate the space of my instalation, so I made some corrections to the script.
> ...

 

thanks for your pm. i had a similar version once. i don't like the fact of creating first the stage4.tar and then zipping it neither. but i chose transparency and simplicity for the script as others are using it too. give me some time, i'm testing your idea and maybe will change the script. a tar gets so big i almost have no choice ...

 *slydini wrote:*   

> Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.

 

don't worry, i haven't forgotten something. what he meant is the issue about tar not backing up files which are in a subfolder of an excluded folder

----------

## Coume

Hello,

I just come accross this script but have no time right now to test it :/

But after reading the thread, I'm wondering one thing.

couldn't it be possible to generate directly backups CDs which include the LiveCD on the first one, so it could be possible to create an automated restauration script? Cos' the HUGE advantage I see with this script, is to recover a fucked up system while travelling and not having Internet access.

So if it was possible to only carry the backups CDs and knows that everything is in there... It would be perfect.

Just my 2 Euro cents

Ludo

----------

## javock

 *slydini wrote:*   

> Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.

 

As blinkeye said... those files are not left away in the script, it is a matter of implementation... tar does not allow you to use --exclude=<dir> and then try to explicity include a file that exists inside the excluded dir. So right now, the script first creates a tar with these files, and then appends the rest of the backup... simple, but as always it has a trade off, it first creates a full tar of the system (in my case around 8G  :Shocked:  ) writing it to disk *and* afterwards it compresses it. For me this was not an option as I cannot spare another 8G  :Rolling Eyes:  so I prefered to pipe the tar to gzip and only use the disk space to store the backup until I got to write it to a DVD.  :Laughing: 

The solution was to create an exclude file with where you explicitly list the files to exclude, and in creating this file you /omit/  :Wink:  the files you want to keep. Not so simple, but a little more efficient. 

 *BlinkEye wrote:*   

> but i chose transparency and simplicity for the script as others are using it too. give me some time, i'm testing your idea and maybe will change the script. a tar gets so big i almost have no choice ... 

 

As you can see my sugestion  :Cool:  is being taken into consideration... I hope it makes it to the stable branch  :Laughing: 

----------

## feld

If anyone else is having problems compressing with the bzip2 method please take the .tar and attempt to manually bzip2 it with the -s (--small) switch. I was having problems not being able to make it past 400mb and currently with the --small switch I have made it to : 1.3gb (almost done). The speed compressing also appears to be the exact same.

I guess this is supposed to be more memory-friendly and not thrash and stress your RAM so much.

I know my RAM isnt faulty cuz i just tested it with memtest86+ not too long ago... and I have plenty (1gig).

-Feld

----------

## BlinkEye

 *feld wrote:*   

> If anyone else is having problems compressing with the bzip2 method please take the .tar and attempt to manually bzip2 it with the -s (--small) switch. I was having problems not being able to make it past 400mb and currently with the --small switch I have made it to : 1.3gb (almost done). The speed compressing also appears to be the exact same.
> 
> I guess this is supposed to be more memory-friendly and not thrash and stress your RAM so much.
> 
> I know my RAM isnt faulty cuz i just tested it with memtest86+ not too long ago... and I have plenty (1gig).
> ...

 

please not that the issue about bzip2 and broken bzip2 files has nothing to do with the script.

Feld just dropped me a note and he "confessed" that his bzip2 isn't working - so, the --small switch does no good. i don't know what your problem is - i just tested the script over some of my mp3 files (resulting tar was 25G) and after 2.4GB of the resulting tar.bz2 i gave up. bzip2 is working flawlessly here and obviously is able to handle large files. i suggest you open a new thread about your issue ...

----------

## BlinkEye

 *Coume wrote:*   

> Hello,
> 
> I just come accross this script but have no time right now to test it :/
> 
> But after reading the thread, I'm wondering one thing.
> ...

 

if you backup the stage4 on another media than the to-be-fsck-harddisk you are set. boot to your fsck system with a regular livecd (whatever that might be) and untar the stage4. i thought about the solution of creating a livecd, but it's simply not worth the effort. i don't have a DVD burner yet and a stage4 of <670MB is simply very small. even if i had a DVD burner i don't gain anything if i'd create a livecd with the stage4: you get the same result just burning it on a cd/dvd and using another livecd. someone posted a link about creating a bootable livecd - so, if you really want to do it, just follow the link. i created a custom livecd, so, it's working. but if i fsck my harddisk/sytem i'd want a current stage4, so, i burn the cd/dvd if in need.

----------

## Coume

Erm, so u really just have to untar the file and you are set??

Sounds too good to be true...  :Smile:  But it will keep in memory, rights, user, etc. ? everything?? no need to chmod thingie, ?

Ludo

----------

## BlinkEye

 *Coume wrote:*   

> Erm, so u really just have to untar the file and you are set??
> 
> Sounds too good to be true...  But it will keep in memory, rights, user, etc. ? everything?? no need to chmod thingie, ?
> 
> Ludo

 

i mentioned that in "5. What now". it's not too good to be true, it IS true.

----------

## ctachta

Everyone,

Can I ask something stupid?  :Embarassed:   :Very Happy: 

When the script is interrupted with ctrl-c does it leaves everything somewhere at the hard disk to float. Is that  right? Where is that at tha hard drive?

Thanks a lot,

ctachta

----------

## BlinkEye

did you change something? if not - it's in /mnt/backups/stage4/

----------

## ctachta

No I did not. I told you it was stupid question I just figure it out just now.  :Very Happy: 

Thanks VERY VERY much for the script.  :Very Happy: 

----------

## ctachta

Guys,

I did try to make the script perform the operation silently  by redirecting the output to a file like this

sh mkstage4.sh > stage4.log

and I've got this error/warning

```

Total bytes written: 348160 (340KiB, 93KiB/s)

tar: /var/lib/dbus/system_bus_socket: socket ignored

tar: /var/run/mysqld/mysqld.sock: socket ignored

tar: /var/run/cgisock: socket ignored

```

 is it safe to ignore it? What does this mean and what is caused from?

Thanks

----------

## BlinkEye

yes. this is safe to ignore. such messages result from files which have changed/were deleted while tar is backing them up.

----------

## slydini

A question for BlinkEye: Is there any way you can eliminate the zip command in your script? What is the purpose of zip? I found that in order to use your script on most of my machines it is nescessary to install zip first. That kind of blows the "out of the box" goal out the window.  I really like the script otherwise.

I have some older machines that never get updated at all that I would like to backup with your script but I can't because of the need for zip to be installed. These machines are not connected to the net in any way.

If not, it would be great if someone could create a live-cd that had zip and your script on it.

----------

## BlinkEye

well, $zip is a variable for either bzip2 or gzip - depending on your choice at the beginning. these are tools, i'd say every system has installed. at the beginning of the script i set a list

```
command_list="echo tar hostname date split zip bzip2"
```

which are the tools needed. 

ah i see - this is a typo. zip should be gzip. the command_list should be:

```
command_list="echo tar hostname date split gzip bzip2"
```

thanks. updated v.1.4.0 to 1.4.1

----------

## slydini

Now it makes sense. After looking at the script it looked like zip was a variable but I couldn't figure out why it was in the command list. So I checked on portage for a program called zip and sure enough there was one. I emerged zip and the script worked fine after that. So the last few days I have been trying to figure how to get that little zip program on my older "never been updated in almost 2 years" gentoo machines.

I should probably note for the beginners out there that this script or any other for that matter will give you the following error if you copy the script from a windows machine without converting the text file:

```
# ./stage4.sh

: No such file or directory.
```

You can use the dos2unix utility to convert any such files to the proper format.

```
emerge dos2unix

dos2unix stage4.sh
```

Not knowing the above caused great frustration for many hours on some of my older machines that were not on the network. I have a 40gig firelite usb drive that I use to Xfer files between machines. Anyways, just somthing to consider.

----------

## gdi2k

Just wanted to say thanks for an excellent script! I used it to convert my 2 identical hard drives into a RAID 1 array today, without the fear of having to reinstall gentoo again from scratch. It took me a few hours to shift all the data around, and get the disks synched, but it appears to have worked nicely  :Very Happy: 

----------

## thomasa88

I mounted a ntfs disc through samba and run the script.

When the tar file is about 2GB this error appears in the script and it starts bzipping.

```

./backup.sh: line 196: 24302 File size limit exceeded$tar_command_append

```

I suppose my tar isnt complete or?  :Confused: 

Edit.

Seems like smbfs has problems with making tars over networks:

https://forums.gentoo.org/viewtopic-t-130241-highlight-tar+size+limit+exceeded.html

----------

## slydini

Damn, I tried to restore and as it was restoring I noticed the leading slashes. So now I am lost as how to get the .tgz file to extract without writing over the currently running Live CD filesystem???? I can't chroot into anything because there is nothing there to chroot too. It's like a catch-22. It makes more sense to have backed up each main filesystem sepearately then use relative pathes so I could restore to the /mnt/gentoo and /mnt/gentoo/boot directory. See what I mean?

If there is another way to restore using the Live-CD and a separate Data CD I am all ears.

Here is the commands I used to restore from my data-CD after I repatitioned and formated:

```
mount /dev/hda5 /mnt/gentoo

mount /dev/hda3 /mnt/gentoo/boot

cd /mnt/gentoo

tar xzvpf /mnt/cdrom/host-stage4-16.04.2005-custom.tar.gz .

........After only 6.5 minutes

/.reiserfs_priv/

tar: .reiserfs_priv: CAnnot mkdir: Permission denied

/stage4.sh

/lost+found/

tar: Error exit delayed form previous errors
```

A backup isn't a backup unless you can restore it.

----------

## BlinkEye

 *slydini wrote:*   

> Damn, I tried to restore and as it was restoring I noticed the leading slashes. So now I am lost as how to get the .tgz file to extract without writing over the currently running Live CD filesystem???? I can't chroot into anything because there is nothing there to chroot too. It's like a catch-22. It makes more sense to have backed up each main filesystem sepearately then use relative pathes so I could restore to the /mnt/gentoo and /mnt/gentoo/boot directory. See what I mean?

 

no, i don't.

 *slydini wrote:*   

> Here is the commands I used to restore from my data-CD after I repatitioned and formated:
> 
> ```
> mount /dev/hda5 /mnt/gentoo
> 
> ...

 

well, sure it is. but you simply forgot that the above command extracts your stage4 to /mnt/cdrom. either copy your stage4 to /mnt/gentoo or use

```
tar xzvpf /mnt/cdrom/host-stage4-16.04.2005-custom.tar.gz -C /mnt/gentoo
```

which does extract your stage4 to /mnt/gentoo

----------

## slydini

I actually forgot to put the dot at the end of the tar command in my post to indicate the current directory. I did attempt to cp the tarball to /mnt/gentoo and then extract it but I got the same error. And just think about it a minute if you tar up your system using absolute pathes then / is the same root your running from the Live-CD!!!!!!! So absolute paths don't make any sense at all while booted to a Live-CD. The gentoo stage1-3 tarballs are done with relative paths which makes sense to me. However if there is another method for doing this I am ready to learn.

If anyone has the steps they use to restore using a regular gentoo-live-cd please post your steps. Creating the tarball with this script works great it's the restore steps that need to be clarified.

The error I am getting happens after about 6.5 minutes of extracted files scrolling up the screen then:

```
tar: .reiserfs_priv: Cannot mkdir; Permission denied
```

Here are the steps I am doing so far:

```
1. Boot off of Live-CD and repartition and create filesystems as necessary.

2. Reboot, use the option: gentoo docache

3. umount /mnt/cdrom

4. Remove the Live-CD and insert the CD with the backup info on it.

5. mount /dev/cdrom /mnt/cdrom

6. Restore MBR using: dd if=/mnt/cdrom/mbr.img of=/dev/hda bs=446 count=1

7. mount /dev/hda5 /mnt/gentoo

8. mkdir /mnt/gentoo/boot

9. mount /dev/hda3 /mnt/gentoo/boot

10. mount any other filesystems you have on your system.

11. cd /mnt/gentoo

12. tar xzvpf host-stage4-16.04.2005-custom.tar.gz .

13. cd /

14. umount /mnt/cdrom

15. remove backup cd

16. Reboot
```

Only trouble is it doesn't work for me.

----------

## BlinkEye

 *slydini wrote:*   

> Only trouble is it doesn't work for me.

 

thanks for the mini-howto. i'll put that in my initial thread. but how about actually trying a suggestion others gave? you may extract this stage4 wherever you want - if you do that INSIDE an existing directory you'll get 

```
tar: Removing leading `/' from member names
```

which means it doesn't use absolute pathes anymore. your directions can't be right as you haven't yet copied over the stage4 to /mnt/gentoo. either way, 

```
cd /mnt/gentoo 

tar xzvpf host-stage4-16.04.2005-custom.tar.gz . 
```

simply doesn't work, it should be

```
tar xzvpf host-stage4-16.04.2005-custom.tar.gz
```

if you have your stage4 in /mnt/gentoo and your current dir is /mnt/gentoo

if you're somewhere else use

```
tar xzvpf /path/to/stage4/host-stage4-16.04.2005-custom.tar.gz -C /mnt/gentoo
```

----------

## slydini

BlinkEye, thanks for your help. Just so you know I tried what others have suggested including what you have suggested and I and still get the exact same error as posted previously. In fact I removed the --absolute-paths option temporarily so I could test it that way but it still doesn't work. So at this point I simply do not understand what is going on. If the only thing you would change in my restore steps is cp the tarball in the /mnt/gentoo directory first before extracting then there is something very strange going on with my laptop and your script. I don't know what it is I only know that I cannot extract any files past that stupid .reiserfs_priv: Cannot mkdir: Permission denied.

I even chmod 777 gentoo with no luck. I will try to exclude the .reiserfs_priv directory from the restore and see what happens. Ok, now I have gotten past the error and it looks like everything else is restoring OK.  :Very Happy: I will examine the contents of that .reiserfs_priv directory to see whats going on. Wait a minute, while extracting the second tarball I got the same exact error upon restore. This is really weird. Let me see if it even will come up after a reboot. Yes, it seems to have booted OK so whatever that directory is it did not want to be restored.

----------

## BlinkEye

glad to hear it's working. i thought it was just an issue of where and how you tried extracting the tarball. anyway, i put up a new section: "5. Restore" in the initial thread - thanks to your help.

----------

## saskatchewan46

Thanks for this cool script.  I've have been needing something like this; especially after a recent bad ram scare.

I did have problems though when I wanted to exclude a folder with a space in its name (ie: "this folder").  I have tried adding "--exclude=/home/me/this\ folder/*" , "--exclude=/home/me/this folder/*", variations with ' ' and etc but no good.  I though maybe there was an issue with the For loop in your verify routine, but even when I hardcoded the folder to the tar_command_append, it still never excluded it.  Maybe does tar have issues with excluding "this folder"?  Probably just me though..

Anyways, long story short, rather then excluding the folder, I just excluded *.iso files from my tar.  Would adding an option like exclude_file_pattern="--exclude=*.iso" be useful to your script?

Thanks again.

----------

## Slimm

thanks. thanks to a backup made from this script, i was just able to recover from the aftereffects of a temporary bout of stupidity.

----------

## BlinkEye

 *saskatchewan46 wrote:*   

> Would adding an option like exclude_file_pattern="--exclude=*.iso" be useful to your script?

 

it most certainly would. good idea, thanks.

updated script to v1.4.2

----------

## Lucky B

I did a script to do something similar to yours but... how shall we say... more crude/hackish.

It actually just backs up only the stuff in your system profile, which, arguably, is all I need to recover =P

have a look, maybe you can implement it as a 5th option =P

https://forums.gentoo.org/viewtopic-t-325979.html

----------

## robgrady

For all those having difficulties with tar not allowing you to include a file or directory under an excluded directory, I think I found a solution.

```
# tar -cvvzf kernel.tgz --exclude=/usr/src/ /usr/src/linux-`uname -r`

tar: Removing leading `/' from member names

drwxr-xr-x root/root         0 2005-04-11 12:48:55 /usr/src/linux-2.6.11-gentoo-r5/

drwxr-xr-x root/root         0 2005-04-11 12:45:23 /usr/src/linux-2.6.11-gentoo-r5/drivers/

drwxr-xr-x root/root         0 2005-04-11 12:45:23 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/

drwxr-xr-x root/root         0 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/

-rw-r--r-- root/root     29845 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/radeon_monitor.c

-rw-r--r-- root/root     11897 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/mach64_accel.c

-rw-r--r-- root/root     20936 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/mach64_gx.c

-rw-r--r-- root/root       479 2005-04-05 10:19:17 /usr/src/linux-2.6.11-gentoo-r5/drivers/video/aty/Makefile

<snip>
```

Let me assure you, the text in the <snip> included only files from "/usr/src/linux-2.6.11-gentoo-r5".  Now, if you use "--exclude=/usr/src" or "--exclude=/usr/src/*" it won't work.  Admittedly, I have not tried this with more complicated commands, so maybe something else will break it, I'm not sure.  Hope that helps.

----------

## BlinkEye

 *GNU Tar wrote:*   

> -I include-file   Opens include-file containing a list of files, one per line, and treats it as if each file appeared separately on the command line. Be careful of trailing white spaces. Also beware of leading white spaces, since, for each line in the included file, the entire line (apart from the newline) will be used to match against the initial string of files to include. In the case where excluded files (see X function modifier) are also specified, they take precedence over all included files. If a file is specified in both the exclude-file and the include-file (or on the command line), it will be excluded.

 

important part: If a file is specified in both the exclude-file and the include-file (or on the command line), it will be excluded.

i just tried it again first with some small archives and then with the script. i don't see any logic when some files get backed up or not when a folder is both excluded and included. well, it simply doesn't work properly.

----------

## robgrady

Yep, I ran a few more test cases and I'm seeing oddities also.  I saw that part of the man page, I just don't believe that there isn't a way around it.

----------

## Sheepdogj15

very cool.  :Cool:  right now i'm using the script and it looks like things are going well. 

if i may make a suggestion, i've noticed that the portage directory gets a bit bloated. ATM it is an excess of 1.1GB on my system. it might just compact down well, but for some people where disk space is a concern, if they have to do a restore they might be better off d/ling a portage snapshot and then doing an rsync. 

also, you can shave hundreds of MBs (if not GBs) by excluding ccache's ... cache. to my knowledge these files are not necessary for emerge to work properly. i could be wrong though

so, what i suggest is that:

```
--exclude=/usr/portage/*

--exclude=/root/.ccache/*
```

... be added to custom_exclude_list (that way, people can choose whether or not to back these up).

what do you think?

----------

## thomasa88

THX DUDE  :Very Happy: 

The script worked perfectly

what I did:

backup:  :Wink: 

- bz2 using the script

- cp bz2 to other computer

destroy:

(dont do this if u dont want to restore:  :Rolling Eyes: )

-time: 23:52

-try to resize ext3 doing following

-convert to ext2

-cant resize so I decide to extend partition with fdisk (putting on same startblock)

-disk is mountable

-cant run resize, trying fsck, cant find superblocks

-testing diff superblock positions, cant find any

-try to find some way to search for them on google (time: 1:00)

-dont find any, but wait: mke2fs -S -b 4096

-just one error in that thinking: the superblocks are pulled sidewise, it overwrites file data, yes! :S (should have fdisked it back  :Razz: )

restore:

-booting with livecd

-fdisk

-no samba :S  :Razz: , but it got ssh so I able to use sftp

-putting bz2 on temp-partition (going to be win sometime  :Razz: )

-mounting /mnt/gentoo and /mnt/gentoo/boot

-cd /mnt/gentoo

-tar xjvpf bz2

-chroot /mnt/gentoo /bin/bash

-mount -t proc proc /proc (necessary for grub?)

-emerge --sync

-edit /etc/fstab

-configure and install grub - cant find any device - looking into /dev no devs - udev?

(-trying to things, doesnt work)

-restart, in grub: setup (hd0,0)

-boot - ooops, havent changed grub.conf since partition  :Razz: 

-boot with right =root and change my grub.conf

-WEE  :Very Happy: , up'n'running

conclusion:

- i have a very destructive behavior

- got partitions with better sizes

btw, thx again  :Very Happy: 

Edit.

is it any possibility to get this script into portage? (just do emerge mkstage4 and get latest ver)

Edit.

went from 10GB disk usage to 4GB, nice to get rid of tmp-files and stuff  :Smile: 

----------

## BlinkEye

 *Sheepdogj15 wrote:*   

> so, what i suggest is that:
> 
> ```
> --exclude=/usr/portage/*
> 
> ...

 

i do that already  :Rolling Eyes:  . my ccache goes into /var/tmp/portage which is excluded and i do exclude /usr/portage/* already. maybe i forgot to do that in one version as i'm still playing around with some options. 

a new version is on the way. it does need some testing though, but the problem about first creating a tar and compressing it afterwards will be gone.

@thomasa88: i don't know how something goes into portage - and i haven't thought about that yet.

----------

## Sheepdogj15

weird. i wonder why my ccache goes in to /root instead.  :Confused:  and ah yes, i didn't see the line regarding portage before. my bad

----------

## jtaylor72

partimage works for me

----------

## BlinkEye

thanks for the input anyway - i'm working on a "suit everybody" solution now. i manually set the ccache variable as i don't want some temporary files in /root. the right location is /etc/make.conf:

```
CCACHE_SIZE="2G"

CCACHE_DIR="${PORTAGE_TMPDIR}/ccache"
```

where my $(PORTAGE_TMPDIR) is /tmp. this isn't ordinary neither, but portages tmp files are really tmp files for me so i don't want them in /var/tmp but in /tmp. of course, suit yourself. i think it didn't work for me though (check with "echo $CCACHE_DIR") so i added the following to /root/.bashrc: 

```
export CCACHE_DIR=/tmp/portage/ccache

export CCACHE_SIZE=2G
```

----------

## Sheepdogj15

i see now. and that isn't a bad idea, i'll probably change my setup for ccache too.

----------

## BlinkEye

udev users: please read the section 0.a. if you have any further info/failure/success/hints, please post it here.

----------

## ewildgoose

Just discovered this thread.  Nice script!

Just a couple of comments.  I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes.  Also I would put /var/log into the optional bucket as well.  For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them.

In practice of course one might do an (infrequent) system backup and a (frequent) data backup, and so I agree that they deserve to be kept in the optional bucket.

The lack of compression is a major bind for me though.  I don't have twice the diskspace available for the backup...  

One possible solution might be switching to "dar".  See http://dar.sourceforge.net/

DAR is basically an updated tar, but it has a few other advantages like recovery from corrupted archives (nice) and also backup of extended attributes (which are useful on a hardened server and a few other reasons).

I will be switching to DAR myself now, will drop a new command line here if it all work out.

----------

## BlinkEye

 *ewildgoose wrote:*   

> Just a couple of comments.  I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes.  Also I would put /var/log into the optional bucket as well.  For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them.

 

well, /usr/portage/* is in the default_exludes, meaning everything deeper is excluded too. what i'm trying to guarantee is a minimal working system backup. if you don't agree my policy change the script - it's easy and well documented. 

i got a version working which doesn't need to create a tar first and then compress the tar. as there are still some issues with udev i don't change the version yet (or everybody thinks this is a issue of my script). if someone wants the new version right away, pm me.

----------

## Sheepdogj15

 *BlinkEye wrote:*   

>  *ewildgoose wrote:*   Just a couple of comments.  I think that the default excludes should definitely list /usr/portage/distfiles, but I would put the whole of /usr/portage as one of the optional excludes.  Also I would put /var/log into the optional bucket as well.  For me both of these would be useful if I was doing a baremetal re-install, and I would prefer to keep them. 
> 
> well, /usr/portage/* is in the default_exludes, meaning everything deeper is excluded too. what i'm trying to guarantee is a minimal working system backup. if you don't agree my policy change the script - it's easy and well documented. 

 

yeah... i found it was pretty much as easy as moving --exclude=/usr/portage/* from the default list to the custom list. getting more granular with what all is in portage is similarly as simple. 

also on another point, i had a similar problem involving /dev on the restore, except i already had devfs set up as a fall back. what happened was i changed my partition structure, and thus i had to rerun Grub. however, it couldn't find my hardware. well, it took a bit of time before i recalled the script excluded the dev filesystem. i found that the easy solution to getting that working is, from the LiveCD (outside the chroot), mount dev like this:

```
mount -t devfs none /mnt/gentoo/dev
```

and then chroot. 

I just thought i would mention it in case other people have trouble with this.

----------

## bigfunkymo

IIIRC, the udev HOWTO mentions needing to have /dev/console and /dev/null.  perhaps you don't need to backup all of /dev, maybe just those 2 files

----------

## BlinkEye

the script does that. i don't backup /dev at all (only /dev/console and /dev/null). it's not that simple.

----------

## bigfunkymo

perhaps there are other /dev files that should be part of the specific inclusions?

If we had more information on why the systems using udev are failing, we (the forum community) might be able to provide a little insight.

Do you think it would hurt anything to remove the /dev exclusion?

Also along those same lines, for those who have a "/boot/boot -> ." symlink.  Would it help to add:

```
--exclude=/boot/boot
```

----------

## BlinkEye

i tried that of course. it doesn't work neither. i don't understand udev yet, but it doesn't use a device tarball (at least not if you follow the udev guid on gentoo.org). but i figured it out (at least for my case): to get rid of those nasty errors about not finding the device (/dev/sda1) i added the following three lines to /etc/init.d/checkroot:

 */etc/init.d/checkroot wrote:*   

> start() {
> 
>         local retval=0
> 
>         if [ ! -f /fastboot -a -z "${CDBOOT}" ]
> ...

 

this will create the necessary nodes before actually trying to mount non-existing ones. i can't test it with a raid system at the moment, but i booted my laptop and my amd64 raid0 system with my external usb harddisk. i bet you would have to do exactly the same thing if you try to boot a raid system which uses /dev/md*, i.e. just replace /dev/sda above with /dev/md.

after booting the system once with these lines you may afterwards remove them (i don't know why).

----------

## bigfunkymo

you can delete the recursive boot symlink if you install the bootloader with the right parameters.  Here is how: 

https://forums.gentoo.org/viewtopic-p-2353791.html#2353791

----------

## BlinkEye

or you just READ my thread, section "6. Warnings".   :Shocked: 

----------

## bigfunkymo

actually... I didn't read your whole thread  :Razz: 

so

I still feel smart!  :Shocked: 

----------

## BlinkEye

new version (v2.0.0) is out. it's faster now and no more additional (temporary) space is needed. the compression is done in-place. please be careful while customising - the script is using "find" now and hence has gotten more sensitive.

i tested it several times on three different machines. i didn't find any mistakes. please report any issues.

----------

## BlinkEye

minor fix (bzip2 options created a tar.gz stage4). new version: v.2.0.1

----------

## javock

Nice, that's what you get when you read the man pages... simple and useful.

Gonna try it later.

----------

## DrWoland

So wait, do I need to do anything special if I run pure udev or do the /dev/null and /dev/console backups take care of that?

----------

## DrWoland

```

hardass bin # stage4 

/bin/stage4: line 40: slashes.: command not found

/bin/stage4: line 55: be: command not found

/bin/stage4: line 62: folders,: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

/bin/stage4: line 98: [: missing `]'

/bin/stage4: line 99: -a: command not found

```

What?  :Sad: 

----------

## BlinkEye

no, for udev my script should take care of. it's just not that easy at the moment (depending on your hardware configuration) to restore a udev system backup without devfs. if you follow the instructions there shouldn't be any problem. 

i don't understand your problem. could you elaborate? what are you doing in /bin/stage4? have you put your backup in there? or is it just the script? if yes there happened probably some line breaking issues while copying/saving the script.

----------

## DrWoland

 *BlinkEye wrote:*   

> no, for udev my script should take care of. it's just not that easy at the moment (depending on your hardware configuration) to restore a udev system backup without devfs. if you follow the instructions there shouldn't be any problem. 
> 
> i don't understand your problem. could you elaborate? what are you doing in /bin/stage4? have you put your backup in there? or is it just the script? if yes there happened probably some line breaking issues while copying/saving the script.

 

Yeah that was right and I figured it out right away, just forgot to post back in here, sorry.

----------

## Alighieri

First off, thanks for the terrific tool! I've been using it to clone desktop machines around the lab. I get one setup the way I want and then make a system backup and copy it to another machine. A few /etc tweaks and I'm done. Home dirs are automounted off a server, so its all pretty automatic.

Here's the suggestion: I'm saving system images to an nfs mounted directory on the server. It would be hugely more efficient to use the '--gzip|--bzip2' tar flags rather than tarring uncompressed first and then zipping. It would reduce the data written to the mounted directory by 2/3's. Change the extension to '.tgz|.tbz'. I can hack my copy, but thought I'd pass that along.

I see now that it has already been done!...

Just tried the latest version. Works beautifully!

I've had no issues with udev. Not sure what that is about.

----------

## Alpo Nestori

More positive feedback coming, i just recovered with said script (v.2.0.1) and everything worked fine. Not a single error what so ever. Fluxbox (or xorg) lost my Alt-gr button for a while but it made a comeback with a fresh 'emerge -e world' with faster make.conf settings. Prelink is coming up. Windows is finally gone for good off my laptop. 

Excellent work with the script!

----------

## bingobob

bit wierd this....I love the script, very good indeed. I made a ful system backup and now I`m trying to recover a single file from the bz2 tarball.

I cant achieve this....what am i doing wrong...maybe i just have to wait a long time?

I even used bzunzipped it first but didnt help....I want to just pick and choose some files from this big tarball 2.7 gig.?

root@ttyp1[gentoo]# tar xvf franz-stage4-17.04.2005-custom.tar /etc/make.conf

/etc/make.conf

tar: Removing leading `/' from member names

----------

## slydini

Judging from your output it looks like it worked just fine. Once tar extracts the file you needed just hit ctrl-c to stop it from scanning the rest of the tarball.

----------

## oneeyedelf1

 *bingobob wrote:*   

> bit wierd this....I love the script, very good indeed. I made a ful system backup and now I`m trying to recover a single file from the bz2 tarball.
> 
> I cant achieve this....what am i doing wrong...maybe i just have to wait a long time?
> 
> I even used bzunzipped it first but didnt help....I want to just pick and choose some files from this big tarball 2.7 gig.?
> ...

 

I used the stage4 technique and hosed my kernel config, fetched it out of the previous dates via

```
tar xzvpf stage4.tar usr/linux/.config
```

note, I was using gzip compression (much faster, a little bit fatter), I think the script uses bz2, which means chnage the z in xzvpf to j, and note you omit the starting / from the file name

----------

## BlinkEye

 *oneeyedelf1 wrote:*   

> I used the stage4 technique and hosed my kernel config, fetched it out of the previous dates via
> 
> ```
> tar xzvpf stage4.tar /usr/linux/.config
> ```
> ...

 

you meant 

```
tar xzvpf stage4.tar.gz /usr/linux/.config
```

or  

```
tar xjvpf stage4.tar.bzip2 /usr/linux/.config
```

or  

```
tar xzvpf stage4.tgz /usr/linux/.config
```

either way, slydini is right: according to bingobob's output he did everything right and his make.conf has been extracted. it may take a long time - espacially if you use bzip2. but even for the tar - it's like find, it searches the hole tar until it is finished. if you're lucky and your /etc is at the very beginning of the tar you get /etc files in no time at all. but /etc files are probably not the first files which were written on your harddisk. so, after you see a line with the file/folder you want to extract hit control+c to break the extracting or else it will continue until the hole tarball has been searched through.

 *oneeyedelf1 wrote:*   

> note, I was using gzip compression (much faster, a little bit fatter), I think the script uses bz2, which means chnage the z in xzvpf to j, and note you omit the starting / from the file name

 

you think wrongly. the script makes a bzip2 or gzip depending on your choice. and the removing of the starting '/' is wrong. it MUST be there, that's why he gets this message 

```
tar: Removing leading `/' from member names
```

. if he searched like 

```
tar xvf franz-stage4-17.04.2005-custom.tar etc/make.conf
```

 he would have found nothing.

i bet all he did was not see his /etc/make.conf as it is not extracted as make.conf but as etc/make.conf. confused me several times myself. look out for the a etc folder where you executed the command!

----------

## bingobob

blink eye.... bang on the money...it had worked, for some reason (it was late and i had had a couple of beers), I was expecting to see it in my . directory, as you said though it was extracted to /etc/make.conf, I just didnt see it...I've now extracted my old .config successfully and rebuilt a kernel with it...so all is well and i now have the power to pick and choose from this mighty tar ball.

Still very impressed with this script, makes doing a live system backup v easy.

bb.

----------

## BlinkEye

i'm glad it worked. just one correction (for others). the make.conf was in your . dir - i.e. in your current directory, but not as ./make.conf but as ./etc/make.conf (mind the dot at the beginning). so if your in /mnt/backups/stage4 and extract /etc/make.conf from a stage4.tar.gzip your extracted make.conf will be in /mnt/backups/stage4/etc/make.conf.

----------

## mr.ed

Hey blinkeye!

Grats on a wonderfull script! Was playing around with previous versions, but this one really needs no updating IMO! It does it's job and more... I configged it to backup my WHOLE system to a seperate partition which gets unmounted when the backup is finished.... Even converted it to a cron-job to run once a week! As for the udev part: i don't get what the probs are, i recently b0rked my system really bad (accidentally unmerged glibc lol  :Shocked:  ) and i used the backup to restore my system and it worked!!!

At least now i can b0rk my system as much as i want (rofl) without having to worry if i have to re-install or not!!  :Twisted Evil: 

Thnx a lot for your hard work!  :Very Happy: 

----------

## dickeymeatballz

just tried to run this script and it told me permission denied, as root and as user. Am I overlooking something extremely obvious, or is my system just bugged out? Also, was that Ron Jeremy posting on this thread just now!?

----------

## DrWoland

 *dickeymeatballz wrote:*   

> just tried to run this script and it told me permission denied, as root and as user. Am I overlooking something extremely obvious, or is my system just bugged out? Also, was that Ron Jeremy posting on this thread just now!?

 

chmod +x scriptname

----------

## dickeymeatballz

Wow, i didn't expect anybody to reply that quick! Dr. Woland, your prescription worked like a charm, I thank you, sir. If I had known the forums replied so quickly I would not have spent all night trying to teach my computer to talk to itself with all those network backup solutions(or spent that month figuring out how to compile my gentoo kernel -they said i was crazy but it was worth it it LIVES HA HA HA HA HA!) ...sorry about that tangent, jut get a little carried away sometimes, heh heh. Thanx also to blinkeye for the script and the whole gentoo development squad for providing me with such a fantastic piece of software to tweak out on. -luv ya!

----------

## Master One

BlinkEye, please add the info to your WARNING, that grub has to be reinstalled into the MBR, after the "boot -> ." link has been removed! I had to find out the hard way today, when I rebooted my two servers, and was confronted with a "Grub Error 15" on both machines.

----------

## BlinkEye

thanks for that hint. now i know why i my grub.conf wasn't read on one box and i had to manually enter the boot/kernel options on every reboot although my settings were right (i almost lost my mind until the idea of re-installing grub struck me). i didn't get the error 15 though, just the grub prompt.

----------

## ewildgoose

I think that the error is that your kernel line params probably had an extra /boot in them.  The symlink makes that not a problem, but then we deleted it...

Remember that if /boot is on a seperate partition, then the real path on *that* partition doesn't need the /boot bit... So your command line to the kernel either needs a /boot or not depending on whether you use a seperate partition.  The symlink is a clever way around that.

I'm not sure why it has to be deleted though?  Tar had better be able to backup symlinks properly or all the rest of your system is buggered anyway?

----------

## BlinkEye

err, this is exactly what i've explained in the "6. Warning" section! well, you right about one thing, it shouldn't be necessary and in fact isn't anymore. i removed the hole warning section.

----------

## tfh

Great work.

What would be the best way to automate this script in order to call it from cron ?

----------

## BlinkEye

new version 2.1.0.

fixed issue with default_include_list and not recursively backed up folders. now, $default_include_list should not be changed, instead add your subfolders/files of excluded folder to $custom_default_list.

----------

## mr.ed

I've modified the script of blinkeye to suit my needs and to be able to run it as a cron. I'll post it here, but note that i`ve made some custom modifications, which may vary for some users. So don't blindly copy it and use it, cause it'll definitely not work well on other setups without some modifying!!!

Here it is:

 *Quote:*   

> #!/bin/bash 
> 
>  # Backup script for Gentoo Linux 
> 
>  # Copyright Reto Glauser aka Blinkeye 
> ...

 

----------

## Garibaldi

This is a nice script; however, there is a minor problem I thought I'd point out  :Smile:  .

You have 

```
path=`which $command | grep "no $command in"`
```

  The problem is, the "no blah in" message from which gets printed to stderr, not stdout, so your grep will never find anything.  I think a cleaner way to do your check would be something like:

```
for command in $command_list; do

    which $command > /dev/null 2>&1

    if [ $? -ne 0 ]; then

        echo -ne "\n\nERROR: $command not found! Check your \$command_list "

        echo -e  "and/or your \$PATH"

        exit -1

    fi

done
```

Andy

----------

## BlinkEye

 *Garibaldi wrote:*   

> You have 
> 
> ```
> path=`which $command | grep "no $command in"`
> ```
> ...

 

that's interesting, because i actually do try the things i post/provide. please try things first out yourself and suggest/correct afterwards in order not to confuse others. thanks.

----------

## DrWoland

 *BlinkEye wrote:*   

>  *Garibaldi wrote:*   You have 
> 
> ```
> path=`which $command | grep "no $command in"`
> ```
> ...

 

OH SNAP!!  :Shocked: 

----------

## luthias

This is fantastic, my thanks to you, sir

----------

## Garibaldi

 *BlinkEye wrote:*   

>  *Garibaldi wrote:*   You have 
> 
> ```
> path=`which $command | grep "no $command in"`
> ```
> ...

 

In fact I did test it.   You getting expected output doesn't prove the correctness the statement in question.

Consider the following simple examples:

The statement as you have it

```
$ which stupid | grep "no stupid in"

which: no stupid in (/bin:/usr/bin:...)
```

IF the "no stupid in" was getting processed by 'grep', the following should print nothing since we're inverting the match.  It does in fact print since the output is generated on the stderr stream (i.e. not processed by 'grep').

```
$ which stupid | grep --invert-match "no stupid in"

which: no stupid in (/bin:/usr/bin:...)
```

Now, the original statement, but redirecting stderr to stdout.  No surprise here since the pattern matches.

```
$ which stupid 2>&1 | grep "no stupid in"

which: no stupid in (/bin:/usr/bin:...)
```

And finally, proof positive that 'grep' is processing a string with "no stupid in" with stderr redirected

```
$ which stupid 2>&1 | grep --invert-match "no stupid in"

$
```

In the end, that's your name on the script.  You're free to distribute it with whatever errors you see fit.  Personally I think others would be more confused by the inclusion of statements with no effect than a post with a simple correction.  Sorry to bruise your ego, I just thought I'd lend a helping hand.

----------

## Lorijho

Thank you, the script work perfectly!

But why not keep the empty /tmp, which has special attributes (chmod 777 & chmod +t), by default ?

----------

## BlinkEye

 *Garibaldi wrote:*   

> In fact I did test it. You getting expected output doesn't prove the correctness the statement in question. In the end, that's your name on the script.  You're free to distribute it with whatever errors you see fit.  Personally I think others would be more confused by the inclusion of statements with no effect than a post with a simple correction.  Sorry to bruise your ego, I just thought I'd lend a helping hand.

 You didn't bruise my ego at all. Although i see your point you haven't pointed out a situation where this statements behaves wrongly. 

you initially said:  *Garibaldi wrote:*   

> You have 
> 
> ```
> path=`which $command | grep "no $command in"`
> ```
> ...

 which is wrong and logically led to the conclusion you didn't try it.

----------

## BlinkEye

 *Lorijho wrote:*   

> Thank you, the script work perfectly!
> 
> But why not keep the empty /tmp, which has special attributes (chmod 777 & chmod +t), by default ?

 

- changed it, thanks for the input. 

-new version does now validate files/folders in your $custom_include_list

new version: 2.1.1

----------

## Garibaldi

 *BlinkEye wrote:*   

> You didn't bruise my ego at all. Although i see your point you haven't pointed out a situation where this statements behaves wrongly. 
> 
> you initially said:  *Garibaldi wrote:*   You have 
> 
> ```
> ...

 

I did not mean to suggest your script behaved wrongly.  I merely saw a piece of code that, in context, did not function the way I suspected you wanted it to.  I considered that to be a "problem" whether or not it effected the outcome of the script or not.  I posted a suggested change that provided the same functionality without the "problem."

As I said in my original post, I think that the script is a nice piece of work.  Sorry for any confusion.

----------

## Master One

I just performed a complete stage4 system recovery on one of my machines. It's an UDEV only installation (udev-056), and there were no problems at all concerning /dev, so I think the warning for UDEV users can be kicked.

I had to recreate some dirs, which haven't survived the procedure (/tmp, /var/log/portage, /usr/local/portage), but everything else seems to be back and working. The only strange thing was, that unpacking the stage4 archive ended with 

```
tar: Error exit delayed form previous errors
```

also I couldn't see any errors, and all files see to have been restored.

Any idea, what this could be about?

----------

## Master One

BTW My suggestion for improvement: Change the date-format in the stage4-archive-name to `date +\%Y.\%m.\%d`, because otherwise you will pretty fast get a messup in your stage4 dir. This way you get the correct order when using "ls -la" in a console.

----------

## Zugot

i used your script last night to save my setup while i converted from ext3 to reiserfs.   Everything worked well.   Need to manually recreate /tmp.   /var/tmp and /usr/portage will be recreated with an 'emerge sync'

thanks.

----------

## gandulazul

From what I understand, this Stage 4 backup will backup all of the files on your system except those which are excluded by the script or by the user. Do you think it would be possible to take advantage of portage in minimizing the files that have to be copied? How much harder do you think it would be to only backup the files that portage did not install and/or modify? The restore process can then be a special minimal stage1 install that implements a basic portage system on the desired partition. 

Basically:

1. Backup all files not originally modified/created by portage.

2. Boot up to liveCD and make all of the partitions in your new system.

3. Install basic stage1 image to new partition.

4. Restore portage specific configuration settings from backup(make.conf package.* ...)

    Make any modifications to those portage config files taylored to your new system specs. 

    i.e. Change make.conf to reflect a system upgrade, or leave as is for a restore to the same system.

5. Emerge every package that was previously installed with corresponding USE flags.

6. Restore previous backup over new system.

    If System Upgrade, don't overwrite previously modified files that were created by portage unless they are configuration files(/etc).

    If basic System Restore, overwrite it all.

----------

## Sheepdogj15

since i use an external USB drive that is not auto-mounted, i set it up so it mounts and unmounts the backup backup partition itself (i.e. added "mount -v /mnt/backup" towards the beginning of the script, add "umount -v /mnt/backup" at the end, then create the directory for the mount point and specify it in me fstab).

what i like to do is use rsync to backup my home folder. if you don't need compression for it, you have a lot of files and only a few change at a time, and back it up more often than you do a full system backup (i'd recommend it), rsync is a good way to maintain a /home backup. 

if you mount your backup drive on /mnt/backup, this is the command with some suggested options: 

```
# rsync -azvu --stats --progress /home/ /mnt/backup/home/
```

don't forget the trailing slash on the /home/.

if you want it to delete old files from the backup that you no longer have in your working home directory:

```
# rsync -azv --delete --stats --progress /home/ /mnt/backup/home/
```

check the man page for rsync for details

----------

## Sheepdogj15

 *gandulazul wrote:*   

> From what I understand, this Stage 4 backup will backup all of the files on your system except those which are excluded by the script or by the user. Do you think it would be possible to take advantage of portage in minimizing the files that have to be copied? How much harder do you think it would be to only backup the files that portage did not install and/or modify? The restore process can then be a special minimal stage1 install that implements a basic portage system on the desired partition. 
> 
> Basically:
> 
> 1. Backup all files not originally modified/created by portage.
> ...

 

well, all configuration files should be in /etc, and all files that are not installed by portage should be in /opt or /usr/local. they may also have scripts or symbolic links in either /bin, /sbin, /usr/bin, or /usr/sbin.

the way i'd do what you want with the least administrative effort is just backup /etc, /opt, /usr/local, and also anything you've customized elsewhere (e.g. custom scripts in /usr/bin). you'd also want to document all the packages you installed from portage (i'd expect that portage creates a log of this or something, but i don't know enough about portage to say for sure). 

i know /etc will contain more files than what you may have modified, but i wouldn't worry about it: /etc is small.

i'm not sure if there is a better way. i'm sure there is and i bet it is more complex, but i think that's a good start. I'd suggest trying out a backup like my suggestion above, and then on an extra partition or seperate computer, similate the restore process you are looking to do. that way you can figure out for sure if this is adequate or if there are other files or folders you need to backup.

edit to add: also don't forget to backup your /home folder, and the config file for your kernel. it should be /usr/src/linux/.config

----------

## fnjordy

Shouldn't this include a partition dump (sfdisk -d), and raid (raidtab) & lvm (vgcfgbackup) configuration?

I would be looking for a backup that I can boot from a LiveCD, then run a script to restore.  So for instance with a new empty hard disk:

Create partition information (sfdisk < backup)

Create raid arrays (mkraid)

Create lvm volumes

Create swap / tmp systems

Restore remaining systems from archive

----------

## ppurka

 *Master One wrote:*   

> BTW My suggestion for improvement: Change the date-format in the stage4-archive-name to `date +\%Y.\%m.\%d`, because otherwise you will pretty fast get a messup in your stage4 dir. This way you get the correct order when using "ls -la" in a console.

 What if you do ls -la -v?

----------

## Sheepdogj15

BlinkEye, have you considered adding an integrity checking feature to your script? because i just attempted to restore for a stage4 that got corrupt. fortunately i have older backups (even despite that, i've been needing an excuse to rebuild my system anyways, so either way don't feel bad about this or anything). 

tar -xvv apparently can do this... i'm currently trying it on a known good archive, though, and if it hasn't frozen it's taking a long time.  :Confused: 

----------

## cummings66

Maybe I'm dense or something, but this script is a PITA to get working.  What I mean is that it's in text format and depending on your display dimensions you will get split lines, that means lots of work editing the script to get it working again.

Unless you're doing something other than cut and paste.  How are you guys managing to get this text script to work without lots of work, is there a link to it somewhere I missed?

I hate to complain, but I tried it after my editing of it and I think I must have missed something because it did not preserve all my permissions.  I'm trying it on a spare machine to test things out so no big deal, but really, how are you capturing this script?

----------

## cummings66

OK.  I managed to get w3m to capture it with little need for editing by using the -cols bit.  But, it still does not retain the correct ownership and permissions on some files.  For example, look at the /var/spool directory and you'll see that the news subdirectories are owned by root, yet in the original it's owned by news.  The odd thing is the . files used to keep track of news runs are owned by news.

I see the same thing with other directories as well, in essense everything in /var/spool is owned by root and that's not right.  I'm sure there are other areas I'm not aware of yet that have the same issues, so what caused that in the script and how can I be sure it really created the backup correctly?  The following is the command that is used.

creating the stage4 at /mnt/backups/stage4 with the following command:

(echo /var/log/emerge.log; echo /sys; echo /proc; echo /dev/console; echo /dev/null; find "/usr/src/linux-2.6.11-gentoo-r9"; find /HDSware /bin /boot /dev /etc /home /lib /mnt /opt /proc /root /sbin /sys /tmp /usr /var -not -name *.iso -path /home -prune -o -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /mnt/backups/stage4 -prune -o -not -type d -print) | tar --bzip2 --create --absolute-names --preserve-permissions --totals --ignore-failed-read --verbose --file /mnt/backups/stage4/stingray-stage4-03.06.2005-custom.tar.bz2 --no-recursion -T -

----------

## BlinkEye

ftp://blinkeye.ch/gentoo/mkstage4.sh

----------

## Sheepdogj15

weird, i had no trouble from just copying and pasting right from the thread.

----------

## cummings66

Maybe it's our webbrowser?  I use Mozilla and I had a lot of trouble.

I still have ownership problems that I don't think I can solve.  The biggest problem is that I run a news server and the ownership must be news.news for most of the files.  Well, the backup script will change that to root.root and that of course doesn't work.

Can somebody tell me why this happens?  As it stands it seems like I am the only person who has problems with this script working right.  By all rights it should be right, look at the posted options tar gets, it's right.  It must maintain news.news according to the encoding, and the decoding is done the same way as detail so it should work.

But, the only thing that maintains news.news are the .daily files, all else gets root.root when it concerns the news system.  Now, the other directories seem to be right, except for the mail and lp directories which also have root.root ownership.

I have also seen permissions change as well, and that I don't understand either.  It should work but just will not maintain ownership or permissions.

I should add I know the tar command works because if I do a tar -czpf /mnt/news.tgz /var/spool/news and extract it by doing a tar -xvzf /mnt/news.tgz it works.  The only diff is compression and also I cd to the right directory to restore.  So, what's different between how I do it manually and how the script does it, look at my args for tar in a previous post.

----------

## Sheepdogj15

hrmm...

```
# options for the tar command

tarOptions="--create --absolute-names --preserve-permissions --totals --ignore-failed-read --verbose --file" 
```

this is right from the script... make sure that there is a line likd the above in their, and that it is not mangled or anything, and make sure --preserve-permissions is in there. [edit] ignore this line, i just went back and you are getting "--preserve-permissions" in that output you posted.

it could be your version of tar. if that line is ok, try switching --preserve-permissions with -p. it means the same thing, but you never know.

if that doesn't work, emerge tar again (and cross your fingers  :Smile:  )

----------

## Sheepdogj15

 *Sheepdogj15 wrote:*   

> BlinkEye, have you considered adding an integrity checking feature to your script? because i just attempted to restore for a stage4 that got corrupt. fortunately i have older backups (even despite that, i've been needing an excuse to rebuild my system anyways, so either way don't feel bad about this or anything). 
> 
> tar -xvv apparently can do this... i'm currently trying it on a known good archive, though, and if it hasn't frozen it's taking a long time. 

 

i got this working. if someone wants to add error checking to their script, put this at the end:

```
echo "Checking tarball integrity"

tar -tvvf $stage4Name.$stage4postfix

echo "Integrity check: done. If you see any errors in the above, please run the backup script again"
```

I don't know jack about bash scripting, so it's rather crude, but it get's the job done. if someone wants to work it in as a feature, please do so. (it takes a long time to check, and all it does is list files without extracting them. if someone knows a quicker/better way to check integrity for .tar.bz2's, please let us know).

By the way, while i was testing for this addition, i actually had a tarball come back as corrupted. seriously, this ought to be added. it could just be my system, but still...

----------

## funkoolow

ehm, i'm quite scared to post about that, but.... the script won't run on my machine......

```
root@funkserver funkoolow # ./stage4.sh 

: bad interpreter: No such file or directory

root@funkserver funkoolow # sh ./stage4.sh 

: command not found: 

: command not found0: 

: command not found3: 

'/stage4.sh: line 15: syntax error near unexpected token `do

'/stage4.sh: line 15: `for command in $command_list; do
```

why does this happen? please don't flame at my ignorance  :Embarassed: 

----------

## BlinkEye

please download it again:

```
wget ftp://blinkeye.ch/gentoo/mkstage4.sh

chmod +x mkstage4.sh

./mkstage4.sh
```

what does 

```
echo $PATH
```

 output?

----------

## funkoolow

no hint for that, but it seems to work now. thanks for the light-speed support, man  :Very Happy: 

----------

## BlinkEye

 *Sheepdogj15 wrote:*   

> tar -xvv apparently can do this... i'm currently trying it on a known good archive, though, and if it hasn't frozen it's taking a long time. 

 

this is wrong. this command just untars a *.tar with verbose output. what you want is checking your stage4, which is either a tar.bzip2 or a tar.gz. both commands (bzip2 and gzip) have a integrity check command implemented.

 *Sheepdogj15 wrote:*   

> BlinkEye, have you considered adding an integrity checking feature to your script? because i just attempted to restore for a stage4 that got corrupt.

 

good idea - although i never had a stage4 failing on me. well, better be safe than sorry - it just takes a minute or two either way.

new version out: v2.2.0. added integrity check.

----------

## Sheepdogj15

 *BlinkEye wrote:*   

>  *Sheepdogj15 wrote:*   tar -xvv apparently can do this... i'm currently trying it on a known good archive, though, and if it hasn't frozen it's taking a long time.  
> 
> this is wrong. this command just untars a *.tar with verbose output. what you want is checking your stage4, which is either a tar.bzip2 or a tar.gz. both commands (bzip2 and gzip) have a integrity check command implemented.

 

yeah, i discovered it was supposed to be tar -tvvf ...

by trial an error in fact, meaning i made a mess on my backup drive  :Laughing: 

 *Quote:*   

> new version out: v2.2.0. added integrity check.

 

sweetness  :Smile: 

----------

## cummings66

Here's an example of what I mean, permissions and ownership change.  But, it's right in some directories such as the /var/log I added to the script.  The first one is from the machine the backup is from, the second is after I restored it to another machine.  Under those directories where the ownership is wrong it gets even worse file wise.

Has anybody else run into problems like this?  Can you look at your /var/spool directory and see how it's done?  Here's my version of tar.  I edited things a bit to shorten it.

stingray spool # tar --version

tar (GNU tar) 1.15.1

From the online machine...

stingray spool # ls -al

total 4

drwxr-xr-x  10 root  root   272 Feb 27  2004 .

drwxr-xr-x  16 root  root   432 Feb 22 07:36 ..

-rw-r--r--   1 root  root     0 Oct 24  2004 .keep

drwxrwx---   2 smmsp smmsp 1160 Jun  7 02:06 clientmqueue

drwxrwxrwt   2 root  mail   192 May  9 20:11 mail

drwxrwx---   2 root  smmsp 1256 May  9 20:35 mqueue

drwxr-xr-x  11 news  news   288 May  7 10:01 news

drwxrwxrwt   2 root  root    72 Jan 20 11:59 samba

Here's the mounted directory on the offline computer.

stingray spool # ls -al

total 0

drwxr-xr-x   8 root root 224 Jun  7 02:02 .

drwxr-xr-x  15 root root 408 Jun  7 02:02 ..

-rw-r--r--   1 root root   0 Mar 11 07:40 .keep

drwxr-xr-x   2 root root  72 Jun  7 02:02 clientmqueue

drwxr-xr-x   4 root root 120 Jun  7 01:54 cron

drwxr-xr-x   2 root root 192 Jun  7 01:54 mail

drwxr-xr-x   2 root root  72 Jun  7 02:02 mqueue

drwxr-xr-x  11 root root 288 Jun  7 02:02 news

drwxr-xr-x   2 root root  72 Jun  7 02:02 samba

Here's from the subdirectory news.  It should all be owned by news.  You'll notice the .keep is right, why?  Any guesses here?  I'm lost, it should work.

stingray news # ls -al

total 1

drwxr-xr-x  11 root root 288 Jun  7 02:02 .

drwxr-xr-x   8 root root 224 Jun  7 02:02 ..

-rw-r--r--   1 news news   0 May  7 09:02 .keep

drwxr-xr-x   2 root root  72 Jun  7 01:55 archive

drwxr-xr-x   2 root root  72 Jun  7 02:02 articles

drwxr-xr-x   2 root root  96 Jun  7 02:00 cycbuffs

drwxr-xr-x   3 root root 432 Jun  7 01:54 db

drwxr-xr-x   3 root root  96 Jun  7 01:55 incoming

drwxr-xr-x   2 root root 472 Jun  7 02:02 innfeed

drwxr-xr-x   2 root root  72 Jun  7 01:55 outgoing

drwxr-xr-x   7 root root 224 Jun  7 01:55 overview

drwxr-xr-x   9 root root 760 Jun  7 01:54 tmp

Now I manually did a short test using 

tar -czpf /home/cxxx/ov.tgz /var/spool/news/overview/

then extracted it using 

tar -xvzpf ov.tgz

and this is the result.  You'll notice it's correct.  Why can I do it by hand and it preserves permissions and the script does not, yet the script appears to run the command correctly?  I use the - because when I learned that was how you did it, now it's not necessary but does not change how it works.  I'm open to ideas here?

drwxr-xr-x   7 news news 224 Jun 12  2004 overview

----------

## BlinkEye

 *cummings66 wrote:*   

> Here's an example of what I mean, permissions and ownership change. Has anybody else run into problems like this?  Can you look at your /var/spool directory and see how it's done?  Here's my version of tar, it's from the current Gentoo system.
> 
> 

 

 :Embarassed: 

this is my mistake - and i'm sorry but have not been aware of it. after playing around i finally found the bug. i'll get back as soon is i've run some test. i'm sorry for the inconvenience. the problem is 

```
-not -type d
```

 of the find command. this prohibited a folder to be found by find and hence would not be processed to tar. as we backup files deeper in a folder anyway tar had to create a default folder with permission root:root. 

 :Embarassed: 

if you can't wait change (about line 87)

```
find_command="$find_command -not -type d -print"
```

 to

```
find_command="$find_command -print"
```

this is what i'm currently testing.

----------

## cummings66

At least I'm not insane.  It's really odd that I'm the only one who found this problem.   Thanks for the clue, I'll test the mod tonight.  It takes me about 4 hours to backup that machine so it'll be a little while before I can confirm your fix.

Thanks for the heads up, it really helps when the author looks at things since you know your program better than I dare say most.

----------

## BlinkEye

new version out: v.3.0.0. permission issues solved.

thanks to cummings66 for pointing me there. this was a nasty little bug. the code fragment i had to remove was a leftover from a very old version ...  :Embarassed: 

----------

## Master One

That's nasty, I just checked, quite some directories have the wrong ownership now. I already searched and fixed all in /var and subdirs.

Which dirs/files else were effected, and is there an easier way to fix that problem (since then, I already recovered two different machines with such an faulty stage4)?

----------

## cummings66

I'll leave the directories for the author to speculate on, but I did notice several with wrong permissions and ownership.

I believe this version does fix the problems I noticed so I restored to my Storageworks 4314r array and put it online.  I've been wanting to get the array online to replace my Proliant F1 and thus far it's doing fine.  I'd say this version is good to go as far as I can tell.

I modified the script to do what I want by adding the directories to the script's built in variables, but here's a couple wishes I have and why I think it's a good idea to make them universal.

1 - Allow the user using interactive to backup /var/log/*

     This allows you to restore a working system used for ISP duties, many directories are under there that are placed there during the install of certain programs typically used for ISP's.

2 - Allow the user using interactive to backup /usr/portage/*

     If you have many machines that sync off of this it's easier on all concerned to just be able to copy it over.  It's maybe 400 - 500 Megs if you keep the distfiles directory out of it, might consider a question on that directory.

3 - Allow the Interactive mode to backup only the /home/* and/or the /var/spool directories.

     For an ISP sometimes you want only the users webpages and email, you don't really care so much about the machine itself because you have either a means for a fresh install, or have it backed up.  On an ISP those 2 directories can get unbelieveably large.  Currently I use a tar command run each night for that, but this script would be nicer if it could do it as well.

As is, I like the script, it makes things easier for many.  I used to use a long tar command in the past to backup my systems, and it had to be modified for each machine I run it on, I like this script because it's close to universal and I think could be used on other versions of Linux as well.

----------

## BlinkEye

 *cummings66 wrote:*   

>  ... but here's a couple wishes I have ...
> 
> 1 - Allow the user using interactive to backup /var/log/*
> 
>      This allows you to restore a working system used for ISP duties, many directories are under there that are placed there during the install of certain programs typically used for ISP's.
> ...

 

well, thats possible. just add these folders to $custom_list. if you do a minimal backup they get exclude, if you do a custom backup may choose to include or exclude the files/dirs.

----------

## ptrain

One small suggestion would be to include the absolute path for GNU tar (i.e., /bin/tar) in the script for the tar_command variable. I had issues on an Xfce4 system, which includes star as a dependency (which then becomes the default tar), and the mkstage4.sh script would always complain about bad tar options.

----------

## helpmhost

First of all, I'd like to say that this is a great script. Thanks!!

I made a backup of a gentoo computer which I would like to use to create a 2nd gentoo computer. The hardware is a little different so I know that that could and most likely would cause some problems.

I followed the restore steps exactly. Everything worked. I reran /sbin/lilo to install lilo in the MBR using the /etc/lilo.conf. I verified that the lilo.conf and the fstab files were set up correctly (I checked for spelling errors multiple times) However when I boot, after choosing Gentoo in the lilo prompt, I see the following:

[code]

Loading Gentoo.................................................................

BIOS Data Check Successful

[/code]

After displaying that, the screen goes black for a second or 2 and then the computer reboots (I see the POST and everything).

Is this a kernel problem (ex: missing a needed driver)? I think the kernel has all the drivers I need for that computer. Shouldn't I get further than that in the boot process and get some sort of error message? Should I try genkernel?

Does anyone have any ideas?

----------

## Sheepdogj15

IIRC, if you are seeing "Loading Gentoo................................................................. " that means the BIOS successfully handed off to the kernel. meaning this is a kernel problem

so, get your liveCD out, load it in... do everything you need to get to where you can set kernel options. do a "make menuconfig" and verify that it's properly configured for your hardware.

if i were to guess, i think this computer has a different processor, and you have to reconfigure that. but check everything else as well.

----------

## helpmhost

Thanks a lot! Everything works now. The computer does have a different CPU.

Once again, this script is awesome. It saved me hours of compiling.

Thanks.

----------

## NocturnalCoffeeMan

I am using it right now, I already had to re-install when I used the -P flag instead of -p, that was a big mistake. Thanks for the stage4 script, it is what I have been looking for!  :Razz: 

----------

## Master One

I think, I need a little help. I have a weird problem on one of my machines. I want to exclude the /home/user/vmware subfolders (where the vmware-images are stored), but whatever I try, the backup-script nevertheless tries to backup those folders.

I have set the following:

```
# these files/directories are always excluded. don't add trailing slashes.

default_exclude_list="

/dev

/home/user/vmware/*

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

$stage4Location

`echo $CCACHE_DIR`"
```

On intended backup I want to include /home, so this is what it tells to use as backup command:

```
creating the stage4 at /mnt/rescue/stage4 with the following command:

(echo /var/log/emerge.log; echo /var/log/news; echo /var/log/portage; echo /mnt; echo /tmp; echo /sys; echo /proc; echo /dev/console; echo /dev/null; find "/usr/src/linux-2.6.11-gentoo-r9"; find "/home"; find "/root/.ccache"; find /bin /boot /dev /etc /home /lib /media /mnt /opt /proc /root /sbin /sys /tmp /usr /var -not -name *.iso -path /dev -prune -o -path /home/user/vmware/Windows_XP_Professional -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /mnt/rescue/stage4 -prune -o -print) | tar --bzip2 --preserve-permissions --create --absolute-names --totals --ignore-failed-read --verbose --file /mnt/rescue/stage4/stage4-2005.06.11-custom.tar.bz2 --no-recursion -T -
```

So everything seems to be ok, but it still goes on with the files in /home/user/vmware/Windows_XP_Professional!

Any ideas?

----------

## helpmhost

I have successfully restored the backup to another computer now, but I am having another problem.

When I pretend to emerge some things, I get errors such as the following:

[code]

QA Notice: has version() in global scope: eclass gtk-engines2

QA Notice: has version() in global scope: eclass gtk-engines2

QA Notice: sed in global scope: x11-themes/gtk-engines-2.6.3

[/code]

This just goes on for a while (it repeats those 3 lines over and over). Eventually, it works though.

Any ideas why this is happening and what I need to do to stop it? It takes a long time just to get past this phase.

I also got this with mod_php which I had not already installed.

I have another question. After restoring my backup, how does portage know which packages I have installed already? I did not include the portage directories in the backup. I just used the default in the backup script v.2.2.0 and included home and the kernel source. Just to test things, I pretended to emerge gnome (which was already installed when I backed the system up) and many (not all) packages are coming up a 'N' for new install. Did the gnome metapackage just add many packages recently or is there a problem with portage detecting installed packages.

Also, when I emerged mod_php (on the restored PC) it told me that it detected 2 versions of apache. I only ever installed 1 (version 2).

Any ideas for either of these questions/problems?

----------

## BlinkEye

 *helpmhost wrote:*   

> I have successfully restored the backup to another computer now, but I am having another problem.
> 
> When I pretend to emerge some things, I get errors such as the following:
> 
> ```
> ...

 

i've had this error about two years back once too. check the forums, maybe you find an answer (i didn't find anything on a quick view).

 *helpmhost wrote:*   

> I
> 
> I have another question. After restoring my backup, how does portage know which packages I have installed already? I did not include the portage directories in the backup. I just used the default in the backup script v.2.2.0 and included home and the kernel source. Just to test things, I pretended to emerge gnome (which was already installed when I backed the system up) and many (not all) packages are coming up a 'N' for new install. Did the gnome metapackage just add many packages recently or is there a problem with portage detecting installed packages.
> 
> Also, when I emerged mod_php (on the restored PC) it told me that it detected 2 versions of apache. I only ever installed 1 (version 2).
> ...

 

have you run 

```
emerge --sync
```

 since the restore? did you backup /var/db/*? this is portage's memory! for example do 

```
ls /var/db/pkg/net-www
```

i would have directed you to http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=1 but it lacks this path  :Shocked:  (i dropped a note). i need more specific infos to help you further. besides, 4 or 5 days ago a lot of gnome libs have been updated.

----------

## BlinkEye

 *Master One wrote:*   

>  ... 

 according to your find command you're lying  :Laughing: 

where are these lines come from:

```
find "/usr/src/linux-2.6.11-gentoo-r9"; find "/home"; find "/root/.ccache";
```

you still have /home in your $custom_include_list. its a good point though, i might try to catch such issues from within the script.

----------

## BlinkEye

 new version out: v.3.1.0.

see first post for the change log (quite a lot this time).

----------

## Master One

 *BlinkEye wrote:*   

>  *Master One wrote:*    ...  according to your find command you're lying  

 

BlinkEye, I am sorry, but I think I don't understand.

 *BlinkEye wrote:*   

> where are these lines come from:
> 
> ```
> find "/usr/src/linux-2.6.11-gentoo-r9"; find "/home"; find "/root/.ccache";
> ```
> ...

 

The intention is to make a full system backup including /usr/src/linux, /home and /root/.ccache, except /home/user/vmware/*

I looks to me, that the configuration is right, but nevertheless it tries to backup /home/user/vmware/*

Am I overseeing something?

----------

## Sheepdogj15

i've noticed this too. if you have something in one of the include lists and yet are trying to exclude something within the include, the include will override the exclude. 

i know because it set up to include /tmp, but i had the script building the tarball in /tmp/backup. i set it to exclude /tmp/backup, but i found it was still trying to back it up.

From what i can tell, anything that is not explicitely included or excluded is by default included into the tarball. so, the operative thing to do (unless Blinkeye finds a way to fix this) would be to just not explicitely include a folder where you are trying to exclude something within that folder. 

the only problem here is that you may still find that you want to have the option to include something or not.

----------

## BlinkEye

i'm working on it. i got a version with this feature up but want to run a full test again.

----------

## BlinkEye

 new version out: v.3.2.

It's now possible to add a folder to $custom_include_list while excluding a folder withing that folder. 

Example:

```

custom_include_list="/home/*"

custom_exclude_list="/home/foo/downloads"

custom_exclude_pattern="*.iso *.mp3"

```

now, if you choose to do a custom backup, you'll be asked to back up any folder within /home. if you say 'yes' for /home/foo it will nevertheless exclude /home/foo/downloads and any files ending with .iso and .mp3 as you specified these restrictions in $custom_exclude_list and $custom_exclude_pattern. 

please report any issues/questions.

[EDIT]

new version out: v.3.3

* hard coded find and tar.

* routine to verify and possibly suggest a path to the commands if one can't be found.

* added parameters. as a default, the script does not show the files processed by tar on stdout but solely the error (quite useful). the old behaviour can be attained by executing the script

```
mkstage4.sh -v
```

 or 

```
mkstage4.sh --verbose
```

* updated howto. some sections added, some changes made ...

----------

## kommissar

I just wanted to leave this post (six pages down) to let you know that this is a great tool; I've often wondered if I could just bzip my system.  I intend to use it and tell my friends about it.  Keep up the good work!

----------

## troyboy162

great script!!!! i have a identical system to my freind and we stole his install with your script and dumped it on my comp quite easily. 

i am having one problem... i get the "unable to open an intial console" message right before it turns on my swap partition. everything seems to work fine it just stops giving messages in the boot process and goes to gdm after a long pause where it would be normaly be spitting boot messages down the screen. 

anyone else having this problem?

is there a cure?

for infos sake this is a 2005 amd64 install

----------

## BlinkEye

this sure sounds like an udev problem. which version of the script did you use? /dev/null and /dev/console are backed up since quite some time (these are two devices udev needs before initializing). For solving the problem see the gentoo udev guide.

 in short

```
# mknod -m 660 console c 5 1

# mknod -m 660 null c 1 3

```

[EDIT] updated howto, added new section "3. Portability/Prerequesites"[/EDIT]

----------

## Jedidiah

Does anyone know will this work on a mac running OSX?

i have tried it on my linux machine, and i have need for a utility like this at work. the senario is this, we just got in 42 new macs that need to have identical setups, using the mac utils is out of the option becuase it takes way to long to transfer/copy the data from one machine to another. will this script work on a mac, so i can make one backup of the working machine and just unarchive it on everyother machine? will it work if i remove all of the folders from the default exclude list etc? I need to get a complete copy

thanks in advance

----------

## troyboy162

 *BlinkEye wrote:*   

> this sure sounds like an udev problem. which version of the script did you use? /dev/null and /dev/console are backed up since quite some time (these are two devices udev needs before initializing). For solving the problem see the gentoo udev guide.
> 
>  in short
> 
> ```
> ...

 

he used v3.0.0

i tryed the two commands you put and they didnt fix it. still its the only thing wrong with the install so i cant complain:)

----------

## iverson0881

hmm i think your server links are down and i'm not if that is the latest version or the one embedded into the forum is the latest.

just thought you'd like to know.

EDIT: nevermind the one embedded into the site is the latest one. very nice script man!! thanks

----------

## Pseud

Thanks for making our lives easy  :Smile: 

Just have one question:

The script resulted in a ~1.7G tar.gz file. After "splitting" filename.tar.gz at the end, I'm left with filename.tar.gz and filename.tar.gz_a

Is this the way the split command works? I was expecting filename.tar.gz_a and filename.tar.gz_b

So, I don't know which comes first when i cat the files to merge them back later.

Thanks!

----------

## BlinkEye

 *Jedidiah wrote:*   

> Does anyone know will this work on a mac running OSX?
> 
> i have tried it on my linux machine, and i have need for a utility like this at work. the senario is this, we just got in 42 new macs that need to have identical setups, using the mac utils is out of the option becuase it takes way to long to transfer/copy the data from one machine to another. will this script work on a mac, so i can make one backup of the working machine and just unarchive it on everyother machine? will it work if i remove all of the folders from the default exclude list etc? I need to get a complete copy
> 
> thanks in advance

 

i don't see why it shouldn't work. i can't test it because i don't have a mac. removing the folder from the default exclude list is an option.

----------

## BlinkEye

 *Pseud wrote:*   

> Thanks for making our lives easy 
> 
> Just have one question:
> 
> The script resulted in a ~1.7G tar.gz file. After "splitting" filename.tar.gz at the end, I'm left with filename.tar.gz and filename.tar.gz_a
> ...

 

command line: 

```
cd /mnt/backups/stage4

split --suffix-length=1 --bytes=680m amd64-stage4-2005.06.14-minimal.tar.gz amd64-stage4-2005.06.14-minimal.tar.gz_
```

which resulted in:  amd64-stage4-2005.06.14-minimal.tar.gz_a, amd64-stage4-2005.06.14-minimal.tar.gz_b. the original file wasn't removed. so,  amd64-stage4-2005.06.14-minimal.tar.gz_a is the first one. what happens if you split it from the command line? are you sure you haven't set the --bytes=XXXm wrong?

----------

## Pseud

BlinkEye,

I used the split command from inside the script (just uncommended it). I'll give it another shot and see what happens.

Meanwhile, I have another question/suggestion:

I saw the man page for the split command and there seems to be no option for not keeping the original file. This means if one intends to actually burn the backup onto CDs, just having to split the original into pieces will imply double the disk space requirement! (and this matters a lot for backups going into a few GBs). Is there a simple way around this? It would be great if the script itself could handle such a thing through an optional command line parameter ... generating a backup as a set of tar.gz files instead of one tar.gz file, each file limited to a reasonable size limit (like 690m, as you suggest). Is this do-able?

----------

## bigmatt

umm I get this error when running the script 

ERROR: /var/log/portage not found! Check your $default_include_files

and i dont really know what to do.

----------

## BlinkEye

just remove /var/log/portage from the $default_include_files. originally, it looked like:

```
# files/devices/folders, which need to be backed up (preserve folder structure). 

 # don't touch it unless you know what you are doing! no recursive backup of folders. 

 # use $default_include_folders instead. 

 default_include_files=" 

 /dev/null 

 /dev/console 

 /mnt 

 /proc 

 /sys 

 /tmp 

 /usr/portage 

 /usr/src 

 /var/log/emerge.log 

 /var/log/portage"
```

removing /var/log/portage results in:

```
# files/devices/folders, which need to be backed up (preserve folder structure). 

 # don't touch it unless you know what you are doing! no recursive backup of folders. 

 # use $default_include_folders instead. 

 default_include_files=" 

 /dev/null 

 /dev/console 

 /mnt 

 /proc 

 /sys 

 /tmp 

 /usr/portage 

 /usr/src 

 /var/log/emerge.log"
```

----------

## bigmatt

thanks alot worked fine  :Smile: .

----------

## NightTwix

thanks BlinkEye, this script is really useful and works fine after little customiziation.

just a quick reminder to everyone:

be sure you doublecheck that your archive is not corrupted.

Especially when you copy it to a windows-box via Samba and burn it on CD.

I ended up with a corrupt archive somehow...

The best thing is to make a md5sum after each copy process and compare it to the original.

Maybe the script could automatically generate a md5sum after making the archive

----------

## BlinkEye

well, the script verifies if tar/gzip/bzip creates a corrupted archive or not. i don't know why samba would cripple a file during transfer ...

either way, if you want a md5sum add 

```
echo "* md5sum archive:  `md5sum $stage4Name.$stage4postfix`"
```

to the end of the script.

----------

## Jedidiah

 *BlinkEye wrote:*   

> i don't see why it shouldn't work. i can't test it because i don't have a mac. removing the folder from the default exclude list is an option.

 

BlinkEye - it took a little bit of playing around but i think i got the script to work, for a backup at least (it took nearly 2 hours to run the backup, on a fresh install). however, copying it over to another disk is another story, the first time i untarred it, it stalled out with an error along the lines of cannot copy simlink, file exists (i think that is the error, but i don't recall). I am currently trying to untarr it again to see if i get the same results, if i do i will post the exact error message

also, another question, in one of my tests i had added the /System /Application and /Library direcotries to the default_include_folders_list. when i did an interactive backup, it copied those directoried first, then i noticed that it seemed like it was copying them again later, is this the case, or was it just verrifying that it had already coppied them?

thanks

----------

## Jedidiah

Ok it just errored out again running it on the mac, here is the error, any ideas how to overcome it?

```
tar: var: Cannot create symlink to `private/var': File exists

tar: Error exit delayed from previous errors
```

i was going to try using the "-h" option when i untarred it, but i wasn't sure if that would mean that i would not have any symlinks then.

any suggestions?

jedidiah

----------

## BlinkEye

 *Jedidiah wrote:*   

> also, another question, in one of my tests i had added the /System /Application and /Library direcotries to the default_include_folders_list. when i did an interactive backup, it copied those directoried first, then i noticed that it seemed like it was copying them again later, is this the case, or was it just verrifying that it had already coppied them? thanks

 

the idea about this $default_include_folders is to include folders listed in there and to make sure, they get backup-ed anyway. it happened that users added /var to the $default_exclude_list and hence portage's memory was wiped out (that's why /var/db and /var/cache/edb MUST be backup-ed). for your case, add /System, /Application and /Library additionally to the $default_exclude_list to prevent double backup. 

 *Jedidiah wrote:*   

> Ok it just errored out again running it on the mac, here is the error, any ideas how to overcome it? 
> 
> ```
> tar: var: Cannot create symlink to `private/var': File exists 
> 
> ...

 

did you first wipe out files/dirs on that partition you're trying to extract? if yes it may just be the behaviour explained above - a dir has been backup-ed twice and it now complains while extracting (which doesn't hurt).

on your previous post you said, that it takes two hours to make the stage4. if that's true this script is probably not the right thing for you as you are just backing up anything. the idea of this script is to exclude anything not necessary to save time and disk space. for this to work you need a thorough understanding of your system. well, you may use it either way or at least find use of the tar command.

----------

## Jedidiah

Thanks Blinkeye i added the /System /Library and /Applications folders to the default exclude list, i am running the command again so i will let you know in the morning weather or not it worked. the reason why i am using this script is because there seems to be a serious lack of tools for backing up an entire mac system (what i really need is an app that does what Norton Ghost does on the PC for a mac, this script is close enough). also, of the few apps that i have found, they all take just as long, if not longer, so the two hour thing seems to be a mac problem and not a problem with the script (a fresh install of osx 10.4 with microsoft office seems to take up roughly 26 gigs) so if you ask me, two hours to compress 26 gigs down to about 11 gigs isn't too bad (espesially in comparison with the two days it used to take on our os9 machines, now if it will only work)

thanks for your help, expect at least one more post from me tomorrow

jedidiah

----------

## BlinkEye

 *Pseud wrote:*   

> Meanwhile, I have another question/suggestion:
> 
> ...
> 
> It would be great if the script itself could handle such a thing through an optional command line parameter ... generating a backup as a set of tar.gz files instead of one tar.gz file, each file limited to a reasonable size limit (like 690m, as you suggest). Is this do-able?

 

good suggestion. it's done. 

 new version out: v.3.4

there's a new variable $split_options you may need/want to adjust. now, if you want to create a stage4 consisting of chunks the size you set in $split_options (instead of one big tarball) use the -s or --split parameter while calling the script:

```
./mkstage4.sh --split

./mkstage4.sh -s
```

this will prevent the script of first creating a huge tarball which you may need to split afterwards.

example: instead of a 2GB stage4.tar.gz you'll get (with default options and calling the script with --split/-s) 3 files:

stage4.tar.gz_a

stage4.tar.gz_b

stage4.tar.gz_c

[EDIT]fixed language[/EDIT]

----------

## Pseud

Coolio! Thank you! Will check out asap.

----------

## Pseud

BlinkEye,

I haven't tried the new script yet, but I just glanced through the code and have a small concern. As I understand it, the script accepts the -s option but still resorts back to using the split command. Wouldn't this mean that there's still double the hard-disk space usage? (because it's an intrinsic problem with the split command itself: when it splits a file into pieces, there's no way to avoid keeping the to-be-split file.) I gave it some thought and couldn't even think of a way of writing a split program (in C, for instance) that will avoid this doubling of storage.

What I suggested earlier was something like this:

The script should generate split tarballs on the fly: not by creating a 2G tarball and then splitting it into three 600m pieces and one 200m piece (for example), but by creating a second tarball as soon as there is more data than will fit in a 600m tarball (and so on for more than two tarballs, etc.)

This won't be trivial though, because there's no way of guessing how much data will fit into a 600m tarball because the amount of compression depends on the data itself! Also, this might cost us in terms of lower compression overall, but that's a trade-off if one doesn't want a mammoth tarball.

So, is there an intelligent way out? Sounds like an interesting problem, but I can't think of a solution right away.

----------

## BlinkEye

it works without first creating a big tarball. i'll explain the accordant command:

```
tar_command="$find_command | $tar $zip $tarOptions --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"
```

tar doesn't write to a file (i removed the --file option from $tarOptions), so it writes to stdout. now, split takes input from stdin because of the "-" after the $split_options. this results in chunks the way you set in $split_options.

[EDIT] fixed language [/EDIT]

----------

## Pseud

Heh,

Rather dumb of me, not thinking about piping the commands!  :Embarassed:  (and not looking carefully at the script ... I got misled by the commented split statements you've still left at the end, thinking that one would have to uncomment those as earlier.)

This just set me wondering though, could there be a memory choke when (RAM + swap) < archive size ?

----------

## profy

Helllo,

with version=v3.4

```

PROXYGENTOO ~ # ./mkstage4.sh 

: No such file or directory

PROXYGENTOO ~ # sh mkstage4.sh 

: command not found: 

: command not found0: 

: command not found3: 

: command not found6: 

'kstage4.sh: line 18: syntax error near unexpected token `do

'kstage4.sh: line 18: `for command in ${command_list[@]}; do

PROXYGENTOO ~ # find /bin/tar 

/bin/tar

```

Any suggestion to help me ?

Sorry if it's an easy error i'm a beginner.

----------

## NightTwix

 *profy wrote:*   

> Helllo,
> 
> with version=v3.4
> 
> ```
> ...

 

looks like your script is broken.

did you copy&paste it oder did u download it?

try to download it directly here: ftp://blinkeye.ch/gentoo/mkstage4.sh

----------

## profy

Yes it works  :Smile: 

----------

## torchZ06

got a little problem.  after using the script to create the tarball and following the instructions to extract it to a new partition, everything worked fine, except on my new partition i can only login as root!  when i try another user, it says

```

No directory /home/username!

```

but the directory DOES exist and has the correct permissions, so where did i go wrong?

i even tried adding another user to the new partitions copy using the useradd method as described in the handbook, but that produced the same results as above.

----------

## BlinkEye

does your user exist in

```
cat /etc/passwd
```

and does the group "users" exist in

```
cat /etc/group
```

are you able to login (just can't change to your home) or does the login itself fail?

----------

## Jedidiah

So its been several days, but i finally got it to work on a Mac - almost. i can now sucessfully back up and untar the stage4 without getting any errors (this is on a running computer, where i just untar the stage4 into a new directory, with no intention of booting to it). however, when i try to copy it to a clean hard drive, with the intentions of booting from it, i get an error. here is what i am doing.

1 - i run the stage4 script on a mac that is up and running, have it save the file to an external USB harddisk

2 - i put the MacOSX cd into the computer that i want to copy to, boot to it, erase the harddisk, and then open up the command prompt - all of this is supposedly runing from the mac install cd

3 - when i try to untar the file, it gives me the following error 

```
tar (child): gzip: Cannot exec: No Such file or directory

tar (child): Error is not recoverable: exiting now

tar: Child returned status 2

tar: Error exit delayed from previous errors
```

this error is returned to me wether i cd into the external hardrive and tell it to untar and copy it to the local hd, copy the file to the local hd via cp command. i have tried a variety of different methods for untarring this archive onto the local hd and they all return some varriation on this message. i have succesfully untarred the archive to a directory on the external harddrive (and a local harddisk) while i was booted into macosx.

does anyone have any ideas? 

the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s [url]http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa /73002/wo/M36xDLuJ1Pif2YYZuP32rh4J0ii/1.0.11.1.0.6.9.1.2.21.3.1.1.0?44,20[/url], and becuase this is for the college i am working for, installing gentoo is not an option. so unless someone has a better idea, i really just need to know which live cd (that has tar on it) supports the imac G5s that we have

thanks a ton

jedidiah

----------

## dundas

[Solved] & [Repeated Question, sorry, feel free to delete this post]

 *Quote:*   

> Hi
> 
>     Impressive script, however when I tried to execute
> 
> ```
> ...

 

[solution] just remove /var/log/portage from the $default_include_files

thx man.Last edited by dundas on Tue Jun 28, 2005 4:15 am; edited 2 times in total

----------

## BlinkEye

 *Jedidiah wrote:*   

> 
> 
> does anyone have any ideas? 
> 
> the only other idea i had was to boot from a gentoo live cd for the sake of untarring the archive, but i wasn't sure if there was a livecd that supported writing to a macosx formated partition that is journaled. any suggestions for which live cd i should use? i am on the new imac G5s [url]http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa
> ...

 

it's obvious that the problem is the tar version or the tar capabilities of the mac boot disk. either you gunzip/bunzip your stage4 (if you can handle the size of hte uncompressed tar file) or you try out a ppc livecd (i assume a 64bit version): ftp://mirror.switch.ch/mirror/gentoo/releases/ppc64/2005.0/installcd. i don't own a mac.

what fs are you using? HFS+ or UFS? do a google on the writing capabilities or check the forums. this is what i've found: http://johnleach.co.uk/documents/powerpc/

----------

## alsuren

Okay, I've already used this script to do a clone of my AMD box onto the HDD of an old Pentium3 that I decapitated. I did it by plugging the old HDD into the new comp, and it got my old box bootable again, and after some tapping away, I managed to test SSH on it. This was quite a while back and I got an error when I fired up KDE about inter-process communication, which I've just established is because of my n00bishness in forgetting to create my ~ directory before testing. By the time I re-tested it, I had both upgraded to the latest KDEbase and re-created ~ so I foolishly assumed that it was the update that fixed it. OOOOOOOPS

Just a hint that I picked up elsewhere: The rsync servers are quite slow if you're trying to grab your whole portage tree from them (which is why the Gentoo universal install CD contains a portage snapshot to get you started). It might be nice to get the script to create a snapshot of portage as well (not including distfiles) that you can put on the DVD/multiple CDs when you're burning it/them so it's there if you want it. My backup is just over 1GB so I would need to put it on 2 CDs, with room to spare. A portage snapshot would be quite nice for when I go around being evangelical.

I saw a couple of little links on google and stuff that *might* be useful for people who want to make a bit more of an incremental backup every few days than a stage4 tarball (I know it's not strictly on topic, but they might still be useful)

http://apfelboymchen.homeunix.net/gnu/scripte/backup/ (written by Irom, script using "tar" for incremental backups. I think someone suggested incremental tarballs on this thread, so this could be good for you)

http://dar.linux.free.fr/tutorial.php3 (found on google, tutorial on using "dar". Seems quite old because it refers to zip drives, but might also be useful)

What would be nice is if the script automatically makes a script that you can use for restoring the backup (I'm going to have some time over the summer, so I've posted those links as much for myself as anything else. I'll try hacking together some scripts over the summer and if anything interesting becomes of them, I'll re-post)

----------

## alsuren

While I'm waiting for the tarball to be made, I just had an idea. Is there any way to create a pipe (or entire plumbing system, in this case) so that the data is passed through md5sum at the same time as it's being written to disk? That way, you don't have to read the whole thing off the disk again to md5sum it if you want before you put it onto a CD

EDIT:

oh..... ignore me! it says it's "checking integrity" which suggests some kind of checksumming. It's just that I did a quick find in Kdevelop for "sum" and it came back with nothing...... So if my tarball is corrupt after however long sitting on a CD, will tar simply fail? :O I never knew that..... or does it just continue and tell me which file is corrupt? *hopes it's the second one*

----------

## Pseud

BlinkEye,

Just two more (non-urgent) things which you might want to fix up. I did a backup yesterday using the split option and things went fine, except for these:

1. mkstage4 -v -s  does't work (can't both split and be verbose ... the script only takes the first option and ignores the next) ... not a big deal I guess, but would be nice (since -v and -s are unrelated options, they shouldn't be mutually exclusive).

2. the integrity check "fails" when the script is called with the split option ... so the script should either skip the integrity check when called with the split option, or should do something different to check integrity.

----------

## BlinkEye

thanks for pointing that out.

new version out: v3.5

* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order.

* Fixed integrity check for splitted files - it's now possible.

* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up.

* Removed /var/log/portage per a default as new users get confused about adjusting the script.

* Added used commands to be checked at the beginning.

----------

## ktech

BlinkEye, thanks a lot for your efforts. This script is highly appreciated here   :Smile: 

One suggestion:  Perhaps you can do an option for the script to print on the screen the size of the files being backed-up. Something like  du {list of all directories to backup}. That way, you can guess if you're backing up all the directories you want... Not too difficult to implement and very useful to me  :Smile: 

----------

## JeffBlair

slydini I am running into the same problem. Could you post what you typed to get past that? I thought it had worked as well, but it looks like some of the /var dir did not get extracted out. I couldn't start webmin/mysql/etc...  Otherwise nice little script.

This is about it stoping when it hits .reiserfs_priv directory.

would I type in something like this?

```

tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/  --exclude=.reiserfs_priv

```

Thanks again.

----------

## JeffBlair

Ok, I answered my own question. That command will do it, it got all the way thru without any errors. Now to reboot.    :Smile: 

----------

## ktech

I have a problem:

```
Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)
```

Then I look at the generated file:

```

evanescence stage4 # ls -ll -h

total 2,0G

-rw-r--r--  1 root root 2,0G jul  2 01:38 evanescence-stage4-2005.07.02-custom.tar.gz

```

so, what's the deal? The filesystem where the backup is, is reiser3.

----------

## DrWoland

 *ktech wrote:*   

> I have a problem:
> 
> ```
> Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)
> ```
> ...

 

Maybe that's how much was backed up?

----------

## Pseud

 *ktech wrote:*   

> I have a problem:
> 
> ```
> Total bytes written: 4423526400 (4,2GiB, 2,4MiB/s)
> ```
> ...

 

4.2G raw data; 2.0 G after compression

----------

## ktech

So the words "Total bytes written:" should be "Total bytes readed"?

How strange...

----------

## BlinkEye

rather do a 

```
du -sch filename
```

than

```
ls -l filename
```

du means diskusage and shows the exact space occupied by the filename on your filesystem.

----------

## ktech

it's still 2.0 GB.

BlinkEye, there is any logic in the script telling me that it has writen 4 MB and I having only a 2 MB file?

----------

## BlinkEye

pseud answered the logic. and it's not the logic of the script but the one of tar:

 *man tar wrote:*   

> --totals
> 
> print total bytes written with --create

 

if you don't like it remove that parameter from the $tarOptions and add

```
du -sch $stage4Name.$stage4postfix
```

at the end of the script.

----------

## dundas

[question]

Dear Blinkeye:

        I just ran the script version=v3.5 as root on my gentoo, 

```
./mkstage4.sh -v -s
```

I chosed:

```
Best (tar.bz2):

 (4) Interactive backup (added some small folders in the /home seperate partition, including /smb for samba)
```

and got the following ending errors:

 *Quote:*   

> 3840215040 (3.6GiBï¼1.1MiB/s)
> 
> * stage4 is done
> 
> * umounting boot
> ...

 

        And I supposed that the stage4 creation is successfull, my root partition got 300M left, but I suppose it is still not quite running out of space. 

        So is there any way for me to manually check the integrity again?

                --just add ? 

```
echo "* md5sum archive:  `md5sum $stage4Name.$stage4postfix`"
```

 and then use md5sum to check it?

Or, if u happen to know what might be done wrong to cause the integrity failure??

[solution] 

 *Quote:*   

> hello dundas. 
> 
> echo "* md5sum archive:  `md5sum $stage4Name.$stage4postfix`" 
> 
> this is just meant to be added to the script if you want to do a md5sum each time you run the script. an md5sum doesn't tell you if your stage4 has been built correctly - it just helps to verify upon burning and copying the stage4 if it is still the same as the original. 
> ...

 

I tested using sheepdogj15 and blinkeye's suggestion as above, it goes ok, but returned this warning:  *Quote:*   

> bzip2: (stdin): trailing garbage after EOF ignored

 

after discussing wiz them, and googling, we believe this is just a trivial warning, and my stage4 was made successfully, thx to you all!

----------

## ktech

A little bit of off-topic:

do you guys know if I can restore a backup made in an Athlon XP 32 Bits into an Athlon 64 computer? I think I'm using the most restrictive of -marc/-mcpu (don't remember why as I'm not in that computer). Are they "binary compatible"?

Once I restore my backup I'll be compiling all of my apps into 64 bits, but I don't know if the backup will be running after restored...

Thanks!

----------

## Sheepdogj15

 *dundas wrote:*   

> and got the following ending errors:
> 
>  *Quote:*   3840215040 (3.6GiBï¼1.1MiB/s)
> 
> * stage4 is done
> ...

 

no, that wouldn't do you any good. md5sum doesn't actually error check the file itself: it just gives you a value based on some sort of algorithm ran against the file's binary value. this is good for ensuring there was no corruption from a file transfer (as the file you received should be exactly the same as the original, and thus will have the same md5sum value). 

if you want to manually check for integrity, i suggest this command: 

```
# tar -tvf filename.tar.bz2
```

this will act like it is extracting the tarball, but it will actually only output names of files, and if you run into a corruption, it will die with an error. 

my suspicion, though, is that the spliting process interfered with the error checking. i dunno though, i'll leave it to Blinkeye to figure out. what you could do is delete the resulting tarball(s), and run the script again.

----------

## Sheepdogj15

 *ktech wrote:*   

> A little bit of off-topic:
> 
> do you guys know if I can restore a backup made in an Athlon XP 32 Bits into an Athlon 64 computer? I think I'm using the most restrictive of -marc/-mcpu (don't remember why as I'm not in that computer). Are they "binary compatible"?
> 
> Once I restore my backup I'll be compiling all of my apps into 64 bits, but I don't know if the backup will be running after restored...

 

yes. amd64 is compatible with x86 compiled binaries. in fact, if one wanted to use a pure 32bit environment on an Athlon 64, the recommendation would be to use -march=athlon-xp (there was a thread on this in the amd64 forum all about what CFLAGS and such to use). 

the thing is, if you use the whole backup, you will have a 32bit Gentoo. if you want to compile in 64bit, you need to install from an amd64 stageX tarball. and it would be messy to just dump 32bit stuff on a 64bit system. you CAN run 32bit apps on 64bit Gentoo, it's just that you don't want to get things too mingled together (maybe what you could do is extact the backup to a separate partition, and then chroot in. the amd64 hardware docs explain this process pretty good).

----------

## Pseud

BlinkEye,

I faced one small issue when I used the backup script to move an install from my desktop to my laptop: my hardware clock on the laptop was drifting, gaining something like 10 minutes every hour! (everything else went perfectly; thanks once again for the script!)

I searched the forums and found that the thing causing problems was the /etc/adjtime file. When I deleted this file and rebooted, the issue resolved itself (if the file is not found, it is regenerated upon reboot, and that fixes everything)

So, this is my suggestion:

Add /etc/adjtime to the default exclude list.

Reason: this is necessary if you plan on using your backup to copy your install to another machine; its not necessary to back this up because a reboot regenerates it anyway.

Thanks!

----------

## smurfd

Wow, this script is awsome!! Thanks for sharing  :Smile: 

----------

## funkoolow

ehm, can't understand why, but actually the script won't let me pass the integrity check. that's the end of the output, mybe it's useful:

```
/var/www/localhost/htdocs/the54.com-OLDSITE/modules/gallery/netpbm/ppmtojpeg

sh: line 1:  8883 Broken pipe             /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /mnt/p4-camera -prune -o -path /home/ftp -prune -o -path /home/funkoolow -prune -o -path /home/funkweb -prune -o -path /home/t0mcat -prune -o -path /usr/src/linux-2.6.11-gentoo-r9 -prune -o -print

sh: line 1:  8843 Done(141)               ( /usr/bin/find /var/db; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /usr/src/linux-2.6.11-gentoo-r9 -print; /usr/bin/find /home/t0mcat -print; /usr/bin/find /home/funkweb -print; /usr/bin/find /home/funkoolow -print; /usr/bin/find /home/ftp -print; /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /mnt/p4-camera -prune -o -path /home/ftp -prune -o -path /home/funkoolow -prune -o -path /home/funkweb -prune -o -path /home/t0mcat -prune -o -path /usr/src/linux-2.6.11-gentoo-r9 -prune -o -print )

      8845 Broken pipe             | /bin/tar --bzip2 --preserve-permissions --create --absolute-names --totals --ignore-failed-read -v --file /mnt/p4-camera/funkserver-stage4-2005.07.17-custom.tar.bz2 --no-recursion -T -

* stage4 is done

* umounting boot

* Checking integrity

* Integrity check failed. Re-run the script and check your hardware.
```

oh, i'm using the v3.5 in the best and custom modality (option 4)

----------

## smurfd

hmm, where would i change for it to Not add my /home to the .tar ?

----------

## scoobydu

Great script! Thanks  :Very Happy: 

Just backed up my new powerbook install and restored it onto my mac mini!

Works a treat!

----------

## carpman

Love the script thanks  :Smile: 

Just one small thing that could be added to recovery section, that is the use a portage snapshot before emerge sync. This would save some time plus reduce bandwidth usage on rsync servers.

----------

## dundas

last updated Aug 4, 2005 wiz "modify /etc/resolv.conf"

Hi, I just thought maybe this is helpful to someone testing stage4 under vmware, I got it up wiz the following notes.

followed some of http://gentoo-wiki.com/HOWTO_Install_Gentoo_on_VMware_in_Windows_NT/2K/XP

In my case I set up the vmware wiz similar partition like my stage4 backup.

 *Quote:*   

> /dev/sda1   /boot
> 
> /dev/sda2   swap
> 
> /dev/sda3   /
> ...

 

 1. boot off a live-cd and repartition and create filesystems as necessary 

(remember to set active boot flag to /boot as well)

 2. eventually reboot, using option: gentoo docache 

 3. umount /mnt/cdrom 

 4. remove the live-cd and insert the cd with the stage4 

 5. mount /dev/cdrom1 /mnt/cdrom 

 6. mount /dev/sdaX /mnt/gentoo    #X stands for the number of primary partition u set for your root.

 7. mkdir /mnt/gentoo/boot 

 8. mount /dev/sda1 /mnt/gentoo/boot 

 9. tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/ 

 or 

 9. tar xjvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.bz2 -C /mnt/gentoo/ 

10. mkdir /mnt/gentoo/proc 

11. mount -t proc none /mnt/gentoo/proc 

12. mkdir /mnt/gentoo/dev 

13. mount -o bind /dev /mnt/gentoo/dev 

14. chroot /mnt/gentoo /bin/bash 

15. env-update 

16. source /etc/profile 

if in need adjust necessary files (/etc/fstab, /boot/grub/grub.conf) and/or 

install grub)

[EDIT Aug 4, 2005]modify /etc/resolv.conf and change the nameserver to the ones of this new network, if u dont do this, emerge --sync will report errors that:

 (-2, 'Name or service not known')

(activate swap, not sure if this is necessary)

# mkswap /dev/hda2

# swapon /dev/hda2

17. emerge --sync    (rebuild portage tree) 

( may need to emerge grub)

# emerge grub

# cp /proc/mounts /etc/mtab

Now we can install GRUB using grub-install: I was stuck here while booting up wiz black screen, that's because I didn't install grub into mbr yet

# grub-install /dev/sda

then if grub-install /dev/sda doesn't work try

```
# grub

grub> root (hd0,0) (since fedora /boot is on /sda1)

grub> setup (hd0) (still the mbr of the first hd)

grub> quit
```

maybe at this point we need to configure the kernel a bit to suit the new 

system. u may refer to kernel config how-to.

18. exit 

19. cd / 

20. umount /mnt/cdrom 

21. remove backup cd 

22. umount /mnt/gentoo/boot 

23. umount /mnt/gentoo/dev 

[not sure why, but sometimes I just can't umount /mnt/gentoo/dev, 

but...doesn't hurt much]

24. umount /mnt/gentoo/proc 

25  umount /mnt/gentoo 

26. Reboot

Other things are fine, need a bit configure to startx

[EDIT Jul 26, 2005][/url]

u may refer to mastabog's nice workthrough to do the rest.

https://forums.gentoo.org/viewtopic-t-246788-postdays-0-postorder-asc-start-0.html

p.s. if u had log for portage, u need to mkdir /var/log/portage again. And u should be careful that some file permission for your home directory (if u did backup them), and chown chmod accordingly.

so again, thx BlinkEye

----------

## Xerxes83

Very nice script indeed! For my own backups I am using a modified version of a really old version of this script which uses DAR and does some other stuff as well. I was wondering which directories are absolutely necessary to get a system to work again after the bad stuff has happened. I hope it hasn't been asked already... but 10 pages of posts is just too much to read.

Are the following files really needed? And why?

/dev/null

/dev/console

/proc

/sys

/usr/portage

All the files in /dev and /proc can be re-created using a Gentoo live CD? Or can something go wrong in that case? And /usr/portage should restore itself with 'emerge sync' and /sys is empty on my system. I understand that it is more convenient to have them included in a backup, but since I am storing my backups on a remote server I prefer to have them as small as possible.

----------

## Stephonovich

Worked great, with only a few problems.

First, portage complained that /var/log/portage doesn't exist.  I did a quick mkdir and it fixed it.

Second, on boot, my gensplash image refuses to display, claiming that the filesystem / is mounted on doesn't contain a /dev/tty1.  I checked, and it's there.  echo "blah" > /dev/tty1 works as well.  That probably isn't relevant to this support thread, but I thought I'd mention it anyway.

----------

## Matteo Azzali

Some strange socket ignored message, I never saw these:

```

* mounting boot

Do you want to continue? (y/n) y

* Please wait while the stage4 is being created.

/bin/tar: /var/lib/dbus/system_bus_socket: socket ignorato

/bin/tar: /var/run/dbus/system_bus_socket: socket ignorato

/bin/tar: /var/run/acpid.socket: socket ignorato

/bin/tar: /var/run/xdmctl/dmctl/socket: socket ignorato

/bin/tar: /var/run/xdmctl/dmctl-\:0/socket: socket ignorato

Total bytes written: 2529454080 (2,4GiB, 2,9MiB/s)

* stage4 is done

* umounting boot

* Checking integrity

* Everything went smoothly. You successfully created a stage4.

```

Maybe it's cause my version (previous of this 3.5) wasn't goin in /var/ directories.

Is this ok???

----------

## BlinkEye

yes, this is ok. but you've pointed me to a good idea - one could exclude the /var/run folder ...

----------

## Ma3oxuct

I am not sure if you intended this or not, but when to the script are added excluded folders which riside in /home/{user} and the option "y" is chosen to back up /home/{user}, the folders previously excluded are not.

Here is what I am talking about:

```
default_exclude_list="

...

/home/biter/.ut2004

/home/biter/.etwolf

...
```

It includes the excluded folders any way.

```
/home/biter/.ut2004/

/home/biter/.ut2004/Cache/

/home/biter/.ut2004/Cache/0F0802C011D8FD9430000C8CB7637684-1.uxx

/home/biter/.ut2004/Cache/3ADC18814EB51B81C1B77E96E6F7CE63-1.uxx

/home/biter/.ut2004/Cache/C68430BC4794813072F223813183E4CA-1.uxx

/home/biter/.ut2004/Cache/8ECBE65540CF6B3C05930F841C17CC11-1.uxx

/home/biter/.ut2004/Cache/cache.ini

/home/biter/.ut2004/Cache/7492CDAD47C002364C6F408C429537FE-1.uxx
```

Here is precisly what I have been choosing:

```
Cago-Ma3oxuct / # sh stage4-3.5.sh -v

Backup script v3.5

==================

What do you want to do? (Use CONTROL-C to abort)

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup

Please enter your option: 2

Do you want to backup /home/biter? (y/n) y

Do you want to backup /home/common? (y/n) n

Do you want to backup /usr/src/linux-2.6.13-rc6? (y/n) y

* mounting boot

* creating the stage4 in /work/stage4-backups with the following command:

(/usr/bin/find /var/db; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /usr/src/linux-2.6.13-rc6 -print; /usr/bin/find /home/biter -print; /usr/bin/find /bin /boot /dev /etc /home /in /lib /mnt /opt /proc /root /sbin /stage4-3.5.sh /stage4.sh /sys /tmp /usr /var /work -path /home/common -prune -o -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /opt/ut2004 -prune -o -path /opt/enemy-territory -prune -o -path /home/biter/.ut2004 -prune -o -path /home/biter/.etwolf -prune -o -path /in -prune -o -path /work -prune -o -path /work/stage4-backups -prune -o -path /home/biter -prune -o -path /home/common -prune -o -path /usr/src/linux-2.6.13-rc6 -prune -o -print) | /bin/tar --gzip --preserve-permissions --create --absolute-names --totals --ignore-failed-read -v --file /work/stage4-backups/Cago-Ma3oxuct-stage4-2005.08.17-custom.tar.gz --no-recursion -T -

Do you want to continue? (y/n) y
```

----------

## abracadabra

I get this when i try and dl the script:

~ $ wget ftp://blinkeye.ch/gentoo/mkstage4.sh

--17:19:46--  ftp://blinkeye.ch/gentoo/mkstage4.sh

           => `mkstage4.sh'

Resolving blinkeye.ch... 213.202.33.122

Connecting to blinkeye.ch[213.202.33.122]:21... connected.

Logging in as anonymous ...

Login incorrect.

Could somebody please post a copy of an up-to-date script? Thanks

----------

## funkoolow

http://temp.the54.com

the latest v3.5 is the mkstage4.sh, i've just changed the parameter stage4Location=/mnt/p4-camera to fit my needs, the rest is absolutely intact. note that these will not be available during night-times at CET, since i turn off my machine. Just in case, pm me an email and i'll send them you.

----------

## NightTwix

abracadabra, you could also copy&paste the script from the inital posting.

But beware of wrapping and linefeeds

----------

## BlinkEye

ftp is working again. sorry for the inconvenience - i assembled a new server and had to switch some harddisk but forgot to adjust the ftp paths (partition changes).

----------

## slydini

Anyone know how to use this script on say RED Hat 9? What would the exclude list look like?

Also I am trying to do a full disk to disk backup can this script do something like that?

----------

## BlinkEye

well, the script was made for that purpose. i haven't used red hat for a long time, but you probably know where cached/temporary/downloaded files are stored and exlude those instead of the portage specific ones.

----------

## riczan

Great script! Thanks. I just moved my gentoo to my new 160GB hard drive and it worked flawlessly EXCEPT for:

this might be a bug or my mistake. Please clarify

- I consider my kernel source to be part of minimal so:

- I erased /usr/src from the    default_exclude_list

and the result was that everything on /usr/src was backed up except for the source kernel I care:

```
original

-rw-r--r--   1 root root    0 Oct 27  2004 .keep

lrwxr-xr-x   1 root root   31 Mar 21 13:24 linux -> /usr/src/linux-2.6.10-gentoo-r6

drwxr-xr-x  19 root root 4096 Mar 26 12:46 linux-2.6.10-gentoo-r6

drwxr-xr-x  18 root root 4096 Mar 26 22:15 linux-2.6.11-gentoo-r4

drwxr-xr-x  19 root root 4096 Mar 21 13:27 linux-2.6.9-gentoo-r1

```

and the backed up result

```

-rw-r--r--   1 root root    0 Oct 27  2004 .keep

lrwxrwxrwx   1 root root   31 Aug 26 08:48 linux -> /usr/src/linux-2.6.10-gentoo-r6

drwxr-xr-x  18 root root 4096 Mar 26 22:15 linux-2.6.11-gentoo-r4

drwxr-xr-x  19 root root 4096 Mar 21 13:27 linux-2.6.9-gentoo-r1

```

Please let me know what I did wrong or if this is a bug

Thanks

----------

## dinomyte66

Nice program, works well, but---since running it I have lost my /boot/grub/grub.conf file.  Has anything like this happened to anyone before?

I just happened to be searching for the new backup tar and when I went into my boot directory I noticed my grub config was gone!  I know it was there earlier because I had modified it earlier in the day with a reboot parameter.  I rebooted my machine several times today to get xorg working so it had to be there all along---is it possible that more than just this file is gone?

Please help---It's probably something I did but it seemed like everything went as it was suppose to.

Should I just recreate my grub.conf in the meantime?

Thanks in advance for all of your help,

Dino

----------

## dinomyte66

I was able to get some help on one of the support forums [Kernel & Hardware] sorry, I sould have read the guidelines for this forum prior to posting  :Embarassed:  , and found that the script does umount the /boot file. So all I had to do was mount my /boot back to /dev/hda1.

Thanks again for your help,

Dino

----------

## Matteo Azzali

Just used this way after a resize of my / partition (the last on the disk and reiserfs, needs to backup and restore...)

After restore process:

/mnt directory was empty, I had to recreate manually some dirs.

(i.e.: #mkdir /mnt/cdrom )

needed to reemerge splashutils (#emerge --oneshot splashutils)

and then to follow the emerge info to recreate the fake /dev/tty1 dev in

the boot partition

```

 mount --bind / /lib/splash/tmp

mknod /lib/splash/tmp/dev/tty1 c 4 1

umount /lib/splash/tmp

```

Now it seems to run properly 

(the first two emerge sync were kinda strange, first one "from chroot" interrupted on metacache computing,

second one from my system updated all the profiles from 2003 to 2005. now even emerge sync seems to

work in the right way)

----------

## nokilli

Here's a post I wrote about how maybe this script could be used to do more than backup and restore.

Maybe I should have posted it here?

----------

## nunogt

Hi, thanks for the great script. One question, though:

I'm considering removing Windows completely from this machine, which is currently installed in hda1 and takes approx 5 GB. hda2 is /boot, hda3 is swap, hda4 is / . Will it be necessary to make adjustments to the backup images once the partition table changed and I want to restore them? All the partititons will decrease 1 unit (hda2 -> hda1, etc) and hda4 will be /var/cache/squid/

----------

## BlinkEye

first: good idea   :Wink: 

second: partition your harddisk in any way you want (resizing, removing, creating) - just remember to adjust your (extracted old) fstab from the stage4 - maybe even grub/lilo's config.

----------

## gnychis

hey all, i'd love to use this script, I am just a little confused about something...

i don't see /etc as being backed up, why is /etc not backed up since /etc/conf.d and /etc/init.d contain all of our startup scripts and configurations??

does this mean that everything in my home directory will NOT be saved during a minimal backup? What if I want my home directories to be included during minimal backup? Can I take "/home/*" out of here?

```

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will

# include/exclude these files/folders depening on your answer.

custom_include_list="

/home/*

/usr/src/linux-`uname -r`"

```

after i create the backup file, how can i "cat" or output all the files contents so i can check over what is in it to make sure all my important files were backed up? 

Thanks!

George

----------

## nunogt

 *BlinkEye wrote:*   

> first: good idea  
> 
> second: partition your harddisk in any way you want (resizing, removing, creating) - just remember to adjust your (extracted old) fstab from the stage4 - maybe even grub/lilo's config.

 

Sounds easy. Thanks for clarifying this.

Cheers.

----------

## gnychis

another concern of mine, i ran it as a minimal backup, 4 hours elapsed, and it was still running, and i ctrl+c'ed and i found this file:

```

monster stage4 # ls -l

total 33217944

-rw-r--r--  1 root root 33981988864 Sep  4 01:12 monster-stage4-2005.09.03-minimal.tar.bz2

```

massive

what the heck did i do wrong?

```

default_exclude_pattern="*.mp3 *.iso *.gcm"

default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

/var/db

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

default_include_files="

/dev/null

/dev/console

/home

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log"

default_include_folders="

/var/db

/home/*"

custom_include_list="

/usr/src/linux-`uname -r`"

custom_exclude_list="

/home/hedpe/server"

custom_exclude_pattern="*.mp3 *.iso *.gcm"

```

----------

## BlinkEye

about your question of including home in a minimal backup - yes, this is the way to go (remove it from custom_include_list and it will be backed up anytime). to see what is actually being stored run the script in verbose mode (--verbose/-v). /etc is backed-up of course. the script uses a find command (as this allows a lot more finetuning as just listing folders). 

to see if your backup worked read "Re-assemble split-ed stage4" in the first post. 

about your enormous stage4 - it seems like you didn't exclude a big movie/mp3/download folder. it could be that the find command does not work correctly though. if you could have a further look into this you could help us all.

----------

## gnychis

okay maybe you can help me out a little with this, so let me tell you what I want:

I want all home directories backed up recursively, BUT I want this folder excluded:

/home/hedpe/server

OR

I want all home directores backed up recursively, BUT I want .mp3, .gcm, and .iso excluded

I thought I had done the second of the two with the variables I gave above, but I think it was including all the .mp3, .gcm, and .iso files.

What did I do wrong?

They are all in /home/hedpe/server

----------

## gnychis

no matter what i do, it keeps backing up my mp3s, I did a verbose mode on it, and i saw it backing up my .mp3 files under the directory /home/hedpe/server/mp3s/an-album/here.mp3, even with these settings:

```

default_exclude_pattern="*.mp3 *.iso *.gcm"

default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

/var/db

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

default_include_files="

/dev/null

/dev/console

/home

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log"

default_include_folders="

/var/db"

custom_include_list="

/usr/src/linux-`uname -r`"

custom_exclude_list=""

custom_exclude_pattern=""

```

As you can see, the only things I changed from the original script was default exclude pattern and i removed /home/* from custom_include_list

so... therefore it started to back up /home/hedpe ... but it was NOT excluding the *.mp3 like it should have since i put it in default_exclude_pattern

I used menu option 3

thanks!

George

----------

## BlinkEye

please use the script in its original state once again. then, do a custom backup and chose your home to back up. add your desired exclude extensions to custom_exclude_pattern (not default_exclude_pattern):

```
custom_exclude_pattern="*.mp3 * *.iso *.gcm"
```

and try again.

----------

## scoobydu

Whats the best way of including only the configuration files in my /home/local/ directory? ie the dot config files ...

If I add '.*' to my include files will I need to select backup home?

tia

ps great script!

----------

## -=GGW=- $ol!d $n4>|e

hmm, I made a stage 4 when i was about 1/2 through custumizing, then i got done messing with config files and went straight to installing extras. Now i'd like to make another stage4 but this time, instead of bieng a little under a gig, it ran for abou 2 hours and when i checked to see what was goin on, it had 11 cds worth of zips already... did it screw up, or do I have to manually tell it to leave some directory out that it doesnt by default and for some reason, that dir is huge on my system? I'm using --split and then 3 any help on decreasing the size of the zips would be greatly appreciated, although I wouldn't mind a full backup.. i have a 4x cdr   :Embarassed: 

----------

## gnychis

 *BlinkEye wrote:*   

> please use the script in its original state once again. then, do a custom backup and chose your home to back up. add your desired exclude extensions to custom_exclude_pattern (not default_exclude_pattern):
> 
> ```
> custom_exclude_pattern="*.mp3 * *.iso *.gcm"
> ```
> ...

 

worked perfectly, thanks blinkeye!

----------

## BlinkEye

 *-=GGW=- $ol!d $n4>|e wrote:*   

> hmm, I made a stage 4 when i was about 1/2 through custumizing, then i got done messing with config files and went straight to installing extras. Now i'd like to make another stage4 but this time, instead of bieng a little under a gig, it ran for abou 2 hours and when i checked to see what was goin on, it had 11 cds worth of zips already... did it screw up, or do I have to manually tell it to leave some directory out that it doesnt by default and for some reason, that dir is huge on my system? I'm using --split and then 3 any help on decreasing the size of the zips would be greatly appreciated, although I wouldn't mind a full backup.. i have a 4x cdr  

 

the script does backup everything on your computer - except those folder you excplicitly exclude or which are excluded by default. so, if you have created a folder like /mp3 or /backup it will backup this folder if you don't add it to the exclude list (now it depends on your choice where to add that folder).

----------

## -=GGW=- $ol!d $n4>|e

but i cant think of anywhere I've aded stuff besides in /home/fuag :\ I let portage handle pretty much everything and if it doesnt handle it, im sure to keep the program in a place I can see in case I want to delete it, I have quite a bit of game files in /opt such as quake3 doom3 ut2k4 and loads of mods for them but doesnt the sqript auto exclude /opt   :Confused: 

EDIT: Lol, finished my work and checked... wow, apperently it doesnt... ima go tell it to leave out the games   :Laughing:  sorry for the posts

----------

## PRC

Well..I tried the script it worked great on the compression part....but a few problems:

Broke ipw2200, my wireless module

Destroys a few vital symlinks ie. usr/src/linux

The symlink isn't a big problem...but has anyone figured out how to fix ipw2200. It seems to be the only thing forcing me to redo everything...

EDIT: It's not your script that breaks my ipw2200, I aplogize, udev-68 breaks ipw2200. You have to use udev-58, solves problem.

ADDON: Feature Request....it would be nice to renumber backups that happen on the same day...

stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d` 

how do you add the time to that?

----------

## PRC

I have been using the backup script quite alot...but every damn one past xorg breaks. I think it has to do with xdm......but it might not the be script...but it works before i backup.....anyone run into this problem, and fix it?

----------

## BlinkEye

 *G4User wrote:*   

> stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d` 
> 
> how do you add the time to that?

 

you could add hours and minutes:

```
stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d_%H:%M` 
```

which would result in a

```
# date +\%Y.\%m.\%d_%H:%M

2005.09.09_05:13
```

appendix.

----------

## RypPn

First of all I'd like to thank you for taking the time to both write and subsequently enhance this script. It has saved me many hours of work trying to reconfigure weird and wonderful hardware (epkowa scanner,gps,wifi etc). I have noticed a few peculiarities on a freshly restored system however.

Apache fails to start due to /var/log/apache2 missing.

Mysql fails to start (still working on the solution to this one) , mysqld (/etc/mysql/my.cnf) fails to start.

Vmware-workstation fails to start, symptoms as descibed in this forum link https://forums.gentoo.org/viewtopic-t-381798-highlight-vmware.html , solution works.

Xfont Server updates FC cache and loads in fonts on every boot. Previously this only happened when adding a new font-set to /etc/X11/fs/config (I run xfs). This one bugs me the most as it takes an age for the Computer to boot.

I'm sure there is probably a simple solution to all these quirks, but as a non-programmer its a bit beyond me.

Any thoughts/suggestions/slaps on the back of the head are most welcome   :Laughing: 

----------

## BlinkEye

i run into very similar problems while backing up my server and restoring the stage4 on new hardware. i simply couldn't get tomcat to start up again. i can't have a look into it until the next 4 weeks have passed - bear with me, i'll have a look into it.

----------

## RypPn

Update...

Mysql needs removing and recompiling as per this forum link https://forums.gentoo.org/viewtopic-t-286297-highlight-mysql+error+2002.html , 3 cheers for kashani!

Luckily I was upgrading my hard drive, so I can plug in the old one to mysqldump the db's and then restore them. Definately one to watch out for if you use mysql   :Wink: 

Only issue left is the font problem, I'll report back if/when I fix it.

----------

## RypPn

Last Update.

Re-emerging xorg has fixed the constantly scanning in fonts on boot issue, all is now well with the world    :Cool: 

If I come across any other gotcha's I will no doubt be back.

----------

## frodoontop

This is just plain perfect, wish I'd new this before. Now I can easily backup my system when everything works. Almost too easy. Just burn it to a dvd and unpack when needed. 

Thank you very much, now I'll feel less shity when I fucked up everything  :Smile: 

----------

## someguy

wouldnt dd if=/dev/whatever of=/home/whatever.iso bs=512 work ?

----------

## iq_zero

Hello Im having av harddrive that is making a horrible sound so im going to replace it. After looking for easy way of doing this I found this post.

I looks very interesting and I will give it a try.

The link to the script on page 1 is dead please update if the adress is changed i think this would make the wonderful howto complete.

----------

## koelle

Hi,

i just used the backup script and everything went fine. Now i decided to reinstall WindowsXP. So I first booted the Gentoo 2004.2 Minimal LiveCD and repartitioned my harddisk:

 *Quote:*   

> /dev/hda1 WindowsXP as NTFS
> 
> /dev/hda2 /boot
> 
> /dev/hda3 swap
> ...

 

The whole backup process went fine until I wanted to reinstall grub into my MBR:

 *Quote:*   

> grub
> 
> root (hd0,1)
> 
> setup (hd0,1)

 

This resulted in the following output

 *Quote:*   

> Filesystem type is ext2fs, partition type 0x83
> 
> Checking if "/boot/grub/stage1 exists ... yes"
> 
> Checking if "/boot/grub/stage2 exists ... yes"
> ...

 

When I try to boot my System grub loads, but only into the grub shell. 

How can I get my bootloader working again? I tried all day several times and am desperate right now!

Hope someone can help me.

greetings koelle

----------

## koelle

Hey,

I finally made it to install grub. I just installed SuSe, then chrooted into my Gentoo Environment and installed grub with:

 *Quote:*   

> grub-install --root-directory=/boot /dev/hda

 

So grub loaded and I could select my Gentoo System. But then suddenly the following message appeared and the booting process stopped:

 *Quote:*   

> UDF-fs: no VRS found
> 
> kernel panic - not syncing VFS: Unable to mount root fs on unknown-block(3,3)

 

What does this mean and how can i fix it?

----------

## koelle

Hi,

i solved my problem. I still had root=/dev/hda3 in my grub.conf. It had changed to /dev/hda5. Now it works.

greetings koelle

----------

## Hagar

To preserve folder structures you can use the .keep files.

```
default_include_files="

/dev/null

/dev/console

/home

`/find /mnt -name .keep`

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log
```

In this example the mountpoints will be stored in the backup.

Gentoo provides .keep files in the mountpoints by default (cdrom, floppy) so why not benefit from it.

----------

## bingobob

How can i make this automatic (ie non interactive). I'd like to create a cron job to run the script every week.

cheers

bb

----------

## DOSBoy

You could prepare the user input to create the archive as it is very simple. Make a file with something like:

1

y

n

or whatever. (I don't know exactly how the options proceed). Save it to somefile. Then

sh mkstage4.sh < somefile

should do the trick.

----------

## Gotterdammerung

Including me on this thread.

----------

## funkoolow

back here for this simple and direct question: since my gentoo works as webserver with php and mysql, is it relevant that during the backup process it's available on web and so it's possible that db and logs and such can be updated and or modified? will be useful and/or safer to stop apache2-mysql-php?

thanks  :Smile: 

----------

## Bionut

just wanted to say thanks for the script... I'd hate to loose all my config etc because I'm too lazy to do it all myself.

Cheers   :Very Happy:   :Very Happy:   :Very Happy: 

----------

## tgh

Noob questions.

1) What is a good directory to keep this script in?  Where do admins like to keep scripts like this?  I'm guessing "/usr/local/sbin" or should it go straight to "/sbin" since it's a critical file needed to restore the system?  (Well, not really, you could restore by hand.)  A lot of the pages that talk about directory structure lean more towards putting something like this in /usr/local/sbin, if I understand them correctly:

RedHat 5.0 - section 11.1.1.3

Linux Directory Structure (doc.vic.computerbank.org.au)

Linux Directory Structure (www.comptechdoc.org)

(answered my 2nd quesetion)

----------

## funkoolow

 *funkoolow wrote:*   

> back here for this simple and direct question: since my gentoo works as webserver with php and mysql, is it relevant that during the backup process it's available on web and so it's possible that db and logs and such can be updated and or modified? will be useful and/or safer to stop apache2-mysql-php?
> 
> thanks 

 

is this question too stupid? cmon, a simple "yes" or "no" would be enough  :Rolling Eyes: 

----------

## bigbob73

 *tgh wrote:*   

> Noob questions.
> 
> 1) What is a good directory to keep this script in?  Where do admins like to keep scripts like this?  I'm guessing "/usr/local/sbin" or should it go straight to "/sbin" since it's a critical file needed to restore the system?  (Well, not really, you could restore by hand.)  A lot of the pages that talk about directory structure lean more towards putting something like this in /usr/local/sbin, if I understand them correctly:
> 
> RedHat 5.0 - section 11.1.1.3
> ...

 

I personally keep it in /root/scripts, but you could keep it wherever you feel most comfortable.

Bigbob

----------

## iq_zero

 *funkoolow wrote:*   

>  *funkoolow wrote:*   back here for this simple and direct question: since my gentoo works as webserver with php and mysql, is it relevant that during the backup process it's available on web and so it's possible that db and logs and such can be updated and or modified? will be useful and/or safer to stop apache2-mysql-php?
> 
> thanks  
> 
> is this question too stupid? cmon, a simple "yes" or "no" would be enough 

 

I have php and mysql running on my machine. I tryed this backupscript and it worked fine. I didnt stop any servers while creating the backup.

After I had unpacked the stage4 to my disk i rebooted and mysql and some other programs were complaining because i didnt have any log files. I created them with nano and rebooted again just to see that everything was ok.

The backup went great but the disk I was planning to run on crashed 12 hours after the machine was up and running again. IBM Deathstar  :Sad: 

I hope something can be useful

----------

## funkoolow

 *iq_zero wrote:*   

> I have php and mysql running on my machine. I tryed this backupscript and it worked fine. I didnt stop any servers while creating the backup.
> 
> After I had unpacked the stage4 to my disk i rebooted and mysql and some other programs were complaining because i didnt have any log files. I created them with nano and rebooted again just to see that everything was ok.
> 
> The backup went great but the disk I was planning to run on crashed 12 hours after the machine was up and running again. IBM Deathstar 
> ...

 

thanks for your reply, man. I asked cause once I had problems too with mysql after restoring a stage4 and I solved them directly remerging mysql. Beyond that, everything gone smooth, so i wondered if this would happen everytime or not (and it seems yes, for now)...

btw, thanks again for your reply (oh, and sorry for your disk, those deskstar are PURE TRASHWARE  :Evil or Very Mad:  )

----------

## Kuhrscher

Great script! Thank you!

----------

## Lonnie

This may have been answered, but my eyes are starting to cross from reading all the posts, so here goes.

I have a relatively new Gentoo install, only about a couple of weeks old.  However, I didn't RTFM nearly as well as I should have.  Thus, I have a Very nice partitioning scheme, for a Debian install.  Not really very good for a Gentoo install.

What I would like to do is make a complete backup of what I have on here so far.

Totally re-format the hard drive and then restore the system.

I would keep the same partition numbering scheme, but there would be some serious resizing done, and likely make one partition and xfs or reiserfs for use with larger files.

I can burn to DVDs or CDs, and have both available.

So, could I accomplish this by simply commenting out all the "default_exclude_list=" options and changing the destination for the backup?

Here is my df -h output, so a few pointers would also be appreciated, except I will most likely keep all of them.    :Very Happy: 

Filesystem             Size  Used Avail Use% Mounted on

/dev/hda5             2.8G  178M  2.5G   7% /

udev                    501M  212K  501M   1% /dev

/dev/hda6              9.2G   81M  8.7G   1% /home

/dev/hda7              4.6G  3.6G  849M  81% /usr

/dev/hda8              7.4G  228M  6.8G   4% /var

/dev/hda9               4.6G  140M  4.3G   4% /opt

/dev/hda10             8.4G  138M  7.8G   2% /storage

shm                      501M     0     501M   0% /dev/shm

Was thinking I could shrink /home down to around 5gig, increase /usr to maybe 8gig, /var and /opt at 1gig each and put all the rest in /storage and make it the xfs or reiserfs to handle any really large files I might run into.

Other than hosing up the initial partitioning scheme, and still having a bit of trouble with playing audio CDs, I am very happy with Gentoo.

Thanks for any and all help.

----------

## BlinkEye

quote from the script itself about $default_exclude_list:

 *mkstage4.sh wrote:*   

> # don't touch it unless you know what you are doing!

 

you do not know, so leave it. this script works right out of the box for you - it has been made for exactly that purpose. as you'll only resize the partitions you won't even have to change /etc/fstab (well, that wouldn't be a problem, right? you may have to adjust it anyway if you change the filesystem).

for other questions: read at least the first post to get along (while backing up and/or restoring). you eyes won't be starting to cross from that   :Wink: 

in case something doesn't work, please remember:

 *Jan L. A. van de Snepscheut wrote:*   

> In theory, there is no difference between theory and practice. But, in practice, there is.

   :Rolling Eyes: 

----------

## Sheepdogj15

unless you have security related reasons for being that granular in your partition scheme, i actually would recommend simplifying it a bit. my gentoo workstation is only split between /home and everything else in /. there might be good reason to make /boot a seperate partition (backwards compatibility and such). and sure, if you want make another partition for large files, go ahead.

blinkeye may get annoyed if we talk about this more here, so you may want to just start a new thread elsewhere on the subject.

----------

## Lonnie

Very true, I do not know what my ideas would be doing with your script.  So, I will just attempt it.  I do have Gentoo installed, so I was able to muddle through setting up fstab at least once.    :Very Happy: 

I did do a trial run with the script and I come up with about a 1gig stage4, and that was with everything that the script would normally accept.

Eyes didn't start crossing until I was into page 6 of 12 of the comments, at around midnight.    :Confused: 

For sheepdog, I just like to play with partitions.  Had I only made the three partitions suggested, I wouldn't have found this discussion, as I would not have had a need to re-partition my system.

I thank you for your prompt responses, and I assume that what I can't backup with the script is only stuff that will be re-created once I update the system, or in my case, info that I should make copies of before I make the backup attempt.

I really appreciate the hard work that you people do so that non-programmers like myself are able to use such a high quality operating system.  Will really be nice once a few others get on the x86_64 bandwagon, like flashplayer.

Thanks again,

Lonnie

----------

## BlinkEye

hi again. yes, if you haven't fiddled around with serious settings (like changing portage's path and so on) you will not lose any data. there are a lot of things one does not need to backup - because they'll be recreated.

and i do play with partition schemes around a lot myself, and with filesystems too (as i lost a complete reiser4 gentoo installation due to bugs i thought of this backup script). 

i completely understand if your eyes are crossing if you read through this hole thread. but this should not be necessary. the thread itself is about problems, questions, suggestions. there's no need to read all the posts. if one does run into "personal" problems you might pm me - this has been done a lot.

i don't get annoyed about posts here at all - i just think it's better for everyone if we talk only about the script and the backup itself to keep the thread consise.

----------

## Ejunkie

hey,

i have a problem with a similar (sorry i hope i wrote is right) script

```

EXCLUDE="

--exclude=proc/*

--exclude=sys/*

--exclude=mnt/backup/*

--exclude=mnt/install

--exclude=var/log/*

--exclude=dev/*

--exclude=usr/portage/distfiles/*

"

if [ `whoami` != root ] ; then echo "you must be root, exiting" ; exit ; fi

cd /

tar -cjpvf /mnt/backup/backup.tar.bz2 `echo $EXCLUDE` *

```

it worked fine and i can get a fully working system with this, but when i want to emerge / compile it fails

and i dont know why, i hope someone can help me out.

----------

## Sheepdogj15

actually hey BlinkEye, i wonder if there's a way to set it up to save your kernal .config file even if you don't save your sources. that if you have to dokernel recomplies later on you don't have to try to remember all the settings you had.   i'm sure it's doable, i just don't know enough about bash scripting to try to customize it in  :Confused: 

----------

## -valheru-

Great script. I had a spare hard drive so figured, why not try it? Script ran perfectly, popped in new drive, followed instructions, seemed smooth till the portgage sync. Looked like it went all the way through however at the end I received this error:

```
Updating Portgage cache:

Traceback (most recent callout):

flie "/usr/bin/emerge",line 2705,in)

oldcate=portage.catsplit(cp_list[0])[0]

Index Error:list (forgot this part, can't read my own handwriting)
```

Anyway, followed the rest oft he instructions and system booted up just like you said. Perhaps b/c of the above error but the restored system seems very sluggish. Programs take a long time to load, etc. Scrolling has intermittent pauses. Any ideas?

Overall, this is a great script and I am sure this has something to do with me and me alone. Thanks.

----------

## Ejunkie

 *Sheepdogj15 wrote:*   

> actually hey BlinkEye, i wonder if there's a way to set it up to save your kernal .config file even if you don't save your sources. that if you have to dokernel recomplies later on you don't have to try to remember all the settings you had.   i'm sure it's doable, i just don't know enough about bash scripting to try to customize it in 

 

put it under "custom_include_list"

like so

```

custom_include_list="

/usr/src/linux/.config

or

/usr/src/linux-`uname -r`/.config

"

```

----------

## -valheru-

To be on the safe side I am repeating the restore process and will advise. Portgage is syncing right now.   :Razz: 

EDIT: I noticed before and I'm sure I will again after the portage syncs that these commands:

 *Quote:*   

> 10. mkdir /mnt/gentoo/proc 
> 
> 11. mount -t proc none /mnt/gentoo/proc 
> 
> 12. mkdir /mnt/gentoo/dev 
> ...

 

Don't do anything because the proc and dev directories are already there. Does this only apply for someone who does the interactive, FULL, backup?

ThanksLast edited by -valheru- on Mon Nov 14, 2005 6:28 pm; edited 1 time in total

----------

## BlinkEye

 *Ejunkie wrote:*   

>  *Sheepdogj15 wrote:*   actually hey BlinkEye, i wonder if there's a way to set it up to save your kernal .config file even if you don't save your sources. that if you have to dokernel recomplies later on you don't have to try to remember all the settings you had.   i'm sure it's doable, i just don't know enough about bash scripting to try to customize it in  
> 
> put it under "custom_include_list"
> 
> like so
> ...

 

hmm, i had this very line in the custom_include_list myself. must have dropt out while customizing the script. 

 *-valheru- wrote:*   

> To be on the safe side I am repeating the restore process and will advise. Portgage is syncing right now.  
> 
> EDIT: I noticed before and I'm sure I will again after the portage syncs that these commands:
> 
>  *Quote:*   10. mkdir /mnt/gentoo/proc 
> ...

 

well, you mount a partition onto /mnt/gentoo in step 6. if you have a /dev and /proc directory still remaining on that partition you do not need to mkdir them (just to be sure - you don't extract the stage4 onto a partition/harddisk where you had a running (gentoo) linux installation, do you? if so, "delete" all the files by recreating a filesystem for example).

----------

## -valheru-

This was installed upon a freshly formated hard drive. I had guessed it was due to the my doing a full interactive backup. Since I used fdisk to delete and then re-create the filesystems, those directories should not have existed correct? Does doing the full, interactive backup save those directories? Thanks again.

EDIT: this happened 2x actually. Any ideas?

----------

## Ejunkie

 *-valheru- wrote:*   

> This was installed upon a freshly formated hard drive. I had guessed it was due to the my doing a full interactive backup. Since I used fdisk to delete and then re-create the filesystems, those directories should not have existed correct? Does doing the full, interactive backup save those directories? Thanks again.
> 
> EDIT: this happened 2x actually. Any ideas?

 

are the maps /mnt/gentoo/proc and /mnt/gentoo/dev/ are those maps empty if thay do just mount then else clear the partition again.

or do "dd if=/dev/zero of=/dev/hda2" ow where else you want to install gentoo.

----------

## -valheru-

The hard drive the backup is being restored to is a freshly partitioned file system so there is not an existing proc or dev directory. I'll try "dd if=/dev/zero of=/dev/hdaX" and see what happens. I guess that clears or formats the drive?

I'll let you know what happens. I just thought the backup script backed up my existing proc and dev if I did the full, interactive backup telling it to backup everything. No biggie.

EDIT: I reviewed the script. Seems it does backup the proc and dev folders. So no problem then. Thanks. See below:

```
# these files/directories are always excluded. don't add trailing slashes. 

# don't touch it unless you know what you are doing! 

# /var/db and /var/cache/edb are intentionally added here. they are listed 

# in $default_include_folders 

default_exclude_list=" 

/dev 

/lost+found 

/mnt 

/proc 

/sys 

/tmp 

/usr/portage 

/usr/src 

/var/log 

/var/tmp 

/var/db 

/var/cache/edb 

$stage4Location 

`echo $CCACHE_DIR`" 

# files/devices/folders, which need to be backed up (preserve folder structure). 

# don't touch it unless you know what you are doing! no recursive backup of folders. 

# use $default_include_folders instead. 

default_include_files=" 

/dev/null 

/dev/console 

/home 

/mnt 

/proc 

/sys 

/tmp 

/usr/portage 

/usr/src 

/var/log/emerge.log" 
```

----------

## BlinkEye

i made some minor changes to the howto and the script itself. i tried to include all the tips given since the last time. i inserted a new link at the beginning of the first post - i've put the howto into my wiki as it's far easier to maintain and to read.

----------

## -valheru-

Cool. What is the link to your wiki?

EDIT: Nevermind, followed the link in your signature to the other forum which has a link to your WIKI. THANKS   :Razz: 

EDIT: YOU PLACED my name in your WIKI-------WHHOOOOOHOOOOOOOOO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

 :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing:   :Laughing: 

----------

## paulb787

i dont understand steps 10 and 11 under restore? are these steps nessacary if u have done a full(everything) backup. because i have /proc  direstory?

10. mount -t proc none /mnt/gentoo/proc

11. mount -o bind /dev /mnt/gentoo/dev

----------

## -valheru-

well you still have to MOUNT them, just not MKDIR them.

----------

## b8zs

If I have /dev/md1 as /

and /dev/md0 as /boot

Would I be able to use this script to backup my machine?

I'm thinking just make it copy over my raidtab file, but will I run into any snags with a software raid0?

----------

## Ejunkie

 *b8zs wrote:*   

> If I have /dev/md1 as /
> 
> and /dev/md0 as /boot
> 
> Would I be able to use this script to backup my machine?
> ...

 

yes for so var i know you can use it.

----------

## zigver

Not really a problem with the script but has anyone noticed that 'uname -r' for the latest gentoo-sources doesn't report what is in /usr/src?  It's easily fixed with a link but an odd issue.  Has this always been the case and I just didn't notice?

----------

## satanskin

Ok, not sure if this has been mentioned before or not but it seems that no matter what I do, the integrity check at the end always fails. I get this at the end: 

```
* stage4 is done

* umounting boot

* Checking integrity

* Integrity check failed. Re-run the script and check your hardware.

thor satanskin # 

```

I'm not sure why it would fail so many times when nothing should be wrong. Any ideas?

----------

## BlinkEye

this integrity check happens to fail for me too sometimes. i'm not yet sure why this happens - because i'm almost confident the stage4 is intact anyway. 

do you still have suche a "broken" stage4 ? please run the following command and post the output:

```
gzip --test  stage4.tar.gz
```

or 

```
bzip2 --test  stage4.tar.bz2
```

----------

## satanskin

Yes, it still says that it is failing. In regards to the command you suggested:

```
thor satanskin # bzip2 --test /mnt/tyr/stage4/thor-stage4-2005.11.23-custom.tar.bz2_a

bzip2: /mnt/tyr/stage4/thor-stage4-2005.11.23-custom.tar.bz2_a: data integrity (CRC) error in data

You can use the `bzip2recover' program to attempt to recover

data from undamaged sections of corrupted files.

thor satanskin # 

```

----------

## BlinkEye

ah, you use a splitted stage4. could you please post the output of:

```
cat stage4.bz2_* | bzip2 --test
```

----------

## satanskin

```
thor satanskin # cat /mnt/tyr/stage4/thor-stage4-2005.11.23-custom.tar.bz2_* | bzip2 --test

bzip2: (stdin): data integrity (CRC) error in data

You can use the `bzip2recover' program to attempt to recover

data from undamaged sections of corrupted files.

thor satanskin # 

```

----------

## BlinkEye

ok, I think this really is a tar/gzip/bzip2 (or for that matter a hardware) issue and AFAIK has nothing to do with my script. 

http://www.gzip.org/#faq1

http://mail-index.netbsd.org/current-users/1999/11/02/0012.html (very old)

I suggest you try to contact a tar/bzip2 users list, maybe you find the problem. did you ever extract such a faulty stage4? have you  tried on a different box?

----------

## satanskin

no, haven't tried extracting it yet, but perhaps i can try it in a few days. Thanks for trying though man and thanks for the nice script!

----------

## -valheru-

I too get errors trying to bz2 with a file over 2.5 gigs. File intergrity error.

gz is fine though.

----------

## carpman

 *-valheru- wrote:*   

> I too get errors trying to bz2 with a file over 2.5 gigs. File intergrity error.
> 
> gz is fine though.

 

i have a 5gb backup that is fine!

----------

## bingobob

This script is great. One thing though. I would like to run it each week from a cron script. How can i make it non-interactive?

----------

## amaroc

BlinkEye,

great script, really easy to use.

I'm doing the minimal-tgz-backup to my Buffalo-LinkStation via WLAN. The ftp-backupdrive is mounted via lufs

```
lufis fs=ftpfs,host=my_host,username=foo,password=bar,ftpactive /mnt/backups/ 
```

and it works flawlessly. 

Thanks!

PS: I did not modify the script to handle /home, /usr/src, etc. according to my needs. Such directories will be backuped using krusader's built-in ftp- and archive-capabilities.

----------

## andrewd18

I'm getting an issue - it can't pass the integrity check...

```
StubbornAesthetics ~ # ./mkstage4.sh

Backup script v3.6

==================

What do you want to do? (Use CONTROL-C to abort)

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup

Please enter your option: 3

* mounting boot

Do you want to continue? (y/n) y

* creating directory /media/IOMEGA_HDD/Backups/stage4

* Please wait while the stage4 is being created.

sh: line 1:  2708 Broken pipe             /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /media -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /media/IOMEGA_HDD/Backups/stage4 -prune -o -path /home/andrew -prune -o -path /home/lost+found -prune -o -path /usr/src/linux-2.6.14-gentoo-r2 -prune -o -print

sh: line 1:  2663 Done(141)               ( /usr/bin/find /var/db; echo /usr/src/linux-2.6.14-gentoo-r2/.config; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /media; echo /mnt/floppy/.keep; echo /mnt/cdrom/.keep; echo /mnt/.keep; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /* -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /media -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /var/cache/edb -prune -o -path /media/IOMEGA_HDD/Backups/stage4 -prune -o -path /home/andrew -prune -o -path /home/lost+found -prune -o -path /usr/src/linux-2.6.14-gentoo-r2 -prune -o -print )

      2665 Broken pipe             | /bin/tar --bzip2 --preserve-permissions --create --absolute-names --totals --ignore-failed-read --file /media/IOMEGA_HDD/Backups/stage4/StubbornAesthetics-stage4-2005.12.04-minimal.tar.bz2 --no-recursion -T -

* stage4 is done

* umounting boot

* Checking integrity

* Integrity check failed. Re-run the script and check your hardware.
```

Suggestions? Same problem funkoolow had, but there was no answer posted to his inquiry.

~~ Andrew D.

----------

## BlinkEye

i wished i'd receive a mail when this thread gets a new post - sometimes it simply doesn't work (same problem with my phpbb forum) ...

 *bingobob wrote:*   

> This script is great. One thing though. I would like to run it each week from a cron script. How can i make it non-interactive?

 

read me

 *-valheru- wrote:*   

> I too get errors trying to bz2 with a file over 2.5 gigs. File intergrity error.
> 
> gz is fine though.

 

interesting. you may really mail a bz2 users list with this issue.

 *andrewd18 wrote:*   

> I'm getting an issue - it can't pass the integrity check...
> 
> ```
> StubbornAesthetics ~ # ./mkstage4.sh
> 
> ...

 

there's a copy and paste error ( -> broken pipe ... ). try downloading it.

----------

## andrewd18

 *Blink Eye wrote:*   

> there's a copy and paste error ( -> broken pipe ... ). try downloading it.

 

I got the file from doing your wget command, so it might be something there. Come to think of it, my file was over 2.5 Gigs, so -valheru-'s error might apply to me as well. I'll try tar.gz and see how that goes.

~~ Andrew D.

----------

## tgh

 *bingobob wrote:*   

> This script is great. One thing though. I would like to run it each week from a cron script. How can i make it non-interactive?

 

My guess would be to create a response file, with the keypresses that you need in order to answer the questions.  Then pipe that response file into the script, something like:

mkstage4.sh < response.txt

The response.txt file might look like:

```
y

1
```

(I haven't tried this personally.)

----------

## BlinkEye

this has been posted already and is explained it the wiki version (meaning: it works   :Wink:  )

----------

## satanskin

My files were split at a little under 4Gb as well. The bz file fails integrity check and the gz file passes with fly colors.

----------

## carpman

Ok here is one that not sure if been discussed but woth mentioning. i did a full backup /home etc, thing is i had a dvd mounted and script did backup of /media/cdrecorder (this what my dvd/cdwriter is mounted as) could be an idea to add /media as a default excluded dir!

----------

## BlinkEye

ah, /media. i guess this is a new dir due to the lsb adaption. i'll add it to the default exclude dir, thanks.

----------

## carpman

 *BlinkEye wrote:*   

> ah, /media. i guess this is a new dir due to the lsb adaption. i'll add it to the default exclude dir, thanks.

 

pleased to help out. also don't forget /srv this is server related

----------

## allucid

Is it possible to combine this method with one of the 'Make you own Gentoo LiveCD' howtos? That way you have a rescue/recovery disk all in one.  :Smile: 

----------

## BlinkEye

yes of course. it has shortly been mentioned here (although i answered the question not quite correctly about the use of a livecd - i just haven't had the need for it as i do have the gentoo installation of my laptop cloned on a external usb harddisk (my stage4 test installation  :Wink:  ) ).

----------

## carpman

Hello, i have a probelm though not sure if it is csript error or because of what i am trying, i have a working amd XP system and did a stage so as to copy install to a duron 1100 system, although different cpu it should work fine as can use same cflags, the early XP just being a duron with extra cache, thing is the XP i am using is mobile XP2500+.

Problem is that after restoring to new machine and when in chroot i cannot run:

```

nano (illegal operation)

or do 

make menuconfig (complains about no ncurses-devel lib)

or do

make && make modules_install

```

emerge works on some things but not others?

So question is, is this a result of a bad stage4 build/restore or just that restore will not work on new system?

cheers

----------

## carlosman

First of all, Blinkeye, THANK YOU!!!  Great script...

My question:

In your "GNU/Linux System Backup Script (stage4)" wiki you state:

 *Quote:*   

> "march" breaks compatibility, "mcpu" doesn't. There is a simple workaround: replace "march" with "mcpu" and recompile your whole system:

 

I upgraded from GCC 3.3.6 to 3.4.4, as part of a general update.  I have been using the -mcpu flag, following your advice.  Now the GCC 3.4.4 prints this warning message:

 *Quote:*   

> `-mcpu=' is deprecated. Use `-mtune=' or '-march=' instead.

 

I have been reading about it and it seems like the -mcpu functionality is not replicated by -mtune.

Where does that leave us for portability?

P.S.: Just after posting this, I found:

http://www.dis.com/gnu/gcc/i386-and-x86-64-Options.html

It seems like -mtune is indeed the new way of saying "-mcpu":    :Embarassed: 

 *Quote:*   

> -mcpu=cpu-type
> 
> A deprecated synonym for -mtune.

 

----------

## BlinkEye

thanks, i'll update the instructions.

 *http://gcc.gnu.org/onlinedocs/gcc-3.4.4/gcc/i386-and-x86_002d64-Options.html wrote:*   

> -march=cpu-type
> 
>     Generate instructions for the machine type cpu-type. The choices for cpu-type are the same as for -mtune. Moreover, specifying -march=cpu-type implies -mtune=cpu-type.
> 
> -mcpu=cpu-type
> ...

 

in short: use -mtune instead of -mcpu.

----------

## billman

Awsome - it works great. Me being a newbie, I was hoping for some direction. Questions:

1. How can I be certain the backup has all the required files for a complete restore - What are the necessary files it should have including "/home". 

2. How can I use the application vixie-cron to do incremential backup or full back ups to a vfat drive and have it delete the past weeks or something like that?

3. If I need to restore the box and have the ability to boot from an old kernel - what would be the steps.

I know these might be silly questions but new to linux I am and determind to understand and live in the Gentoo realm I am too. Hence I have had to restore the box many times before through v2I (Windoze). I no longer want to depend on that and I want a fail safe solution.

Thanks,

Bill

----------

## twitch

I have had problems restoring hardened gentoo (SELinux + GRSecurity).  I think I have worked through most of the kinks, here is my restore routine:

SELinux + GRSecurity Restore from stage4:

```
 1. boot off a live-cd and repartition your harddisks and create filesystems as necessary (remove any files left on disk)

 2. eventually reboot, using option: gentoo docache

   // If this doesn't work, ie. not enough room, you can leave the cd in and restore from a usb drive

 3. # umount /mnt/cdrom

 4. remove the live-cd and insert the cd with the stage4

 5. # mount /dev/cdrom /mnt/cdrom

 6. # mount /dev/hdaX /mnt/gentoo

 7. # mkdir /mnt/gentoo/boot

 8. # mount /dev/hdaX /mnt/gentoo/boot

 9. # tar xzvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.gz -C /mnt/gentoo/

     or

   # tar xjvpf /mnt/cdrom/host-stage4-18.04.2005-custom.tar.bz2 -C /mnt/gentoo/

10. # mount -t proc none /mnt/gentoo/proc

11. # mount -o bind /dev /mnt/gentoo/dev

12. # mount -t selinuxfs none /mnt/gentoo/selinux

13. fix the system date using the date command

14. # chroot /mnt/gentoo /bin/bash

15. # env-update

16. # source /etc/profile

17. if in need adjust necessary files (/etc/fstab, /boot/grub/grub.conf) and/or install grub

18. # emerge sync (rebuild portage tree)

19. install grub

20. # cp /proc/mounts /etc/mtab

21. run grub-install

22. # cd /etc/security/selinux/src/policy

23. # make load

24. # make chroot_relabel

25. # exit

26. # cd /

27. # cp -r /dev/* /mnt/gentoo/dev/   

      /* Had to do this because on reboot hda could not be found, was getting a superblock error on the / partition */

28. # umount /mnt/cdrom

29. # remove backup cd

30. # umount /mnt/gentoo/boot

31. # umount /mnt/gentoo/dev

32. # umount /mnt/gentoo/proc

33. # umount /mnt/gentoo/selinux

34. # umount /mnt/gentoo

35. # reboot 

When I rebooted I had to make some files suid, apparently that permission did not copy over. (I figured this out because su was not working)...  My list of files I had to edit:

   /bin/passwd

   /bin/su

   /usr/bin/crontab

   /usr/bin/gpasswd

   /usr/lib/misc/ssh-keysign

   /usr/libexec/lockspool

   /usr/sbin/unix_chkpwd

I also needed to run the following after rebooting:

   # python-updater

   # emerge  --update --deep world

   # emerge qmail (it was complaining about a log directory not existing)

   # cd /etc/security/selinux/src/policy

   # make clean

   # make install

   # make load

   # make relabel
```

Thanks for the script!

----------

## BlinkEye

 *billman wrote:*   

> Awsome - it works great. Me being a newbie, I was hoping for some direction. Questions:
> 
> 1. How can I be certain the backup has all the required files for a complete restore - What are the necessary files it should have including "/home". 

 

this is what the script is for - you should not worry about what you want to backup but what you don't want. if you haven't changed anything and chose a custom stage4 where you included /home/username you should be set.

 *billman wrote:*   

> 2. How can I use the application vixie-cron to do incremential backup or full back ups to a vfat drive and have it delete the past weeks or something like that?

 

the script is for full backups only - about the syntax of a cron job you either search the web/forum or wait until i put it in my wiki. unfortunately i'm delayed as i screwed up my main workstation (and as a matter of fact i do not do backups on my own boxes ( i mean, i know what i do, right?  :Evil or Very Mad:  ) - well, isn't this ironically? i'm just about to lose all my data because i can't get my raid0 up and running again.

 *billman wrote:*   

> 3. If I need to restore the box and have the ability to boot from an old kernel - what would be the steps. 

 

i don't understand the question. if you have to restore your box, you have lost/screwed your data/partition whatsoever. after extracting your stage4 you'll get your kernel back and may boot from them. of course, you may boot your system with another kernel. follow the restore section - nothing changes about that. you just have to adjust your boot loader that it loads the particular kernel.

 *billman wrote:*   

> I know these might be silly questions but new to linux I am and determind to understand and live in the Gentoo realm I am too. Hence I have had to restore the box many times before through v2I (Windoze). I no longer want to depend on that and I want a fail safe solution.

 

if you want to be sure you'll have to try it on a seperate harddisk or a partition big enough to take your extracted stage4.

----------

## BlinkEye

 *twitch wrote:*   

> I have had problems restoring hardened gentoo (SELinux + GRSecurity).  I think I have worked through most of the kinks

 

sorry to hear that - and i have only an answer for one problem: it's possible that some daemons do not start because you did not backup the /var/log/daemon_log directory. but you could add that directory to the $default_include_files for the next time. (on second thought it might be a good idea to do that automatically. i'll have a look into it). why you had to adjust permissions i do not understand. could it be that you extracted the stage4 without the -p option (meaning without preserving the permissions)? please extract the stage4 again (don't forget the -p/--same-permissions option) within some directory and check your permissions of /etc/passwd and so on ...

----------

## Lonnie

Just wanted to say that I finally took the plunge and did the backup/restore.   I don't know just what I was doing wrong, but the whole thing gave me fits for a week.   I even had serious problems with trying to do a completely new install.

Managed to get both to reboot, but the restore was messed up on my partitions, finally figured that out today.

The new install would boot up, but the screen was impossible to read.  Haven't a clue what I did wrong there.

So, today, I was going to give it one last shot before trying something else.  I finally mounted ALL of my partitions after I had installed the stage4 dvd.  Followed the instructions and got the restore to work.  umounted ALL the partitions and rebooted.

Started startxfce4 and it acted like it was just hung, so did an 'emerge --deep --update world', the restore was already a week old.

Anyway, I had already learned that this system can be very slow coming back up from something major, so I restarted startxfce4 and when off on another project.  When I came back, X was up, and as of right now, everthing I have tried is working very well, thank you.

My only Real problem today was that firefox didn't want to compile(x86_64 system).  It failed on me a couple times.   Found out about the 'emerge --skipfirst --resume' and got through the update world.  Then I tried to emerge firefox again, and it went.

So, other than the rank-newbie errors, the stage4 work's GREAT.  I went from a 100% full /usr on a 40gig HD to plenty of room on an 80gig with a 40gig just for storage.   And, I was very pleasantly surprised to find that it backed up and restored Everything I was concerned about loosing.

Thanks BlinkEye, another very happy user of your great script.

----------

## mattjgalloway

Hey,

I just started using this script, and I have the following set:

```
default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/var/log

/var/tmp

/var/db

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

default_include_files="

/dev/null

/dev/console

/home

/storage

/backup

/steam

/opt

/mnt

`find /mnt -name .keep`

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log

/usr/src/linux-`uname -r`/.config"
```

And I find that /usr/portage and all files under it are backed up!? Surely they shouldn't be, only the directory structure...

----------

## BlinkEye

you removed /usr/portage (which you should readd) from default_exclude_list:

 *mkstage4.sh wrote:*   

> 
> 
> # these files/directories are always excluded. don't add trailing slashes.
> 
> # don't touch it unless you know what you are doing!
> ...

 

----------

## mattjgalloway

Hi.

Thanks for that! I really have no idea why I'd deleted them from there, but that makes sense now!

Thanks!

----------

## mattjgalloway

That seems to be working now, but I thought of another thing...

Perhaps there could be a progress bar incorporated into the script... It could just be along the lines of files left to do / total files to get the percentage done. Obviously a better way would be the size of files, but it'd be a good start at least.

Is that do-able do you think?

----------

## carpman

hello, ok just done a stage4 backup and restore so i could change FS type and layout.

It worked ok except for couple of issues, one was firefox profile lock file which needed deleting as i could not load firefox as it complained that profile was in use. My touchpad stopped working but just needed to change device type from event one to auto.

I do have one more issue and that is cpufreqd, i have it set so when computer at idel it drops to lowest cpu setting 533mhz and when under load it goes up in stages to 1600mhz, thing is that after restore it has not gone higer than 533 even though i now compiling perl update?

Not sure what is occuring but was working fine before restore, have checked cpifreqd conf.

cheers

----------

## BlinkEye

 *carpman wrote:*   

> hello, ok just done a stage4 backup and restore so i could change FS type and layout.
> 
> It worked ok except for couple of issues, one was firefox profile lock file which needed deleting as i could not load firefox as it complained that profile was in use. My touchpad stopped working but just needed to change device type from event one to auto.
> 
> I do have one more issue and that is cpufreqd, i have it set so when computer at idel it drops to lowest cpu setting 533mhz and when under load it goes up in stages to 1600mhz, thing is that after restore it has not gone higer than 533 even though i now compiling perl update?
> ...

 

i guess you tried to re-emerge cpufreqd? maybe a device is missing and hence cpufreqd cannot reset the speed. the /dev folder does not be backed up and as of udev there may be some nodes which have to be recreated?

----------

## carpman

Seems to have started working but still have one issue, /var/log/portage i created this and given it lots chmod options, it is currently 777 but still emerges complain it is not writable? i set it to owner/group 'portage' as is on other systems but no joy?

PS great script otherwise  :Smile: 

Cheers

----------

## BlinkEye

 *mattjgalloway wrote:*   

> That seems to be working now, but I thought of another thing...
> 
> Perhaps there could be a progress bar incorporated into the script... It could just be along the lines of files left to do / total files to get the percentage done. Obviously a better way would be the size of files, but it'd be a good start at least.
> 
> Is that do-able do you think?

 

yes, i think so. if you use the script with the verbose option ( -v/ --verbose ) you see the command it does. removing the tar command and combining it with 

```
 | wc -l 
```

gives you the amount of files the command will process to tar. for one box of mine this would be about 166000 files (counted within a few seconds) - so, still no loss. i think the number of files suffice for the progress bar. now, using the vebose option will give you the files processed by tar - and from there one could create this progress bar.  i need a command that can say, print a '#' for every 100 lines being output on stdout. any ideas?

----------

## BlinkEye

 *carpman wrote:*   

> Seems to have started working but still have one issue, /var/log/portage i created this and given it lots chmod options, it is currently 777 but still emerges complain it is not writable? i set it to owner/group 'portage' as is on other systems but no joy?
> 
> PS great script otherwise 
> 
> Cheers

 

could it be you you created a file instead of a directory?

```
drwxrws---  2 portage portage   114136 Jan 16 00:28 portage
```

----------

## gnychis

for some reason my custom exclude folders do not seem to be working

I try to exclude /160gb and /200gb, however whenever I run the script in verbose mode it is recursively backing up all of /160gb and /200gb

when i start the script i use option 4

here is my file:

```

#!/bin/bash 

# Backup script for Gentoo Linux 

# Copyright Reto Glauser aka Blinkeye 

# Distributed under the terms of the GNU General Public License v2

# Mailto: stage4 at blinkeye dot ch

# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html 

# Date: 2005-06-30

version=v3.5

basename=`basename $0`

find=/usr/bin/find

tar=/bin/tar

# these are the commands we actually need for the backup

command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.

for command in ${command_list[@]}; do

   if [ ! -x "`which $command 2>&1`" ]; then

      echo -e "\nERROR: $command not found! "

      base=`basename $command`

       if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then

          echo -e "ERROR: $base is not in your \$PATH."

       fi

       exit -1 

   fi

done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# Defaults to creating one tarball

tar_output="--file"

# split command

split_options="--suffix-length=1 --bytes=685m"

# options for the tar command 

tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read" 

# where to put the stage4

stage4Location=/mnt/backups/stage4

# name prefix

stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`

# patterns which should not be backed up (like iso files).

# example: default_exclude_pattern="*.iso *.divx"

# These pattern count only for files NOT listed in the $custom_include_list.

default_exclude_pattern=""

# these files/directories are always excluded. don't add trailing slashes.

# don't touch it unless you know what you are doing!

# /var/db and /var/cache/edb are intentionally added here. they are listed

# in $default_include_folders

default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

/var/db

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

# files/devices/folders, which need to be backed up (preserve folder structure). 

# don't touch it unless you know what you are doing! no recursive backup of folders. 

# use $default_include_folders instead.

default_include_files="

/dev/null

/dev/console

/home

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log"

# folders, which need to be backed up recursively on every backup. 

# don't touch it unless you know what you are doing! the reason for this

# variable is that some users add /var to the $default_exclude_list. here 

# we ensure that portage's memory is backed up in any case.

default_include_folders="

/var/db"

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will

# include/exclude these files/folders depening on your answer.

custom_include_list="

/home/*

/usr/src/linux-`uname -r`"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT

# be backed up. eg. 

#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"

custom_exclude_list="/home/hedpe/server /home/hedpe/ut2004 /home/hedpe/downloads /home/hedpe/.rstw16 /200gb /160gb"

# Only files/folders within the $custom_include_list are checked against these patterns

# custom_exclude_pattern="*.mp3 *.iso"

custom_exclude_pattern="*.img *.VOB *.BUP *.mp3 *.iso *.gcm"

# the find_command

find_command="$find /*"

# don't backup anything which matches pattern listed in $default_exclude_pattern

for pattern in $default_exclude_pattern; do

   find_command="$find_command -not -name $pattern"

done

# assemble the find_command 

function find_files()

{

   for folder in $default_exclude_list; do

      find_command="$find_command -path $folder -prune -o"

   done

   find_command="$find_command -print"

   for i in $default_include_files; do

      find_command="echo $i; $find_command"

   done   

   for i in $default_include_folders; do

      if [ -d $i ]; then 

         find_command="$find $i; $find_command"

      else

         find_command="echo $i; $find_command"

      fi      

   done

}

# check the exclude/include variables for non-existing entries

function verify()

{

   for i in $1; do

      if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then

         echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2

         exit 0

      fi

   done   

}

# check input parameters

while [ $1 ]; do

   case  $1 in

   "-h" | "--help")

      echo -e $help

      exit 0;;

   "-v" | "--verbose")

      verbose=$1;;

   "-s" | "--split")

      tar_output="--split";;

   "");;

   *) 

      echo -e $help

      exit 0;;

   esac

   shift

done

echo ""

# check folder/files listed in $default_exclude_list exist

verify "$default_exclude_list" "\$default_exclude_list"

# check files listed in $default_include_files exist

verify "$default_include_files" "\$default_include_files"

# check folder listed in $default_include_folders exist

verify "$default_include_folders" "\$default_include_folders"

#check folder listed in $custom_include_list exist

verify "$custom_include_list" "\$custom_include_list"

#check folder listed in $custom_exclude_list exist

verify "$custom_exclude_list" "\$custom_exclude_list"

 

# print out the version

 echo -e "\nBackup script $version"

 echo -e "=================="

# how do you want to backup?

echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup\n"

while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do

   echo -en "Please enter your option: "

   read option

done

case $option in

[1,3])

   stage4Name=$stage4Location/$stage4prefix-minimal.tar;;

[2,4])

   stage4Name=$stage4Location/$stage4prefix-custom.tar

   for folder in $custom_include_list; do 

      echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 

      read answer 

      while [ "$answer" != 'y' -a "$answer" != 'n' ]; do 

         echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 

         read answer 

      done 

      if [ "$answer" == 'n' ]; then 

         find_command="$find_command -path $folder -prune -o"

      else

         custom_find="$find $folder"

         for i in $custom_exclude_pattern; do

            custom_find="$custom_find -name $i -o"

         done

         for i in $custom_exclude_list; do

            custom_find="$custom_find -path $i -prune -o"

         done

         find_command="$custom_find -print; $find_command"

      fi

   done ;;

esac

# add $custom_include_list to the $default_exclude_list as we assembled

# $custom_find with $custom_include_list already.

default_exclude_list="$default_exclude_list $custom_include_list"

case $option in

[1,2])

   stage4postfix="gz"

   zip="--gzip";;

   

[3,4])

   stage4postfix="bz2" 

   zip="--bzip2";;

esac

# mount boot

echo -e "\n* mounting boot" 

mount /boot >/dev/null 2>&1

# find the files/folder to backup

find_files

find_command="($find_command)"

# create the final command

if [ "$tar_output" == "--file" ]; then

   tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"

else

   tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"

fi

if [ "$verbose" ]; then

   echo -ne "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command

fi

# everything is set, are you sure to continue?

echo -ne "\nDo you want to continue? (y/n) " 

read answer 

while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do 

   echo -ne "Do you want to continue? (y/n) " 

   read answer 

done 

if [ "$answer" == 'y' ]; then 

   # check whether the file already exists.

   if [ "$tar_output" == "--split" ]; then

      overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"

   else

      overwrite="$stage4Name.$stage4postfix"

   fi

   

   if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then  

      echo -en "\nDo you want to overwrite $overwrite? (y/n) "

      read answer

      while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

         echo -en "Do you want to overwrite $overwrite? (y/n) "

         read answer

      done

      if [ "$answer" == 'n' ]; then

         echo -e "\n* There's nothing to do ... Exiting" 

         exit 0;

      fi

   fi

   

   # if necessary, create the stage4Location 

   if [ ! -d "$stage4Location" ] ; then 

      echo "* creating directory $stage4Location" 

      mkdir -p $stage4Location 

   fi

   echo -e "\n* Please wait while the stage4 is being created.\n"

   # do the backup.   

   sh -c "$tar_command"

   

   # finished, clean up

   echo -e "\n* stage4 is done" 

   echo "* umounting boot" 

   umount /boot >/dev/null 2>&1

   # Integrity check

   echo -e "* Checking integrity"

   if [ "$zip" == "--gzip" ]; then

      zip="gzip"

   else

      zip="bzip2"

   fi   

   

   if [ "$tar_output" == "--split" ]; then

      if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then

         echo -e "* Integrity check failed. Re-run the script and check your hardware."

         exit -1

      fi

   else

      if [ "`$zip --test  $stage4Name.$stage4postfix 2>&1`" != "" ]; then

         echo -e "* Integrity check failed. Re-run the script and check your hardware."

         exit -1

      fi

   fi

   

   # everything went smoothly"

   echo -e "* Everything went smoothly. You successfully created a stage4."

   

else

   echo -e "\n* There's nothing to do ... Exiting" 

fi 

# Split the archive into chunks - uncomment the 3 lines if you want to split the stage4

# echo -e "* split $stage4Name.$stage4postfix" 

# split $split_options $stage4Name.$stage4postfix "$stage4Name.$stage4postfix"_ 

# echo "* splitting is done"

```

----------

## gnychis

problem solved by adding the folders to default_exclude_list

 :Smile: 

----------

## mb

Hi *,

I've got a small suggestion for your (excellent) script. Maybe you should add the dhcpcd cache files to the default exclude list (/var/lib/dhcdc/dhcpcd*.cache). I made a stage4 file on my test server at home, transfered it to my root server, rebooted.. and.. lost connection, because the root server took the (old) ip from the cache file...

#mb

----------

## ewildgoose

My apologies if this has already been mentioned, but also check out app-arch/star for an interesting alternative to "tar".  This alternative app has support for backing up and restoring selinux style extra attributes (which I need in my case) - It may also have the overlapping exclude/include support that tar is also missing?

I mentioned kdar already, but that's another interesting alternative as well.

Nice script though

----------

## BlinkEye

thanks for the hint about star. as any page about star has been moved i cannot test if it is fully compatible with the tar options used in my script.

----------

## ewildgoose

Top hit in google

http://cdrecord.berlios.de/old/private/man/star.html

I am playing with it right now, and at first sight it's not quite right.  Firstly it clobbers your existing tar binary when it installs and the options are not similar enough that this is acceptable.  The only thing I can see which might actually help in the end is: 

- Multi-archive

- Advanced regexp for excludes

- Archiving of linux attributes

So kdar still looks like a better solution, but the negative there was getting a binary when you needed to restore the system in a hurry (from scratch).

Unsure...

----------

## ewildgoose

Hmm, just noticed an interesting feature at:

http://www.gnu.org/software/tar/manual/html_mono/tar.html#SEC57

If you "cat" together two archives then try and list the files you will only see the files from the first archive.  However, if you add the "-i" option then you can "workaround this" and extract the entire archive.  I just tried creating two tar files, then cat'ing them together, then gziping them, and then I could read the whole thing with "tar -tif test.tar.gz"

It's a bit of a hack, but interesting...

Ed

----------

## BlinkEye

 *ewildgoose wrote:*   

> Top hit in google
> 
> http://cdrecord.berlios.de/old/private/man/star.html.

 

i doubt that. and the link doesn't work anyway (only cached). neither does http://www.fokus.gmd.de/research/cc/glone/employees/joerg.schilling/private/star.html which is the homepage referenced by portage. but thanks for checking it out.

----------

## ewildgoose

The link *I* posted works fine.  The one *you* posted has an extra "." at the end?

I dont see that star actually solves the problem in this case, but it does offer a possible solution for selinux backups...  Also the concatenating two tar archives together thing nearly works, but I can't quite see how to efficiently stream the output of two tar commands into bzip?  Any thoughts?

Ed

----------

## Matteo Azzali

Hum,Blinkeye, sorry to bother, but I would like to build a visual frontend in KDE for this wonderful script, 

(Kommander script, check AviUtils or Kfirewatcher to see what I've done before) and I was

swearing if there is some better option to add dirs in default_exclude_list other than modifying the

script itself (an option at script startup? a parameter from command line? )

----------

## ewildgoose

It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist.  Look at any one of dozens of shell scripts on your system to see this being used in practice

Good luck

----------

## Matteo Azzali

 *ewildgoose wrote:*   

> It's just a shell variable, so set the variable in the environment before entering the script and tweak the script to only set the defaults if the environment variable doesnt exist.  Look at any one of dozens of shell scripts on your system to see this being used in practice
> 
> Good luck

 

Thank you, but that implies modifying the original script, thus forking the script for a GUI.I can do, but

doin so I would fork from original project, I'm instead asking Blinkeye to add this/some other kind of option

to make mkstage4 script gui-compatible, thus having cohordinated projects... (and thus having updated

mkstage4 script usable even for GUI users without any need of modification).

----------

## ewildgoose

The usual process on an open source project is that you just send him a diff.  Remember that the original script is on a "wiki".  You can presumably edit the darn thing yourself if you like...

It's hardly a "fork" though.  It's a tiny modification and I'm sure it will be gratefully appreciated

Good luck!

Ed

----------

## ssss25

The script gets stuck! I never get the menu. After some debugging it hangs at:

find /mnt -name .keep

which is in the 

```
default_include_files="
```

if I remove this, it works! Why would find get stuck!! if I copy and paste the line alone, it gets stuck too... any idea?

My other issues is that in default_include_files=, I added /usr/src but it didn't backup that directoy in full. it only did the linux folder.

How can I make this script tar EVERYTHING on the root partition! I only have one root part and one swap... 

Thanks.

----------

## ewildgoose

Try running the command from the command line and see if it still breaks?  I guess you could have some recursion in there which is killing the script.  OR you could have some crookedly mounted filesystem in there with is nobbling the find command.

----------

## ssss25

command line fails too.... just hangs there. I couldn't even break...

I removed the line to make the script work!

----------

## ewildgoose

Well there you go.  You have some filesystem mounted in /mnt which can't be enumarated.  Probably it's some dead NFS mount or something else like that which hangs forever when you try and read it.  Just look in /mnt for stuff that could be mounted and try using "ls" until you find the problem

----------

## ssss25

Find is useless for me! I did this at the root to find where a file is, and I got segmentation fault  :Smile: 

```
Ducky / # find . -name kdmrc -print

Segmentation fault

```

[/code]

----------

## ewildgoose

Look, this is wayyy off topic now.  The script is fine, its your system which is bolloxed.   Ask in a new thread how to fix your problem.

----------

## gnychis

hi, whenever i run ./mkstage4(unmodified)  and do a interactive backup with .tar.gz or .tar.bz2, i get this while trying to extract it:

"Tar: error exit delayed from previous errors"

then it exists, and it never extracts all the files... i've tried making the backup several times and extracting it several times, always the same error right when its about to start extracting /var

anyone else have this problem?

oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ?

Thanks!

George

----------

## opopanax

 *gnychis wrote:*   

> oh... and why doesn't it backup all of /dev, and not just /dev/console and /dev/null ? 

 

Because it doesn't have to, with udev.  you have the option to set up your system so that it has a back-up of the /dev filesystem, just in case you need it.  /dev is dynamically created at boot, and generally you don't need it--unless you're still using devfs.

Please correct me if I'm wrong, I do tend to talk from the heinie once in a while.

----------

## gnychis

I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything.  However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening?

----------

## BlinkEye

 *gnychis wrote:*   

> I followed the instructions on the first stage to restore from a stage4, before i chroot into the environment, i only have /dev/console and /dev/null ... when i chroot, /dev is filled up with everything.  However when I reboot and try to boot into gentoo, it loads the kernel fine, it starts to boot gentoo, but the only things in dev are /dev/null and /dev/console again... so when it tries to load it can't because /dev/sda3 (my root partition), doesn't exist... what could be happening?

 

I don't know exactly what the live cd does but it certainly creates some devices in /dev/ to provide a generic installation possibility. Udev should create the necessary devices upon booting - except you set in /etc/conf.d/rc that you want to use a device tarball. So, please check that file and make sure you use udev and NO device tarball.

----------

## BlinkEye

 *gnychis wrote:*   

> hi, whenever i run ./mkstage4(unmodified)  and do a interactive backup with .tar.gz or .tar.bz2, i get this while trying to extract it:
> 
> "Tar: error exit delayed from previous errors"
> 
> then it exists, and it never extracts all the files... i've tried making the backup several times and extracting it several times, always the same error right when its about to start extracting /var
> ...

 

I think I once had a file in a tarball which could not be extracted either. Could you do a verbose extracting and tell me where it stops? Let's say you exclude the subfolder where that file resides (just for debugging) - does it work then?

----------

## booyaa

Guys, I am sure this has been covered, but i would like to ask:

This is an  excellent tool.. I used it to successfully back up my system..

Upon restore, I had to recreate partitions (/boot, / , and swap)...

Correct me if I am wrong, but the default grub.conf from the gentoo handbook point the kernetl to HD(0,0)/boot/kernel-xxxxxx

Does the kernel need recompiled and bzImage need recopied to /boot?

Also, a side question, it seems as it my symlink (usr/src/linux was missing.. Anyone experience this?

I really need to get this system back the way it was.. Any help would be appreciated.

Booyaa

----------

## Sheepdogj15

 *booya wrote:*   

> Correct me if I am wrong, but the default grub.conf from the gentoo handbook point the kernetl to HD(0,0)/boot/kernel-xxxxxx 
> 
> Does the kernel need recompiled and bzImage need recopied to /boot?
> 
> 

 

did you backup /boot? i don't think that was one that is excluded by default in the stage4 script. but, for example, if /boot wasn't mounted when you made the tarball, your kernel is gone.

otherwise, you shouldn't need to recompile the kernel. just in case, you will want to double check your configuration for grub and have it reinstall to your MBR. just in case.

 *Quote:*   

> Also, a side question, it seems as it my symlink (usr/src/linux was missing.. Anyone experience this? 

 

this is expected if you didn't set the script to backup /usr/src. either way, you should just be able to make the symlink again.

----------

## booyaa

Thanks Sheep.  No I did not back up /boot, nor did I have it mounted when I made the backup.  

I recreated my partitions and unzipped stage4 to /

Then reinstalled grub and configured grub.conf..

When rebooting it says file not found:  /boot/kernel-2.6.15

Am I hosed?  Can I still restore?

----------

## Sheepdogj15

do you have your kernel sources? 

if you still have your original .config file (/usr/src/linux-[version]/.config), you can just recompile the kernel.

----------

## booyaa

I have this:

/usr/src/linux-gentoo-2.6.15-r1

I can make menuconfig from the lived CD.  It gives me the recompile options.  

I have recompiled and copied bzImage to /boot, but it still didnt work.

If I copy that to /boot what tells it where to look?  Do I need to recreate a symlink?  It is hard to believe that If I have a complete backup of / that I cant simply repartition. recompile, and reload grub and edit grub.conf

----------

## Sheepdogj15

double check your grub settings. mount /boot and make sure it has your kernel there.

come to think of it, you *should* have had the kernel as the stage4 script *does* mount /boot before tarring everything.

odd.

----------

## booyaa

I know man.  I am at this point after my first failure:

I wiped everything.

Booted live CD

Created partitions - file systems

mounted root and boot

cd /

Downloaded the stage 4 tarball

unpacked

I will now emerge sync

Ill then emerge grub and configure grub.conf

Ill then recompile and copy bzImage to /boot-kernel***

Weird cause this is what I thought I did before but grub said it couldnt find my kernel

Dumb noob question:  After copying bzImage to /boot/kernel*** , how does the bootloader know where my kernel source is?

----------

## BlinkEye

 *Sheepdogj15 wrote:*   

> double check your grub settings. mount /boot and make sure it has your kernel there.
> 
> come to think of it, you *should* have had the kernel as the stage4 script *does* mount /boot before tarring everything.
> 
> odd.

 

yes, it will mount /boot.

 *booyaa wrote:*   

> I know man.  I am at this point after my first failure:
> 
> I wiped everything.
> 
> Booted live CD
> ...

 

You have to adjust the entry in /boot/grub/grub.conf (kernel line). So, make sure they match names (the entry in /boot/grub/grub.conf and the /boot/KERNELNAME). Creating a new kernel usually means to either add a whole new grub entry block (and adjusting the kernel line appropriately) or copy the new kernel over the old kernel (not suggested).

----------

## RaZoR1394

 *Quote:*   

> 
> 
> Tar: error exit delayed from previous errors
> 
> 

 

I'm getting this error too somewhere during the extraction of /var/lib.

----------

## yabbadabbadont

First, thank you for the script.  It greatly simplifies my backup process which, unfortunately, I implemented *after* testing the new GUI installer.   :Sad: 

I used the instructions in the wiki to restore after I wiped and repartitioned my primary drive.  (put win2k back on the system for games and some work I'm doing)  It worked very well, but I did run into one issue that hasn't been mentioned in this thread, but is found many places in the forums.  Every boot into Gentoo after the restore results in warning messages while the init.d scripts run.  "Re-caching dependencies (mtimes differ)" or something close to that.  Everything worked, but the messages were annoying.  From researching the forums, I found that removing /var/lib/init.d/depcache, /var/lib/init.d/deptree, and then running depscan.sh fixed the problem.  There were also indications in some posts that re-emerging baselayout would fix the problem, but since I had already used the first method, I can't say if doing that will work.  Anyway, I just wanted to make you aware of this, if you weren't already, so that you can add this to the wiki's known issues section.

P.S.  I'm going to re-emerge baselayout just to be safe.

P.P.S.  I just found that I had to re-create the /var/log/portage directory too.  I guess I need to add it to the custom includes section of the script.

----------

## booyaa

OK Guys question:

I got this working cool and I managed to restore a stage 4 with some kernel and grub tweaks..

Now..

Do you guys think this will work if I want to restore to a whole new hardware platform:

restore stage 4/recompile kernel/reinstall and configure grub?

just curious..

If not.. then what is the easiest way to do this?

I basically am running php-syslog-ng on a gentoo box and want to migrate the solution to another box:  more resources

----------

## BlinkEye

depending on your CFLAG setting and your CPU change this will work with no problem at all or need (first) a complete re-compile of the system (emerge -e world would be your friend): wiki link

----------

## simeli

Has anyone thought of a way to back up using ftp, scp or sftp. I'd like to backup my home server to my other box, but don't have the space needed to store the tarball locally befor transferring it. so being able to pipe the output over directly to the other box would be awesome. Any ideas how this could be done?

----------

## GetCool

Is anyone else having trouble getting this script to work properly these days?

I've used mkstage4.sh for quite a while now, and it's always worked great, but recently, I'm noticing that it's not backing up some pertinent files.  Specifically, it skips over the entire /usr directory, as well as certain subdirectories of my /home/user (which are seemingly chosen randomly).

Note that this is happening on a freshly installed system with an unedited mkstage4.sh.  Regardless, I have read all the documentation and am quite familiar with how all the variables work in the script (I have customized it in the past in several different ways), so I'm not inclined to believe it's user error (although I won't rule out that possibility).

Since it worked for me for so long but just recently stopped, I am wondering if some recent package update is causing problems with this script, since the script hasn't been updated in a little while.  Since it uses standard linux tools, though, I can't understand why this might be so.  I've been through the script many times, trying to logically deduce why it isn't backing up /usr and other dirs, but I can't figure it out.

As I said, this even happens on a freshly installed system with a fresh copy of mkstage4.sh (on amd64, at least).  I can post more details and/or due some more testing, if anyone has any insight.

----------

## castra

I have issue with backup ( script ) on my PIII machine.

I runed the same script on my main dekstop ( AMD64 ) and there was no problems what so ever so I am preaty sure that

nothing is wrong with it ( added some excluded stuff ).

I get  following error: *Quote:*   

> 
> 
>  7712 Broken pipe             | /bin/tar --bzip2 --preserve-permissions --create --absolute-names --totals --ignore-failed-read --file /disk3/mahdi-stage4-2006.06.13-custom.tar.bz2 --no-recursion -T -
> 
> * stage4 is done
> ...

 

As first strage thing that happends on this PC is that backup take so long and second it uses all free space on /disk3 ( 5.9 GB ).

So my Q is how can this be that whole system ( which is 2 GB ) takes more then 6 GB of backup space ?

----------

## piwacet

Hi.  I'm thinking of making a stage4 based on the wiki, and had a question about the exclude files:

 *Quote:*   

> stage4.excl contents:
> 
> .bash_history
> 
> /dev/*
> ...

 

I understand excluding everything except /sys/*.  I guess I just don't know what stuff in /sys/ does, 

```
magenta@Nautilus ~/movies $ ls -l /sys

total 0

drwxr-xr-x  8 root root 0 Jun 18 11:10 block

drwxr-xr-x 12 root root 0 Jun 18 11:11 bus

drwxr-xr-x 23 root root 0 Jun 18 11:10 class

drwxr-xr-x  6 root root 0 Jun 18 11:10 devices

drwxr-xr-x  3 root root 0 Jun 18 11:10 firmware

drwxr-xr-x  2 root root 0 Jun 18 11:10 fs

drwxr-xr-x  2 root root 0 Jun 18 11:10 kernel

drwxr-xr-x 43 root root 0 Jun 18 11:11 module

drwxr-xr-x  2 root root 0 Jun 18 11:10 power

magenta@Nautilus ~/movies $ 
```

Is that stuff automatically created at boot and therefore not necessary to back up?

TIA!

----------

## GetCool

 *piwacet wrote:*   

> I understand excluding everything except /sys/*.  I guess I just don't know what stuff in /sys/ does

 

/sys is a pseudo filesystem, just like /proc.  As such, it should not be backed up.

Interestingly, it is not listed in the FHS.  My understanding is that it is fairly new (as of 2.6, possibly).

----------

## BlinkEye

 *castra wrote:*   

> I have issue with backup ( script ) on my PIII machine.
> 
> I runed the same script on my main dekstop ( AMD64 ) and there was no problems what so ever so I am preaty sure that
> 
> nothing is wrong with it ( added some excluded stuff ).
> ...

 

Hmm, this broken pipe issue has been seen before. Could you re-download the script and try with a unmodified version? It usually happens while cutting & pasting the code (please use wget to get the script).

----------

## fkamp

Hi,

Thanks for a great script. This sure beats ghosting!

I have a question. Why not backup all of /usr/src/linux by default? 

Is there a reason to only backup .config?

Seems like it would be a good thing to have a backup of your source tree.

Anyway, thanks again!

fkamp

----------

## GetCool

 *fkamp wrote:*   

> Why not backup all of /usr/src/linux by default?

 

Because your kernel sources are not required for a working system.

After you build your system, you can actually unmerge and delete your kernel sources if you wish.  They might be needed at some point when you install a new package, but if you don't plan on installing any kernel-dependent packages for a while, then the sources don't have to sit on your hard drive.

Since they can occupy ~500 MB on an ext2/3 filesystem, I can see why someone would opt to leave them out of a stage4 backup.

----------

## fkamp

GetCool,

 *Quote:*   

> Because your kernel sources are not required for a working system. 

 

That is a very valid point, thanks for explaining it!

fkamp

----------

## kg4ysy

Awesome script...I have been looking for something like this.  I love it.  It could come in handy when I don't want to spend a week reinstalling Gentoo...

thx...

----------

## gnychis

its a great script, i have it in my crontab to backup my computer nightly to an external drive, great for not having to worry about my raid-0 bombing

----------

## dem1an

I tried restoring my stage4 today and go the following error. =\

tar: /mnt/gentoo: not found in archive

tar: Error exit delayed from previous erros

----------

## neophyte46

Just wanted to say thanks for an awesome script.

My installation was originally on an ide drive. I unpacked it onto an sata raid array and it worked nicely.

I've got this backing up my raid 5 server to an external drive as well.

I just tried it on my desktop and it worked flawlessly except my gnome logs half way in and jusst sits there before it finishes. I haven't been able to solve this issue so i'm going to try and re-emerge gnome to see if it can fix whatever the problem is.

----------

## DEP3

thanks alot. very useful...

----------

## pacho2

It doesn't work properly for me  :Sad: , it only creates a tiny 50 mb file.

I have downloaded the script from ftp://blinkeye.ch/gentoo/mkstage4.sh

This is the script I am using:

```

#!/bin/bash

# Backup script for Gentoo Linux

# mkstage4s.h is free software; you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation; either version 2 of the License, or

# (at your option) any later version.

#

# mkstage4.sh is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# Copyright: Reto Glauser aka blinkeye

# Mailto: stage4 at blinkeye dot ch

# Homepage: http://blinkeye.ch

# Forum post: http://forums.gentoo.org/viewtopic-t-312817.html

# Date: 2005-11-14

version=v3.6

basename=`basename $0`

find=/usr/bin/find

tar=/bin/tar

# these are the commands we actually need for the backup

command_list=(cut date echo $find grep hostname mount sh split $tar umount uname which)

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.

for command in ${command_list[@]}; do

        if [ ! -x "`which $command 2>&1`" ]; then

                echo -e "\nERROR: $command not found! "

                base=`basename $command`

                if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then

                        echo -e "ERROR: $base is not in your \$PATH."

                fi

                exit -1

        fi

done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# Defaults to creating one tarball

tar_output="--file"

# split command

split_options="--suffix-length=1 --bytes=700m"

# options for the tar command

tarOptions=" --preserve-permissions --create --absolute-names --totals --ignore-failed-read"

# where to put the stage4

stage4Location=/mnt/backups/stage4

# name prefix

stage4prefix=`hostname`-stage4-`date +\%Y.\%m.\%d`

# patterns which should not be backed up (like iso files).

# example: default_exclude_pattern="*.iso *.divx"

# These pattern count only for files NOT listed in the $custom_include_list.

default_exclude_pattern=""

# these files/directories are always excluded. don't add trailing slashes.

# don't touch it unless you know what you are doing!

# /var/db and /var/cache/edb are intentionally added here. they are listed

# in $default_include_folders

default_exclude_list="

/dev

/lost+found

/mnt

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log

/var/tmp

/var/db

/mastercd

/var/cache/edb

$stage4Location

`echo $CCACHE_DIR`"

# files/devices/folders, which need to be backed up (preserve folder structure).

# don't touch it unless you know what you are doing! no recursive backup of folders.

# use $default_include_folders instead.

default_include_files="

/dev/null

/dev/console

/home

/mnt

`find /mnt -name .keep`

/proc

/sys

/tmp

/usr/portage

/usr/src

/var/log/emerge.log

/usr/src/linux-`uname -r`/.config"

# folders, which need to be backed up recursively on every backup.

# don't touch it unless you know what you are doing! the reason for this

# variable is that some users add /var to the $default_exclude_list. here

# we ensure that portage's memory is backed up in any case.

default_include_folders="

/var/db"

# IMPORTANT: A minimal backup will EXCLUDE files/folders listed here. A custom backup will

# include/exclude these files/folders depening on your answer.

custom_include_list="

/home/*

/usr/src/linux-`uname -r`"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT

# be backed up. eg.

#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"

custom_exclude_list=""

# Only files/folders within the $custom_include_list are checked against these patterns

# custom_exclude_pattern="*.mp3 *.iso"

custom_exclude_pattern=""

# the find_command

find_command="$find /*"

# don't backup anything which matches pattern listed in $default_exclude_pattern

for pattern in $default_exclude_pattern; do

        find_command="$find_command -not -name $pattern"

done

# assemble the find_command

function find_files()

{

        for folder in $default_exclude_list; do

                find_command="$find_command -path $folder -prune -o"

        done

        find_command="$find_command -print"

        for i in $default_include_files; do

                find_command="echo $i; $find_command"

        done

        for i in $default_include_folders; do

                if [ -d $i ]; then

                        find_command="$find $i; $find_command"

                else

                        find_command="echo $i; $find_command"

                fi

        done

}

# check the exclude/include variables for non-existing entries

function verify()

{

        for i in $1; do

                if [ ! -e "`echo "$i" | cut -d'=' -f2 | cut -d'*' -f1`" -a "$i" != "/lost+found" -a "$i" != "$stage4Location" ]; then

                        echo "ERROR: `echo "$i" | cut -d'=' -f2` not found! Check your "$2

                        exit 0

                fi

        done

}

# check input parameters

while [ $1 ]; do

        case  $1 in

        "-h" | "--help")

                echo -e $help

                exit 0;;

        "-v" | "--verbose")

                verbose=$1;;

        "-s" | "--split")

                tar_output="--split";;

        "");;

        *)

                echo -e $help

                exit 0;;

        esac

        shift

done

echo ""

# check folder/files listed in $default_exclude_list exist

verify "$default_exclude_list" "\$default_exclude_list"

# check files listed in $default_include_files exist

verify "$default_include_files" "\$default_include_files"

# check folder listed in $default_include_folders exist

verify "$default_include_folders" "\$default_include_folders"

#check folder listed in $custom_include_list exist

verify "$custom_include_list" "\$custom_include_list"

#check folder listed in $custom_exclude_list exist

verify "$custom_exclude_list" "\$custom_exclude_list"

# print out the version

 echo -e "\nBackup script $version"

 echo -e "=================="

# how do you want to backup?

echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n

Fast (tar.gz):

 (1) Minimal backup

 (2) Interactive backup

Best (tar.bz2):

 (3) Minimal backup

 (4) Interactive backup\n"

while [ "$option" != '1' -a "$option" != '2' -a "$option" != '3' -a "$option" != '4' ]; do

        echo -en "Please enter your option: "

        read option

done

case $option in

[1,3])

        stage4Name=$stage4Location/$stage4prefix-minimal.tar;;

[2,4])

        stage4Name=$stage4Location/$stage4prefix-custom.tar

        for folder in $custom_include_list; do

                echo -en "\nDo you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "

                read answer

                while [ "$answer" != 'y' -a "$answer" != 'n' ]; do

                        echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "

                        read answer

                done

                if [ "$answer" == 'n' ]; then

                        find_command="$find_command -path $folder -prune -o"

                else

                        custom_find="$find $folder"

                        for i in $custom_exclude_pattern; do

                                custom_find="$custom_find -name $i -o"

                        done

                        for i in $custom_exclude_list; do

                                custom_find="$custom_find -path $i -prune -o"

                        done

                        find_command="$custom_find -print; $find_command"

                fi

        done ;;

esac

# add $custom_include_list to the $default_exclude_list as we assembled

# $custom_find with $custom_include_list already.

default_exclude_list="$default_exclude_list $custom_include_list"

case $option in

[1,2])

        stage4postfix="gz"

        zip="--gzip";;

[3,4])

        stage4postfix="bz2"

        zip="--bzip2";;

esac

# mount boot

echo -e "\n* mounting boot"

mount /boot >/dev/null 2>&1

# find the files/folder to backup

find_files

find_command="($find_command)"

# create the final command

if [ "$tar_output" == "--file" ]; then

        tar_command="$find_command | $tar $zip $tarOptions $verbose --file $stage4Name.$stage4postfix --no-recursion -T -"

else

        tar_command="$find_command | $tar $zip $tarOptions $verbose --no-recursion -T - | split $split_options - "$stage4Name.$stage4postfix"_"

fi

if [ "$verbose" ]; then

        echo -e "\n* creating the stage4 in $stage4Location with the following command:\n\n"$tar_command

fi

# everything is set, are you sure to continue?

echo -ne "\nDo you want to continue? (y/n) "

read answer

while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

        echo -ne "Do you want to continue? (y/n) "

        read answer

done

if [ "$answer" == 'y' ]; then

        # check whether the file already exists.

        if [ "$tar_output" == "--split" ]; then

                overwrite="`ls "$stage4Name.$stage4postfix"_* 2>&1 | grep -v 'No such file'`"

        else

                overwrite="$stage4Name.$stage4postfix"

        fi

        if [ -a "`echo "$overwrite" | grep "$overwrite" -m1`" ]; then

                echo -en "\nDo you want to overwrite $overwrite? (y/n) "

                read answer

                while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

                        echo -en "Do you want to overwrite $overwrite? (y/n) "

                        read answer

                done

                if [ "$answer" == 'n' ]; then

                        echo -e "\n* There's nothing to do ... Exiting"

                        exit 0;

                fi

        fi

        # if necessary, create the stage4Location

        if [ ! -d "$stage4Location" ] ; then

                echo "* creating directory $stage4Location"

                mkdir -p $stage4Location

        fi

        echo -e "\n* Please wait while the stage4 is being created.\n"

        # do the backup.

        sh -c "$tar_command"

        # finished, clean up

        echo -e "\n* stage4 is done"

        echo "* umounting boot"

        umount /boot >/dev/null 2>&1

        # Integrity check

        echo -e "* Checking integrity"

        if [ "$zip" == "--gzip" ]; then

                zip="gzip"

        else

                zip="bzip2"

        fi

        if [ "$tar_output" == "--split" ]; then

                if [ "`cat "$stage4Name.$stage4postfix"_*"" | $zip --test 2>&1`" != "" ]; then

                        echo -e "* Integrity check failed. Re-run the script and check your hardware."

                        exit -1

                fi

        else

                if [ "`$zip --test  $stage4Name.$stage4postfix 2>&1`" != "" ]; then

                        echo -e "* Integrity check failed. Re-run the script and check your hardware."

                        exit -1

                fi

        fi

        # everything went smoothly"

        echo -e "* Everything went smoothly. You successfully created a stage4."

else

        echo -e "\n* There's nothing to do ... Exiting"

fi

```

It runs the following command:

```

* creating the stage4 in /mnt/backups/stage4 with the following command:

(/usr/bin/find /var/db; echo /usr/src/linux-2.6.18-gentoo-r3/.config; echo /var/log/emerge.log; echo /usr/src; echo /usr/portage; echo /tmp; echo /sys; echo /proc; echo /mnt/.keep; echo /mnt; echo /home; echo /dev/console; echo /dev/null; /usr/bin/find /bin /boot /cdrom /cdrom2 /celestia /dev /emul /etc /floppy /home /lib /lib32 /lib64 /lost+found /mandriva /mastercd /media /mnt /opt /proc /root /sbin /sys /tmp /usr /var /windows -path /home/marga -prune -o -path /home/Mi disco -prune -o -path /home/osorio -prune -o -path /home/osorio-mdv -prune -o -path /home/osorio-suse -prune -o -path /home/osorio-ubu -prune -o -path /home/pacho -prune -o -path /home/pacho-mdv -prune -o -path /home/pacho-suse -prune -o -path /home/pacho-ubu -prune -o -path /home/pati -prune -o -path /home/peter -prune -o -path /home/pipo -prune -o -path /home/rey -prune -o -path /home/test -prune -o -path /usr/src/linux-2.6.18-gentoo-r3 -prune -o -path /dev -prune -o -path /lost+found -prune -o -path /mnt -prune -o -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o -path /usr/portage -prune -o -path /usr/src -prune -o -path /var/log -prune -o -path /var/tmp -prune -o -path /var/db -prune -o -path /mastercd -prune -o -path /var/cache/edb -prune -o -path /mnt/backups/stage4 -prune -o -path /var/tmp/ccache -prune -o -path /home/marga -prune -o -path /home/Mi disco -prune -o -path /home/osorio -prune -o -path /home/osorio-mdv -prune -o -path /home/osorio-suse -prune -o -path /home/osorio-ubu -prune -o -path /home/pacho -prune -o -path /home/pacho-mdv -prune -o -path /home/pacho-suse -prune -o -path /home/pacho-ubu -prune -o -path /home/pati -prune -o -path /home/peter -prune -o -path /home/pipo -prune -o -path /home/rey -prune -o -path /home/test -prune -o -path /usr/src/linux-2.6.18-gentoo-r3 -prune -o -print) | /bin/tar --gzip --preserve-permissions --create --absolute-names --totals --ignore-failed-read -v --no-recursion -T - | split --suffix-length=1 --bytes=700m - /mnt/backups/stage4/belkin2-stage4-2006.11.28-custom.tar.gz_

Do you want to continue? (y/n)

```

This is my df output:

```

/dev/hda1             58635388  31885940  26749448  55% /

udev                   1029976       244   1029732   1% /dev

/dev/hda5             20008280  13171428   6836852  66% /home

shm                    1029976         0   1029976   0% /dev/shm

```

Thanks a lot

----------

## pacho2

Nothing?   :Sad: 

----------

## pacho2

This is the end of the tarball created:

```
........

     30 -rw-r--r--  root/root Sep  5 12:51 2006 /var/db/pkg/games-rpg/bass-1.2/RDEPEND

     20 -rw-r--r--  root/root Sep  5 12:51 2006 /var/db/pkg/games-rpg/bass-1.2/CBUILD

     27 -rw-r--r--  root/root Sep  5 12:51 2006 /var/db/pkg/games-rpg/bass-1.2/CFLAGS

    800 -rw-r--r--  root/root Sep  5 12:51 2006 /var/db/pkg/games-rpg/bass-1.2/bass-1.2.ebuild

     45 -rw-r--r--  root/root Sep  5 12:51 2006 /var/db/pkg/games-rpg/bass-1.2/DEPEND

      0 drwxr-xr-x  root/root Nov 28 10:31 2006 /var/db/webapps/

      0 drwxr-xr-x  root/root Oct 20 13:12 2006 /var/db/webapps/mediawiki/

      0 drwxr-xr-x  root/root Oct 20 13:09 2006 /var/db/webapps/mediawiki/1.6.8/

      1 -rw-------  root/root Oct 20 13:09 2006 /var/db/webapps/mediawiki/1.6.8/installs

      0 drwxr-xr-x  root/root Oct 20 13:41 2006 /var/db/webapps/mediawiki/1.7.1/

      0 -rw-r--r--  root/root Oct 20 13:41 2006 /var/db/webapps/mediawiki/1.7.1/.keep_www-apps_mediawiki-1.7.1

     52 -rw-------  root/root Oct 20 13:41 2006 /var/db/webapps/mediawiki/1.7.1/installs

      0 -rw-r--r--  root/root Nov 28 10:31 2006 /var/db/webapps/.keep_app-admin_webapp-config-0

  39579 -rw-r--r--  root/root Nov 28 11:06 2006 /usr/src/linux-2.6.18-gentoo-r3/.config

5888766 -rw-rw----  portage/portage Nov 29 22:00 2006 /var/log/emerge.log

      0 drwxr-xr-x  root/root Nov 28 12:52 2006 /usr/src/

      0 drwxr-xr-x  portage/portage Nov 28 11:54 2006 /usr/portage/

      0 drwxrwxrwt  root/root Nov 29 22:04 2006 /tmp/

      0 drwxr-xr-x  root/root Nov 29 22:12 2006 /sys/

      0 dr-xr-xr-x  root/root Nov 29 22:12 2006 /proc/

      0 -rw-r--r--  root/root Mar  4 16:28 2005 /mnt/.keep

      0 drwxr-xr-x  root/root Jul 16 19:04 2006 /mnt/

      0 drwxr-xr-x  root/root Nov 21 20:05 2006 /home/

  5   1 crw-------  root/tty Nov 29 21:13 2006 /dev/console

  1   3 crwxrwxrwx  root/root Oct 22 01:01 2000 /dev/null 
```

The same script works fine on my notebook, but not in my desktop computer (it worked 15 days ago, but now it doesn't work) 

Thanks

----------

## sfragis

Hi BlinkEye, thanks for the script.

Starting from an idea of a friend of mine, time ago I did a small script, not much different from your.

For those who are interested, you can download it from here.

The goal of my script was to provide a certain safety against data corruption.

While the base logic is quite the same (use find to create the list of files, use exclusion patterns), the tools I used change:

 afio instead of tar because it's more suitable for data recovering (given two offset you can always extract the relative content from the archive).

 par2 (app-arch/par2cmdline in portage) which creates parity files and can be useful to recover damaged data (according with the chosen protection level).

 md5sum to checksum the archives and detect data corruption or loss.

Feel free to use part or all of the code if it can be useful to improve your script.

----------

## pacho2

The problem persists, script stops without make a properly backup  :Sad: 

----------

## hirakendu

frankly, i dont understand why so much effort. i do understand that the intricacy is only because it is being done from inside the working gentoo? what i do is boot using a live cd and simply do tarring along with gzip/bzip2/lzma on the root partition. and a simple extraction on a freshly formatted drive is enough for restoring.

so, for creating image :

$ cd /gentoo/   ( /gentoo/ being the mounted root partition containing bin, boot, usr, etc after booting with a livecd.)

$ tar -cvps --atime-preserve * | lzma e -si -so > /data/gentoo-img.tar.lzma (for a lzma archive)

(or)

$ tar -cvzps --atime-preserve -f /data/gentoo-img.tar.gz * (for gzip archive)

and for restoring from image :

$ mkreiserfs /dev/sdaX

$ mount /dev/sdaX /gentoo

$ cat /data/gentoo-img.tar.lzma | lzma d -si -so | tar -xps --atime-preserve -C /gentoo/ ( from lzma archive)

(or)

$ tar -xvzps --atime-preserve -f /data/gentoo-img.tar.gz -C /gentoo/ (from gzip archive)

and after that, the usual (re)installation of grub and editing some configs (fstab, grub.conf, net , hostname, xorg.conf) to one's liking, just as is done in case of stage3.

btw, i created an archive of a system buit with cflags "o2, prescott" for cpu's with sse3 +. and the image works fine on all machines, inlcuding athlon 64x2's. (yeah all those performance loss fundaes dont work - its 'the' same - atleast with tests like openssl and superpi. ~infinite uptimes, no crashes, stable , fast and responsive.) 

my system is about 5.0gb and it takes 15 minutes with gzip and 1hour,30mins with lzma to compress. while decompressing, gzip takes 15 minutes while lzma takes 7 minutes (flat!). btw, gzip compresses it to about 1.6 gb while lzma does it to about 1.1 gb. (this is on a athlon 64x2 3800+ with sata drive @ 67 MB/s (hdparm -tT) and a 16x dvd drive.)

hirakendu.

----------

## vicaya

 *hirakendu wrote:*   

> frankly, i dont understand why so much effort. i do understand that the intricacy is only because it is being done from inside the working gentoo? what i do is boot using a live cd and simply do tarring along with gzip/bzip2/lzma on the root partition. and a simple extraction on a freshly formatted drive is enough for restoring.

 

frankly, i don't understand why do you need to post this, as you already understand the overwhelming advantage of the script: you can backup your system without downtime, and regardless of your partition scheme! I have all my system on lvm2. Thanks to his script, I can do weekly system backup (and daily incremental rsync of /home) without downtime.

 *Quote:*   

> btw, i created an archive of a system buit with cflags "o2, prescott" for cpu's with sse3 +. and the image works fine on all machines, inlcuding athlon 64x2's. (yeah all those performance loss fundaes dont work - its 'the' same - atleast with tests like openssl and superpi. ~infinite uptimes, no crashes, stable , fast and responsive.)

 

Huh, how can you have ~infinite uptime, when you have to do the backup in livecd all the time?

----------

## hirakendu

k, now i get it - that its if you have whatever funky setups. thanks.

just wanted to reflect on the stability. wonder why you take it literally  :Wink: .

----------

## eduardo451

Does it have to be the LiveCD, or will it work with the minimal install CD?  I don't have enough RAM to docache the LiveCD.

----------

## d2_racing

 *eduardo451 wrote:*   

> Does it have to be the LiveCD, or will it work with the minimal install CD?  I don't have enough RAM to docache the LiveCD.

 

The minimal CD will be just fine  :Smile: 

----------

## eduardo451

Thanks!  Just completed using the script to repartition my drive to get rid of Windows (don't need at all now) and give Gentoo the full drive.  Am typing this from my restored system.  Thanks to the script I was able to easily accomplish this in one evening and preserve my system of two years.  Great work!

----------

## jsporring

Dear all,

This is a nice script.  I used it to install 2 out of 3 identical systems.  However, I met the following problem: "missing eth0" after having copied a fresh install of machine 1 onto machine 2 and 3.  It appears that udev gave a conflict, in any case removing /etc/udev/rules.d/70-persistent-net.rules solved the problem.  Maybe a check should be added to the script.  I would also recommend adding /opt to the list of directories to be backed up.

Best, Jon

----------

## d2_racing

If you want to tansfert a HDD to an another. maybe you should use the Stage 5.

----------

## fidel

GREAT!!! I've been looking for ages for a solution to get a consistent backup of a running system! I was just way to scared to screw things up (forget to exclude /proc and you're really screwed, what about /dev and needed nodes in order to boot and so on...).

--> THANKS for that great script! I will definitely use it!

One little feature request: In times of low prices of storage, I would love to see an option to backup to a none compressed archive, just tar, even without gz compression. It would speed up the backup even more and save cpu usage as well.

Another point: This script rocks that hard, that I would like to see it in portage!.. 

```
#emerge mkstage4
```

 :Very Happy: 

----------

## koo

that is useful

----------

## shazam75

 *pacho2 wrote:*   

> The problem persists, script stops without make a properly backup 

 

What is the actual error message if there is any ?

----------

## ozman

Hi,

Before emerging world mkstage.sh script worked fine, but now i have error and created backup file has only 73MB.

```

* Please wait while the stage4 is being created.

/usr/bin/find: ścieżki muszą poprzedzać wyrażenie: of

Składnia: /usr/bin/find [-H] [-L] [-P] [-Opoziom] [-D help|tree|search|stat|rates|opt|exec] [ścieżka...] [wyrażenie]

/bin/tar: /var/log/emerge.log: Uwaga: Nie można open: Brak dostępu

Licza zapisanych bajtów: 73041920 (70MiB, 2,4MiB/s)

* stage4 is done

* umounting boot

* Checking integrity

* Everything went smoothly. You successfully created a stage4.

```

----------

## Phlogiston

 *sfragis wrote:*   

> Hi BlinkEye, thanks for the script.
> 
> Starting from an idea of a friend of mine, time ago I did a small script, not much different from your.
> 
> For those who are interested, you can download it from here.
> ...

 

I think the link is broken. Anyone has that script or could the author please upgrade it?

----------

## slycordinator

 *Phlogiston wrote:*   

>  *sfragis wrote:*   Hi BlinkEye, thanks for the script.
> 
> Starting from an idea of a friend of mine, time ago I did a small script, not much different from your.
> 
> For those who are interested, you can download it from here.
> ...

 

Even if it's a bit late getting to you. The author compressed the file now by default

http://fstrozzi.web.cs.unibo.it/safebackup.sh.bz2

----------

## Phlogiston

 *slycordinator wrote:*   

> 
> 
> Even if it's a bit late getting to you. The author compressed the file now by default
> 
> http://fstrozzi.web.cs.unibo.it/safebackup.sh.bz2

 

Thanks a lot. I will look into that.

----------

## diloo

Hi,

I would like to re-install Gentoo, but if I'm not satisfied with my new configuration I want to be able to restore my old system.

So, I want to create my own stage3, e.g. an archive which I'll be able to untar on my / to get my old system back.

I think just one tar command will do it but I don't know which options I must use to do it properly.

Thanks.

----------

## yabbadabbadont

What you are looking for is usually called a "stage4" backup or install.  There are various howto's on the topic around here somewhere.  The method that I have used successfully in the past is here: Stage4 Howto

----------

## nixnut

merged above two posts here.

----------

## NotQuiteSane

is it possible to get an option to use lzma to compress with added to the script?

Thanks

NQS

----------

## xnmrphr

Great tool/script  :Smile:  Thanx.

----------

## El_Goretto

It was the second time I used this script to restore or migrate OS from a machine to another after yeeeaaaars of use for backup.

No problem at all once again.

Still years of use to come  :Smile: 

And thanks to its author.

----------

## Gibbo_07

Another thanks to an old but still worthy of praise topic. 

Have tuned the script to my needs and works a damn treat has even saved my gentoo install already. I have a question/bug report however if the author/better gifted scripter than myself can work out why when I had set

```

# patterns which should not be backed up (like iso files).

# example: default_exclude_pattern="*.iso *.divx"

# These pattern count only for files NOT listed in the $custom_include_list.

default_exclude_pattern="*.iso *.pk4"

```

I still have .pk4 files being backed up in the archive? They are located in /opt namely the quake4 data files which I don't need to have backed up, I would however like the config files and binary backed up so this is why I am trying to get this function to work rather than adding the dir to the exclude list.

+1 to anyone who can figure this out  :Smile: 

----------

## Gibbo_07

I worked around the issue - added /opt/quake4 to the custom include list, and set the custom exclude pattern to match *.pk4. This works, my thinking is the global exclude is broken?

Thanks.

----------

## rb34

Hi all!

I found that the line 

```
`find /mnt -mount -name .keep`
```

should be 

```
`find /mnt -mount -name .keep`
```

because if you don't add -mount, find descends the whole directory structure of any mounted partition!! 

That "bug" leads to two problems:

* if you, as me, have a partition with many dirs, the old find command takes a lot of time!

* and, if you have a backup made with rsnapshot (with many .keep files) on a mounted partition, the final command sent to "sh -c" can exceed the argument size limit!

Adding "-mount" makes find remain on the same partition, and this is fine, as we need only to find the .keep under /mnt and not on the mounted partitions

----------

## Gibbo_07

 *rb34 wrote:*   

> Hi all!
> 
> I found that the line 
> 
> ```
> ...

 

Both lines you posted are identical   :Embarassed: 

Also I canno't find a line matching that?

edit: my bad I have v3.5 - should have gotten it from authors webpage first time around  :Smile: 

----------

## rb34

I'm sorry  :Smile:  : the line 

```
`find /mnt -name .keep`
```

should be replaced with

```
`find /mnt -mount -name .keep`
```

----------

## johabba

 *Gibbo_07 wrote:*   

> 
> 
> edit: my bad I have v3.5 - should have gotten it from authors webpage first time around 

 

How exactly do you get this script? I tried to copy/paste from the author website but it does not copy the indents cleanly. I also tried "wget 'http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4'" hoping the script was wrapped in "pre" html flags and I could delete the rest of the html. 

Maybe the author can use the "pre" html flag around the script? The "pre" html flag is used around the section starting with "Backup script v3.5". This section works with wget'ing and removing the surrounding html.

----------

## Killerchronic

 *johabba wrote:*   

>  I tried to copy/paste from the author website but it does not copy the indents cleanly.

 

Not sure what your doing diff or if this has changed but a simple copy and paste worked fine for me, indents and all.

Thanks for this, been hunting for a backup solution, hopefully this will fill my need, was considering bacula but for 2 computers that are now fully setup i think its a bit much.

----------

## kwispy

I went searching for the maintained mkstage4.sh on Google, as the original download link is broken.  I found it on github, which seems to be the most up-to-date version, as it has been re-written a few times.  It seems to be in the chymeric overlay.

FYI for anyone who is following in my frustrated tracks, and for continuity's sake.

----------

## Astronome

 *kwispy wrote:*   

> I went searching for the maintained mkstage4.sh on Google, as the original download link is broken.  I found it on github, which seems to be the most up-to-date version, as it has been re-written a few times.  It seems to be in the chymeric overlay.
> 
> FYI for anyone who is following in my frustrated tracks, and for continuity's sake.

 

Thanks! Works great and runs much faster than it did in 2005  :Very Happy: 

----------

## pa1983

I just noticed that tar 1.29 seems bugged, found a bug report with the same problem as I have.

Basically -X or the exclude options dont seem to work so in my case it started to archive my external hard drive and well eventually would have tried to compress terabytes of data  of my filserver  :Razz: 

Anyway I was also annoyed by the time it takes to use tar. Single threaded so my 6 core CPU with 12 threads cant realy do much work.

I installed pbzip2 and used that as a compressor instead. This is how I compressed my gentoo install and it was MUCH faster, all 12 threads used to 100%.

tar -I pbzip2 -cvf /mnt/sdc1/voyager-20161028-stage4.tar.bz2 / -X stage4.excl

Anyway have not unpacked it since Im waiting for my new SSD but trying to harnes all the cores on a modern CPU shoukd be a goal and added to the wiki I think.

Anyway I got the info from here, testing it out.

http://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression

----------

## cv01302

It's true!

I emerged pbzip2 (I had already installed tar-1.29-r1), modified the initial script to use pbzip2 instead of bz2 , htop shows all cores being used at 100%. 

On i7-4771 , 13 GB of data (root partition only) were compressed to 3.5 GB (best option) in under 7.5 minutes. Unedited script did 16 mins so that's about double time (although I expected the difference to be even bigger).

Kudos to pa1983 for this nice tip!

----------

## davidbrooke

The pbzip2 tip is nice but why not use something that actually works? The --exclude problem is a major road block.

I found fsarchiver

http://www.fsarchiver.org/Main_Page

via

https://wiki.gentoo.org/wiki/Backup

Fsarchiver has pbzip2 with other compression options as well as core / thread setup.

I tried both Fsarchiver and Mkstage4 and Fsarchiver worked better for me.

----------

## cv01302

Well, with tar-1.29-r1 exclusions were kept so I guess no bug there as it was stated that existed in 1.29. My file was 3.5GB compressed, if it didn't include the exclusions, my user home partition would be there too, along with my other HDD partitions, file would be way, way larger than 3.5GB .

Just checked the archive too, indeed no file was included from the exclusion list, so I guess it is good to go. Real test would be to restore the image just to check everything works perfectly, this though at a later time.

I went ahead and tried the fsarchiver (with -A flag in order to simulate all conditions as before), used the exact same exclusions as with script, along with -j 8 (all cores) and -z 9 (best compression with lzma when z>7 . However no pbzip2 support as you stated, just lzo, gzip, bzip2 and lzma) results were 3.7GB data at 11 minutes. Changing it to -z 7 (lzma still but without the best compression), results were 3.9GB and an astonishing 3 minutes, so I guess it is a winner. Decompression will be way faster too due to lzma in comparison with pbzip2.

Fsarchiver looks indeed very promising, liked the checksumming of the data, ability to restore corrupt archives, as well as the ability to encrypt data with a password.

Nice tip David, thanks! Fsarchiver is the winner for me

----------

