# make your own Stage4 mini HOWTO

## allucid

Custom Stage4 mini HOWTO:

NOTE:

check out the gentoo wiki for the most up to date version. I will not be maintaining this post anymore.

New Update: BlinkEye made a script that automates the process and greatly simplifies things!

old outdated stuff starts here:

I will go over the basics of creating a custom stage4 archive in this mini HOWTO. A stage4 is basically an image of your entire root partition. The primary reason I make a stage4 is to allow for a quick recovery if I have a disk failure. It is basically the same as a stage3 only you can select the CFLAGS you want to use and what other software you want installed. You can adapt this method to suit your personal uses.

If you want a more generic stage4 that you can install on multiple systems use genkernel  to set up your kernel so it works like the one on the livecd. You may also want to use less-restrictive CFLAGS (mcpu instead of march). You may still have to modify the fstab and USE flags after extracting the stage4 to suit the user.

Step1:

Install Gentoo.

Step2:

    Configure all drivers (sound, video, usb, etc) and install any software you want to be included on your stage4. For example, I would install X, Xfce4, Sun's JDK, CVS, Emacs, Thunderbird, and Firefox.

Step3:

    Make a copy of your /boot partition:

```
root# mount /boot

root# cp -R /boot /bootcpy

root# umount /boot

```

Alternatively, you can just mount /boot before you tar everything up. There may be problems with doing this when you use this stage4 with different hardware configurations.

Step4:

Clear out temporary files in /usr/portage/distfiles and /var/tmp to save space.

Step5:

    After everything is how you want it, we will create the archive:

```
root# tar cCjpf /path/to/save/at/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc

```

tar options we used:

c - create archive

j - use bzip2 compression

p - preserve file attributes (don't leave this out!!)

f - specify file name

Be sure to look at the tar manpage

It will probably take a long time to create the archive depending on how much you have installed. I usually save the archive on a spare disk I have in my system that I use for backups. You can also burn it to a cd or dvd. If it is too large to fit on a cd you will have to split it up. Large tar files can be split up with 'split'. The parts can be joined later with 'cat'. ('man split', and 'man cat' for more info).

To install from a stage4:

1. Boot live CD

2. Partition and mount filesystem

3. root# cd /mnt/gentoo

4. copy your stage4 archive(s) to disk

(if it is on another CD type "gentoo cdcache" at the boot prompt. Then you'll be able to umount/mount other CDs.)

5. tar xvjpf stage4.tar.bz2

6. root# cp -R bootcpy /mnt/gentoo/boot

   (double check the boot dir after you copy the files over!)

   root# rm -rf bootcpy

7. fstab

8. grub or lilo

That's it! I know it's not the most elegant way to do things but it should get the job done. If anyone would like to offer some input/advice I would be glad to update the HOWTO. Let me know if I made any mistakes or missed anything here...I'm just doing this from memory.  :Wink: 

Updated according to advice given by users below. Thanks for the input.Last edited by allucid on Sun Dec 05, 2004 7:14 am; edited 7 times in total

----------

## Greven

Awesome...... This is like a How-To back too. If you clear your distfiles the tar will be smaller.

----------

## allucid

 *Greven wrote:*   

> Awesome...... This is like a How-To back too. If you clear your distfiles the tar will be smaller.

 

good idea. updated.

----------

## StringCheesian

Deleting the GRP packages in /usr/portage/packages (if any were copied over during installation) can save a lot of space too.

----------

## Greven

I get the broken pipe error. Help??

----------

## Greven

I may of found the problem..... I was trying to dump to a FAT32 FS.  I will try the -L flag.

----------

## markedmann

Hey all:

When I try to do this, tar keeps complaining about changing filesizes and padding extra zeros.  It also exits due to previous errors after about 450 megs.  Is there anything I can do about this?

Thanks,

-Mark

----------

## icywolf

good howto thanks,

Which directory shouln'd be tar?

I know /proc /dev

Am I right? any other?

----------

## bmichaelsen

How about adding this HOWTO to:

http://gentoo-wiki.com

----------

## Jarrod

To avoide having to copy /boot back and forth, I just do it this way...

Backup...

1. Boot from Live CD

2. Mount my partitions

3. tar cCjpf /mnt/gentoo/ /mnt/gentoo/stage4_backup.tar.bz2 * --exclude=stage4_backup.tar.bz2

Restore...

1. Boot from Live CD

3. create & mount partitions

4. tar -xvjpf /mnt/cdrom/stage4_backup.tar.bz2 -C /mnt/gentoo

5. chroot and Run grub or lilo depending on what you use

I actually got this from another thread in these forums, but they did not include the "--exclude" part so it would create a backup as well as include the backup file in the backup and also give you tar errors... ROFL...  I just added the "--exclude" so it does not pick up the file you specify, so do not specify a name that would be used by gentoo!  :Smile: 

----------

## Cocker68

To make a big TAR-archive to fit onto Your CDs, try the split command:

```
# split -db 700m stage4.tar.bz2 stage4.tar.bz2.split
```

will create 700 MB-chunks, and

```
# cat stage4.tar.bz2.split* > stage4.tar.bz2
```

will put them back together.

- Cocker :wqLast edited by Cocker68 on Tue Oct 05, 2004 5:26 pm; edited 1 time in total

----------

## kamilian

I've been using a simple backup script, may be useful to tar the system for the Stage 4 tarball:

```
Cobra root # cat backup.sh

dirs="/mnt/* /proc /sys /tmp /usr/portage/distfiles /var/tmp/*"

tar $(for i in $dirs; do if [ -d $i ]; then echo -n " --exclude=$i/*"; fi; done) -cvjspf /tmp/system_backup-$(date +%Y-%m-%d).tar.bz2 /* > /tmp/backup.log 2> /tmp/backup_errors.log
```

Simply add directories you don't want in the tar archive (I normally add my home directory as well for a system backup) and away you go...

EDIT: Updated thanks to wallace1819!

----------

## nianderson

 *allucid wrote:*   

> Step5:
> 
>     After everything is how you want it, we will create the archive:
> 
> ```
> ...

 

I get tar erroring out with the above command 

currentyl trying it with 

```
tar cCPjpf / /path_to_tar * --exclude=tar_name --exclude=dev/* --exclude=proc/* --exclude=sys/*

```

anyone have some improvements? maybe a -W ?

----------

## fdavid

It's a really good howto, I use a very similar method to backup my /home onto cd-s.

Some remarks:

1. The trick with /boot is unneeded. Just mount it before you make the backup, and umount it afterwards.

2. The large tar files can be splitted with a small utility called split. The parts can be joined later with cat.

3. The exclude option of tar must be heavily utilized, because there are many directories, which must be excluded, when backing up your root. Exclude can also go for /usr/portage, if yo want a smaller archive. Why delete it?

The directories I exclude when backing up my root:

/mnt

/proc

/sys

/tmp

+ any directory, which is backed up separately.

4. Ok, and a small script, which I use for making backup of my /home.

(The /dev/hdX corresponds to the /mnt/tarbackup directory in /etc/fstab.)

```

#! /bin/bash

# Backup script for Gentoo Linux

# Author: fdavid

# Date: 2003.11.29.

# Making backup of home partition to cds

# options for the archive

tarOptions="--create --absolute-names -- preserve-permissions --gzip --file"

# name of the archive

archive=/mnt/tarbackup/$(date +%Y%m%d)home.tar.gz

# mount the backup partition

mount /dev/hdaX

sleep 5

# create the archive

tar ${tarOptions} ${archive} /home/;

echo archive is done

# split the archive to cd size (use: "cat ${archive}.* >> ${archive}" to join the parts)

split --bytes=700000000 ${archive} ${archive}.

echo splitting is done

# unmount

sleep 5

umount /dev/hdaX

```

----------

## CharlieS

if you exclude directories like /home /dev /proc /mnt and /sys , wouldnt you need to create them manually after you extract the backup file?  otherwise when u boot up the computer it wont have a place for what is needed in those...    

If i could program i would write out a script to do all of these things..    tar (with exclude functionality) then split..  then untar..  (and automake excluded directories as mentioned above)

Anyone up for the challenge?

----------

## Delirium

I'm using this method to install onto a toshiba laptop that doesn't have network support. Installing into vmware on my main computer (windows) then moving the install onto the laptop. It also speeds up the install time and doesn't stress out the laptop.

This is the perfect how to that I needed to finish the task, thanks all for the info.   :Very Happy: 

----------

## odessit

 *Delirium wrote:*   

> I'm using this method to install onto a toshiba laptop that doesn't have network support. Installing into vmware on my main computer (windows) then moving the install onto the laptop. It also speeds up the install time and doesn't stress out the laptop.
> 
> 

 

neat trick, going to try it for my lappy (limited HD space, OO needs 4Gig free just to build)

----------

## shanenin

 *Quote:*   

> To install from a stage4:
> 
> 1. Boot live CD
> 
> 2. Partition and mount filesystem
> ...

 

I am probably missing something obvious. If you are booting with the gentoo live cd, how can you take it out and insert another disk? Can the gentoo live cd be unmounted, then mount your cd that contains your system tar file?

----------

## Iroko

 *shanenin wrote:*   

> I am probably missing something obvious. If you are booting with the gentoo live cd, how can you take it out and insert another disk? Can the gentoo live cd be unmounted, then mount your cd that contains your system tar file?

 

When booting the LiveCD, type "gentoo cdcache" at the boot prompt.  Then you'll be able to umount/mount other CDs.

----------

## shanenin

cool. thank you

----------

## robfantini

regarding restoring /boot .

 in  my limited experience  this can cause problems.

 if the new/target computer has differences in  scsi adapters, partitions or filesystem types  then restoring /boot can cause problems. if the hardware is the same then there should not be a problem.

 i'd suggest doing the save ot boot to bootcpy as allucid suggests.

 then cp over  System.map and kernel [init image also if you usee one].

 let grub or lilo setup /boot/grub .   use your /bootcpy/grub/grub.conf .

----------

## AgentSmith

This is exactly what I needed to build an entire gentoo system on my athlon xp for my parent's pentium3 which is much slower and lacks broadband internet access. I made a base system, emerged xfree, kde (almost all kde packages), 2 extra games (digger and frozen bubble  :Smile: ), mozilla firefox and thunderbird and installed openoffice binaries. This finally gave a gzipped tar of about 498 MB. Still considerably less than what fits on a cd, so why warn to not install too much? I have yet to transfer the system to the pentium3, but I am confident it will work.

Making a .tar.gz2 I used this:

```
tar cjpf /stage4.pentium3.tar.bz2 --exclude=stage4.pentium3.tar.bz2 --exclude=/proc
```

Both excludes are needed to avoid errors, so adding these to the HOW-TO would be a good idea.

----------

## mahir

```

root@en3 / # tar cCjpf /home/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

root@en3 / #

```

this is what i get?

any ideas?

----------

## mahir

```

mahir@en3 home $ sudo tar cjpf /home/Stage4.tar.bz2 / --exclude=Stage4.tar.bz2 --

-exclude=/proc --exclude=/dev --exclude=/sys --exclude=/mnt/*

```

is this fine?

removing dev and sys shouldnt cause any problems should it?

and i have my ntfs drives in mnt/

so i dont want them.

sounds ok right?

----------

## househead

 *mahir wrote:*   

> 
> 
> ```
> 
> root@en3 / # tar cCjpf /home/stage4.tar.bz2 / --exclude=stage4.tar.bz2
> ...

 

I get the same, but I removed the 'C' switch, and added a 'v' so I can

see what its doing. It seems to run ok.

```

etc/conf.d/gpm

etc/conf.d/net

etc/conf.d/usb

etc/conf.d/xfs

etc/conf.d/.keep

etc/conf.d/local.stop

etc/conf.d/firebird

etc/conf.d/esound

etc/conf.d/hdparm

```

You get the picture...

Does anyone know of a script that will verify this backup?

----------

## robfantini

from:  man tar

       -W, --verify 

         attempt to verify the archive after writing it

 I've  used --verify , but  have not tried it on a  compressed archive. If you do try please post your result.

----------

## househead

The backup didn't complete correctly, I came back to

```

root/.gconf/desktop/

root/.gconf/desktop/gnome/

root/.gconf/desktop/gnome/url-handlers/

root/.gconf/desktop/gnome/url-handlers/http/

root/.gconf/desktop/gnome/url-handlers/http/%gconf.xml

root/.gconf/desktop/gnome/url-handlers/https/

root/.gconf/desktop/gnome/url-handlers/https/%gconf.xml

root/.gconf/desktop/gnome/url-handlers/%gconf.xml

root/.gconf/desktop/gnome/%gconf.xml

root/.gconf/desktop/%gconf.xml

root/.gnome/

root/dead.letter

root/.fonts.cache-1

mindi.err.31188.tgz

tar: Error exit delayed from previous errors

genbox mark #

```

Gonna run it again and log errors. Will try the verify too.Last edited by househead on Thu May 13, 2004 12:21 pm; edited 1 time in total

----------

## robfantini

could you post  the command you used?

' tar: Error exit delayed from previous errors' does not necessarily mean that the tar file is bad.  I been getting that message off and on for a long time. Search google to see possible causes. 

 Did you try the --verify or -W option?

 In my opinion , if the files verify ok then you can ignore the error.

----------

## househead

I ran the following (didnt try verify, but am verifying it now)

```

genbox root # tar cjpfv ./stage41.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=proc/* --exclude=dev/* --exclude=sys/* --exclude=/mnt/* 2>error.log
```

and then opened another aterm, su'd and ran

```

tail -f /home/root/error.log

```

There we're errors, but they were all files in the temp folder

```

tar: Removing leading `/' from member names

tar: /tmp/.esd/socket: socket ignored

tar: /tmp/jpsock.141.29462: socket ignored

tar: /tmp/.X11-unix/X0: socket ignored

tar: /tmp/.X11-unix/X1: socket ignored

tar: /tmp/ssh-cfOlUM6534/agent.6534: socket ignored

tar: /tmp/.gdm_socket: socket ignored

tar: /tmp/orbit-mark/linc-731d-0-dd68cd8458a2: socket ignored

tar: /tmp/orbit-mark/linc-7316-0-6c3fdd0955c15: socket ignored

tar: /tmp/orbit-mark/linc-2bc0-0-5655a835d8129: socket ignored

tar: /var/tmp/portage/mozilla-firefox-0.8-r2/work/mozilla/embedding/browser/activex/src/pluginhostctrl/StdAo|.h: Cannot stat: No such file or directory

tar: Error exit delayed from previous errors

```

I assume it's fine, am running the following command to check the backup against the file system. 

```

genbox root # tar -dv --directory=/ stage4.tar.bz2

```

I will be V V happy if this works. I'm gonna combine it with the live-cd and whack it on a DVD. I have an empty drive to test it on too:)

----------

## LeTene

Just used this stage4 guide in a slightly different way...

My system was getting clogged up with noob-mistakes so I wanted to start "clean". However, i didn't want to lose Internet access while rebuilding. So here's what i did.

Backed everything up to DVD-R  :Razz: 

On my running gentoo system, created a ~/gentoo directory, copying the stage1 tarball to there, untarring, mounting proc & chrooting into it. For the various little bits like mirrorselect, copying resolv.conf, etc I just edited them from another shell by delving into ~/gentoo/<whatever> and copying working values there. I was then able to bootstrap & emerge system there.

Then i emerged Xorg, video drivers, XFCE4, Firefox & Sylpheed and cleaned up the system of distfiles & tmp stuff. Tarred it all up, copied tarball to another partition.

Rebooted with LiveCD, re-formatted filesystem on original partition (just plain old ext2), untarred the stage4 to it, held my breath & rebooted...

...worked 100%!

A little tiding up by copying config files and importing things like email/bookmarks from the backup DVD rounded it off.

Just to complete the circle, I made another stage4 from the new setup for instant recovery. in fact, I'm now doing the stage4 once a week to keep an instant-restore disk image - and I'm slowly starting to symlink my config files, mail folders & other bits to another partition so that all my data will come back in the event of a restore being neccessary.

Bottom line - a very BIG THANK YOU to all contributors, and especially allucid for starting it all off!

----------

## mahir

maybe you could write/enter ur commands into a sh script for us?

the simlinking mail dirs/bookmarks i mean..

----------

## LeTene

 *mahir wrote:*   

> maybe you could write/enter ur commands into a sh script for us?
> 
> the simlinking mail dirs/bookmarks i mean..

 

Well, not everyone's setup is like mine but here's a few examples of what I'm doing to give the general idea, rather than every file. Bear in mind that this is purely for saving configs, others may not bother with the user-related stuff here as they will have /home on a separate partition anyway. Anyway, this is likely by no means the best layout to go for - so I'm open to suggestions:

Assumptions:

A separate, small partition, mounted on /mnt/syssetup to act as the "config & data repository"

Two directories in this partition, system for root-owned configs and letene for my own user configs

Symlinking some important system config files:

```
mv /etc/make.conf /mnt/syssetup && ln -s /mnt/syssetup/make.conf /etc/make.conf

mv /etc/rc.conf /mnt/syssetup && ln -s /mnt/syssetup/rc.conf /etc/rc.conf

mv /etc/portage/package.keywords /mnt/syssetup && ln -s /mnt/syssetup/package.keywords /etc/portage/package.keywords

mv /etc/portage/package.use /mnt/syssetup && ln -s /mnt/syssetup/package.use /etc/portage/package.use
```

Symlinking some user configs:

```
mv /home/letene/.bashrc /mnt/syssetup/letene && ln -s /mnt/syssetup/letene/.bashrc /home/letene/.bashrc

mv /home/letene/.ssh/known_hosts /mnt/syssetup/letene && ln -s /mnt/syssetup/letene/known_hosts /home/letene/.ssh/known_hosts
```

You get the idea...you could be tidier and have more sub-directories on the /mnt/syssetup partition, but I'm kinda liking having almost all my configs in one place - makes it easier to edit  :Razz: . Of course, all of this is no substitute for a proper backup but can be useful as you can back up this partition for an instant config-restore.

----------

## ruyn

I've been experimenting with stage4 restores and everytime I try it, the restored system is completely broken.  In the lastest attempt, gcc does not work at all.  I'm just curious if othes have experienced problems after the restore.

----------

## strestout1

this is a great How-to thanx for the info! after a debian install went wrong, i learned to keep /boot, /, and /home in separate partitions mainly because i have so much stuff in my /home directory. now, if my install messes up, not only will i be able to keep my personal stuff, but also back up my system! awesome...i will try this soon.

----------

## dmitrio

I have copied this HOWTO, with permission of allucid, to gentoo-wiki.com 

http://gentoo-wiki.com/HOWTO_Custom_Stage4

If you see anything that should be added or changed, feel free to do so. 

Thank you for a great HOWTO.

----------

## fdavid

 *dmitrio wrote:*   

> 
> 
> If you see anything that should be added or changed, feel free to do so.

 

I posted some essential points, with which some pitfalls could be avoided and the life is easier, but the original poster didn't maintain the howto and obviously you also neglected these remarks. It doesn't matter, but I won't do it twice, if it wasn't enough once.

----------

## dmitrio

 *fdavid wrote:*   

> I posted some essential points, with which some pitfalls could be avoided and the life is easier, but the original poster didn't maintain the howto and obviously you also neglected these remarks. It doesn't matter, but I won't do it twice, if it wasn't enough once.

 

I update the article, thank you for the note.

----------

## fdavid

 *dmitrio wrote:*   

> 
> 
> I update the article, thank you for the note.

 

Thank you, very much.

----------

## ruyn

I figured out my troubles.  My stage4 was missing a heap of header files.  I need to be more careful how I type the excludes   :Smile: 

----------

## allucid

 *dmitrio wrote:*   

> I have copied this HOWTO, with permission of allucid, to gentoo-wiki.com 
> 
> http://gentoo-wiki.com/HOWTO_Custom_Stage4
> 
> If you see anything that should be added or changed, feel free to do so. 
> ...

 

sorry guys, i haven't used this method in a really long time and haven't been keeping up with the thread. thanks to dmitrio for posting it on the wiki. now that it is on the wiki, hopefully it will be easy for the community to keep it up to date as necessary.

----------

## dmitrio

always welcome   :Smile: 

----------

## T-Bird

 *mahir wrote:*   

> 
> 
> ```
> 
> root@en3 / # tar cCjpf /home/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc
> ...

 

Did anyone solve this situation...I'm getting the same deal.

----------

## nekranos

 *T-Bird wrote:*   

>  *mahir wrote:*   *snip* 
> 
> Did anyone solve this situation...I'm getting the same deal.

 

Try running without "C". So, using mahir's example:

```
tar cjpf /home/stage3.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc
```

I tried this, and what worries me is that I ended up with a .tar.bz2 file that was under 200MB, while my system is greater in usage, around 600MB or so. Just doesn't seem...right.

Edit: Unmerged my kernel sources and told Portage to fetch a copy of the sources (+~30MB), which killed off 200MB or so. Just re-made the .tar.bz2 file. It's incredible. Stage4 at 158MB. o.O

----------

## T-Bird

Has anyone tried to use this to date?  Any quirks?

----------

## DaNIsH

I successfully ran it recentely on my computer.

I got bored and wrote a java app (console) to generate a backup script:

Source - Updated (Fixed little bug)

Classes (Java) - Updated (Fixed little bug)

tar xvjf backupgen-<src/bin>.tar.bz2

cd backupgen

Source:

javac makebackup.java

java makebackup

Bin:

java makebackup

Obviously requires at least a JRE (Java runtime environment).

Anything breaks its someone elses fault, so best to check the script before executing.

Probably best to hide from my messy coding (source), and lack of programming skills  :Wink: 

If anyone finds it usefull I'll add-on a restore script.Last edited by DaNIsH on Sun Jul 04, 2004 12:55 am; edited 1 time in total

----------

## T-Bird

I just generated a backup...it's freaking 1.0GB in size...could I have done something wrong?

----------

## T-Bird

here was my command line:

```
tar cjpf /home/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/sys --exclude=/tmp --exclude=/mnt

```

----------

## DaNIsH

 *T-Bird wrote:*   

> I just generated a backup...it's freaking 1.0GB in size...could I have done something wrong?

 

You should exclude /usr/portage/distfiles, or at least clear it before backing up.

I'm not sure if I'm right in saying this, but instead of excluding dirs, just exclude the contents.

Eg. /proc/* /mnt/<mount>/*

I may be wrong, but I think that will cause problems when you restore the backup and linux tries to mount devices to non existant dirs etc.

----------

## T-Bird

Ok, that's a good point.  Is there anything in specific minus the distfiles that I should exclude?

----------

## kamina

Will I end up with a working backup if I exclude /proc /var/tmp /usr/portage/distfiles /mnt and /tmp?

----------

## DaNIsH

 *kamina wrote:*   

> Will I end up with a working backup if I exclude /proc /var/tmp /usr/portage/distfiles /mnt and /tmp?

 

Possibly, but IMO it's probably safer to just exclude the contents of certain dirs, as I mentioned in an above post.

----------

## nekranos

I just tested out the backup I made (three times in the past week, in fact, all due to stupidity). Works perfectly.

I didn't pass any options to "exclude" any directories, since I booted from a LiveCD, mounted up the drives, then tar.bz2'd up everything. Restoring it doesn't seem to cause any issues, either.

----------

## T-Bird

Before I try to update my system I'm backing up again just to make sure I can get the system back up if things screw up.  

The command line I used for this was:

```
tar cjpf /home/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/mnt/cdrom/* --exclude=/mnt/floppy/* --exclude=/usr/portage/distfiles/*

```

I'll have to see how big the file is now.

----------

## T-Bird

I found out on of the reasons for such a big backup was because of an mp3 directory that I forgot to exclude.  Hopefully things will become a little less now.Last edited by T-Bird on Thu Aug 26, 2004 4:55 pm; edited 1 time in total

----------

## CiscoSid

Anyone know if it's possible to combine a stage4.tar.bz2 into a live disc?

My stage4 is around 600MB, but is it possible to create a DVD iso image from the original LiveCD and add my stage4 to the stages dir?

That way I could just boot from a live DVD, and extract my stage4...

----------

## amiatrome

Is it possible to tar up our fresh windows install partition and just restore the fresh install from within Linux instead of reinstalling windows all over again the next time?   :Question: 

----------

## Grilo

Something i think everyone could use from this thread is a good script. Since I have no clue what a script is I asked a good friend, the one who introduced me to linux, to make me one. Here is the script for this backing up and a script to put it back together. I hope it is useful and if you have questions ask me and i'll see what I can do.

```

#!/bin/bash

# Backup script for Gentoo Linux

# Author: fdavid

# Date: 2003.11.29.

# Modified by Luke MacGregor, to have more than just the /home dir 

# backed up.

# Date: 2004.07.29

# Actual changes:

# 1. Made it use a command line argument to get the backup destination directory.

#    The usage is like this: "backup <directory to send backup file>"

# Designating which dirs to exclude in the backup process. You might want to change 

# this depending on what your needs are.

if [ $# -lt 1 ]; then 

   "echo usage: ./backup <backup file destination>"

   exit 1

fi

dirsExclude="--exclude=/mnt/* --exclude=/proc --exclude=/sys --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/var/tmp/*"

# options for the archive

tarOptions="cvjpf"

# name of the archive

archive=$1$(date +%Y%m%d).tar.gz

# create the archive

tar ${tarOptions}  ${archive} / --exclude=${dirsExclude} --exclude=${archive}  

echo archive is done

# split the archive to cd size (use: "cat ${archive}.* >> ${archive}" to join the parts)

split --bytes=700000000 ${archive} ${archive}.

echo splitting is done

```

and the put back

```

#!/bin/bash

echo You should be doing this as root. If you are not, it may result in errors.

if [ $# -lt 1 ]; then

        "echo usage: ./putback <backup file root name>"

        exit 1

fi

cat $1* > $1

echo Done putting backup together

echo Now mount your drives, and untar the full backup using something like:

echo tar xvjpf filename.tar.gz

```

I hope this works for you.

Thanks for the help on the forum to all.

Grilo

----------

## Phage64

I believe there is one minor error in the above script...

```

# create the archive

tar ${tarOptions}  ${archive} / --exclude=${dirsExclude} --exclude=${archive}

```

should be:

```

# create the archive

tar ${tarOptions}  ${archive} / ${dirsExclude} --exclude=${archive}

```

right?

and for gentoo you should also have /dev excluded

----------

## kalisphoenix

Yummy howto and thanks for the additional info, everyone.  Let's hope that I don't need this anytime soon  :Wink: 

----------

## ixtow

I was getting:

```

tar: Cowardly refusing to create an empty archive

```

So, as stated in this thread, I eliminated the "C"

```

# tar cjpf /home/public/stage4.tar.bz2 --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/mnt --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/sys

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

```

As you can see, this had no effect at all.

I added the "v" to see if it would tell me something and...

```

# tar cvjpf /home/public/stage4.tar.bz2 --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/mnt --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/sys

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

```

...I got jack...

It would seem that tar is just a coward and plain flat doesn't create archives at all....

----------

## ixtow

And, just to be thorough....

```

# tar cvjpf /home/public/stage4.tar.bz2 --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/mnt --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/sys --exclude=/home

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

# tar cvjpf /home/public/stage4.tar.bz2

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

# tar cjpf /home/public/stage4.tar.bz2

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information.

# tar cCjpf /home/public/stage4.tar.bz2

tar: Old option `f' requires an argument.

Try `tar --help' for more information.

```

----------

## Phage64

you have:

```

tar cvjpf /home/public/stage4.tar.bz2 --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/mnt --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/sys

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information. 

```

add a " /" in there to make it:

```

tar cvjpf /home/public/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc --exclude=/mnt --exclude=/tmp --exclude=/usr/portage/distfiles --exclude=/sys

tar: Cowardly refusing to create an empty archive

Try `tar --help' for more information. 

```

[/code]

and you may also want to exclude /dev

----------

## ixtow

spread cheeks, insert boot....

I really shouldn't be allowed to post messages anymore.....

----------

## tfunk

I'm missing somthing.  Sorry for being so dense....

I made the stage4 backup. everything okay there.

Pulled the backup off via SCP and pushed it to a new/identical machine

booted the new machine to live CD

fdisk, created filesystem, mounted swap, mounted /mnt/gentoo

Tried to use    tar xvjpf stage4.tar.bz2   to unpack it

I keep getting    Cannot write: No space left on device  errors

I know sense everyone else it seems to be working for that I HAVE to be missing something dumb because I'm noob.

Can someone help me figure out what it is?

Thanks!

Tfunk

----------

## allucid

check your file partitions (with fdisk or cfdisk). You may have fudged something up during that part of the install. or, for a quick check, try 'df -h' to see how much space you have.

----------

## tfunk

I give up for the time being...

I'm just going to install from scratch and then try it again.  I'm anxious to get this working for backups  :Smile: 

Thanks for the help thought.  I got into it and relised that for some reason I had /mnt/gentoo/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot so I must have had somthing pretty screwed up.

back to the drawing board  :Smile: 

Thanks!

Tfunk[/code][/quote]

----------

## allucid

nope. boot is symlinked to itself.

----------

## Deranger

Great HOWTO, I just backed up my whole system.

```

tar cjpf /home/oktane/stage4.tar.bz2 / --exclude=stage4.tar.bz2 --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/var/tmp/* --exclude=/dev/* --exclude=/mnt/cdrom/* --exclude=/mnt/floppy/* --exclude=/usr/portage/distfiles/*

```

Hopefully this works when I screw up something  :Wink: 

EDIT: Forgot to exclude /var/tmp/* and /dev/*...Tarring again at the moment.Last edited by Deranger on Sun Nov 21, 2004 8:16 pm; edited 1 time in total

----------

## nianderson

Check out the wiki I added 2 scripts to it one to create a stage4 and one to restore from it. appriciate any testing and feedback anyoen can offer.

http://gentoo-wiki.com/HOWTO_Custom_Stage4

down in step 6

----------

## rushdy

That backup script looks great, thanks. I havnt had chance to test it but i will soon. No more boot from live cd to backup by hand for me...   :Laughing: 

----------

## nianderson

I just tested the script that creates the stage4 it is a little broken but i have it fixed

one thing i added was to exclude /dev/* thats ok isnt it? everything in dev is created on boot if it dosnt exist right?

Wiki is updated with the working (at least for me) script.

including exclude /dev/* i can change that if i am wrong about dev entries being created on boot

----------

## postop

This worked great except for setting up grub off the live cd.

I unpack the tar, chroot'ed into /mnt/gentoo, copied /proc/mounts into /etc/mtab. and tried to run grub-install like the handbook says, but it complains about the device not existing. I looked at the /dev dir and the only thing there is null. 

I copied the /dev of the live cd into /mnt/gentoo/dev but grub-install still wouldn't work. If I did grub manually everything worked fine and runs just fine after rebooting.

Am I missing something? Is there a better way to fix this?

----------

## CRC

First, I don't know how you copied /dev, or why anyone would try to tar /dev.  Most Unix tools don't deal with /dev or sparse files properly at all!  Tar will try doing weird things like putting /dev/hda into the file, which is the entire contents of your hard disk!

Sparse files are created (in C/geek talk, using open, then seek, then write/close) so that the filesystem can store very large files but only store the data you specifically write into the file.  So a 2GB file might only be taking up 20MB, even if the file size says its 2GB.  Tar will try to back up 2GB, not 20MB.

The solution for both problems is simple.  Don't back up (or copy) using tar.  The best way to back up weird stuff and preserve EVERYTHING is to use cpio instead.  cpio is also a great way to copy whole directories including all subdirectories, permissions, etc (using -p).  Check "man cpio".  You usually feed it with "find" and then feed the output into gzip. 

To backup:

    find / | grep -v mybackup.cpio.gz | cpio -o -B | gzip -c >/mybackup.cpio.gz

To restore:

    gzip -cd </mybackup.cpio.gz | cpio -i -Bmd 

The grep just filters off the file from the list.  Cpio uses -i and -o to copy "in" and "out" of the filesystem (to do both, use -p, passthrough).  -B uses a larger buffer size (faster).  The -m is to preserve file modification time of the original and not use the current time.  The -d is to make any directories that it needs.

Have fun.

----------

## Deranger

I don't know about that, but tar-backup works great but you need to exclude things...

I've excluded:

/dev/*

/tmp/*

/var/tmp/*

/usr/portage/distfiles/*

/sys/*

/proc/*

/mnt/*

This Stage 4 is nice, it's like Stage 1 without bootstrapping  :Wink: Last edited by Deranger on Sun Nov 21, 2004 8:17 pm; edited 1 time in total

----------

## CRC

[quote= Oktane]

I don't know about that, but tar-backup works great but you need to exclude things..

[/quote]

Well, I do know about that.  I just told you.   Tar isn't great for system backups.  You can safely backup and restore device nodes with cpio (the major and minor numbers and device types are all backed up, and I feel this is important).  The handling of spare files can be really helpful for some database files.

As for excluding of files or directories, since cpio takes the list of files from stdin, you are free to modify the list however you want with the power of regular expressions.

Try it out on a test directory, try /tmp, or /home/user/testdir.

----------

## Deranger

Hmm, that sounds pretty neat...Maybe I'll give a try. But I don't see any point for backupping device nodes because I'm using pure UDEV  :Wink: 

----------

## CRC

Pure Udev?  Or Gentoo's UDEV?

You do realize that Gentoo does some work there to make udev work, including saving a tarball of devices (which is not possible, traditionally, with tar, but I've yet to scour the INFO pages to see when that feature was added to GNU's version - its certainly not portable).

I've also had some problems where device nodes didn't come back, like /dev/video0 and such.  They work fine now under udev, but I think its actually the device tarball that gentoo does that is saving that.  Basically, it doesn't hurt to back them up as it only takes up the space of the header, and its a good habit since many systems, like Sun or SCO, would be crippled if /dev got messed up.  Also, some linux malware hides itself in /dev.

I also like how cpio tends to be a cleaner interface, fewer flags required.

----------

## postop

 *Oktane wrote:*   

> I don't know about that, but tar-backup works great but you need to exclude things...
> 
> I've excluded:
> 
> /dev/*
> ...

 

Am I doing something wrong with grub then? The wiki says fstab and lilo/grub like it's something really simple.

----------

## Deranger

I'm lost...Is this really everything needed to make full backup? How can I speficy things I want exclude? Tarring is much more simplier.

```

To backup:

find / | grep -v mybackup.cpio.gz | cpio -o -B | gzip -c >/mybackup.cpio.gz

To restore:

gzip -cd </mybackup.cpio.gz | cpio -i -Bmd

```

----------

## postop

 *CRC wrote:*   

> First, I don't know how you copied /dev, or why anyone would try to tar /dev.  Most Unix tools don't deal with /dev or sparse files properly at all!  Tar will try doing weird things like putting /dev/hda into the file, which is the entire contents of your hard disk!
> 
> Sparse files are created (in C/geek talk, using open, then seek, then write/close) so that the filesystem can store very large files but only store the data you specifically write into the file.  So a 2GB file might only be taking up 20MB, even if the file size says its 2GB.  Tar will try to back up 2GB, not 20MB.
> 
> The solution for both problems is simple.  Don't back up (or copy) using tar.  The best way to back up weird stuff and preserve EVERYTHING is to use cpio instead.  cpio is also a great way to copy whole directories including all subdirectories, permissions, etc (using -p).  Check "man cpio".  You usually feed it with "find" and then feed the output into gzip. 
> ...

 

I took a peek into gentoos stage3.tar.bz2 and there is a full /dev in there. So I'm wondering is the only reason people exclude /dev to make it portable for different machines? So why does gentoo release a stage 3 tar with /dev in it if it's supposed to be portable? Does the x86 stage 3 archive just have a /dev that is supposed to be common to all x86 architectures? Is it safer to exclude /dev and run MAKEDEV instead? 

As for how I copied the /dev, I just booted from the live cd, mounted /mnt/gentoo and copied the live cd's /dev into /mnt/gentoo/dev. I figured that if it ran MAKEDEV to build it's /dev dir, then copying it wouldn't be an issue.

I've seen a couple of threads from a google search about tar/cpio comparisons, http://www.delorie.com/gnu/docs/tar/tar_122.html in particular. It seems this debate has been going on for a while. Anyone else like chime in with their pros and cons?[/u]

----------

## CRC

 *postop wrote:*   

> I took a peek into gentoos stage3.tar.bz2 and there is a full /dev in there. So I'm wondering is the only reason people exclude /dev to make it portable for different machines? So why does gentoo release a stage 3 tar with /dev in it if it's supposed to be portable? Does the x86 stage 3 archive just have a /dev that is supposed to be common to all x86 architectures? Is it safer to exclude /dev and run MAKEDEV instead? 

 

Yeah, Admins are creatures of habit.  First time tar gave me issues with /dev, I quit using it for any sort of large system backups and started using cpio.  I've done complete installs from gzipped cpio achives over a network!   GNU Tar under Linux shouldn't be an issue, but if you have problems, use cpio!  What would you be using instead of GNU Tar? Hmm other than many Solaris, Irix, probably AIX, etc, and ... how often have you been using some boot CD, alterative install method, embedded system or whatever, that is using busybox?  I don't know if busybox can handle special files or not.  Maybe it can.  I know cpio does.  Pick whichever one works best for you really, I just wanted to get another option out there, which happens to be my personal preference.

As for MAKEDEV, You would use MAKEDEV on Linux up to 2.4.  For most 2.4 systems you should probably migrate to devfs (although the tarball or cpio-ball helps when devfs doesn't restore permissions just right).  For 2.6, the push is to move to udev.  The benefits of udev won't be immediately noticeable until the desktops catch up.

 *postop wrote:*   

> As for how I copied the /dev, I just booted from the live cd, mounted /mnt/gentoo and copied the live cd's /dev into /mnt/gentoo/dev. I figured that if it ran MAKEDEV to build it's /dev dir, then copying it wouldn't be an issue. 

 

Now I feel old.  In the old days, trying to do a recursive copy of /dev would hang the system and bork up all kinds of stuff.  The solution was simple.  DON'T DO THAT.  And I haven't done it many years.  Looks like even 'cp' will do the right thing now.  No wonder the GNU tools are so getting so big.  Nice to know, but next time I take a snapshot of /dev files, I doubt I'll use "cp" over "cpio" ... habit.  I trust cpio.

 *postop wrote:*   

> I've seen a couple of threads from a google search about tar/cpio comparisons, http://www.delorie.com/gnu/docs/tar/tar_122.html in particular. It seems this debate has been going on for a while. Anyone else like chime in with their pros and cons?

 

Well, the author seemed biased towards tar.  I've never had problems with cpio not recognizing hard links due to the limited inode number size.  Perhaps much larger filesystems would be an issue.  The comparison on that site didn't mention sparse files at all.    There are goods and bads for each approach,  I personally prefer to feed a file list from stdin vs wildcards and "excluding".

Anyone that wants to do real backups should look into a dedicated backup program.  tar and cpio are fine to backup to floppy or CD or even DVD for personal use, but if you have a business running off that data, you might want to look into another method.  For open-source, try Amanda.  You might also want to look into using LVM2 so you can grow/shrink partitions easier and make use of "snapshots" to make backups easier.  FYI, while most source code is downloaded in tar format, most RPMs are a header around a cpio archive.

As for grub/lilo, lilo is just a pain to back up since you have to reinstall it every time the position of a file on disk changes.  For grub, you can save a copy of your partition table which should have the boot record too.  Something like

```
dd if=/dev/hda of=/dev/boot.img bs=512 count=1
```

to save it, and then

```
dd if=/dev/boot.img of=/dev/hda bs=512 count=1
```

should restore it.

I haven't tested this in awhile, so please don't test this on something you can't fix.  And "gpart" is your friend if you bork your partition table.  You can NOT use this boot.img on any other drive or partitioning scheme other than the one it was saved on.  You will end up reinstalling everything (or running gpart off a boot disk like LNX-BBC and praying).

Another way to do a "stage 4" might be to just turn on "buildpackages" in FEATURES (I think thats the right string), or always use the "-b" flag to emerge.  You can then install these again with emerge -k.  Still doesn't help with grub/lilo, but thats a part of disaster recovery and is the job of the backup system to do for you.  Most good restore programs can boot the system restore the backup and reinstall everything.  Also check out this info: 

http://www.backupcentral.com/toc-free-backup-software.html

----------

## Gavrila

I've found to be able to use --exclude options only if put BEFORE the starting directory from where to start tarring

e.g.:   tar cjpf /path/to/backup.tar.bz2 / --exclude=/foo

won't exclude /foo pattern matching files

while:

        tar --exclude=/foo cjpf /path/to/backup.tar.bz2 / 

will do its dirty work. I think   you should update you're wiki how-to.

Kind Regards

----------

## tscolari

Just a question

I would like to prepare a stage4 for a computer at my work.

But here i use a athlon xp and there it is a pentium 233mhz... How can I (is it possible?) make my althlon compile all for a pentium? or i will have a lot of problems?

----------

## Gavrila

 *tscolari wrote:*   

> Just a question
> 
> I would like to prepare a stage4 for a computer at my work.
> 
> But here i use a athlon xp and there it is a pentium 233mhz... How can I (is it possible?) make my althlon compile all for a pentium? or i will have a lot of problems?

 

u can choose or -mtune=athlon-xp which will try to optimize for the amd but will keep compatibility for any other x86.

Otherwise u could try -march=i686 (is it ? o i586 for that pentium?)  which will be compatible with both.

As last option u can compile with -march=pentium2 (or whatever it is), but u won't be able to use it on ur amd machine.

regards

----------

## CRC

 *Gavrila wrote:*   

>  *tscolari wrote:*   Just a question
> 
> I would like to prepare a stage4 for a computer at my work.
> 
> But here i use a athlon xp and there it is a pentium 233mhz... How can I (is it possible?) make my althlon compile all for a pentium? or i will have a lot of problems? 
> ...

 

Ah, well, the Athlon can and does run Pentium and Pentium2 code, and any pentium3 code that doesn't use sse instructions.

Your pentium is a pentium, 586, not 686, so you want to compile for 586 being the minimum architecture but tune for 686 (never tune for pentium unless you will never run the code on anything faster).  Tuning for 686 doesn't really hurt pentium speed, but vice-versa is true.

As for AMDs, K5 is 486, K6 is 586/pentium, K6/2 is K6 with MMX (and 3dnow), K6/3 is K6/2 with Mobile/PowerNow.  

The Pentium MMX was Intel's new chip at the time for desktops, while the Pentium Pro was for servers.  The Pentium Pro was the 686.

The Pentium2 (686) is the Pentium Pro with MMX enhancements.  The Pentium 3 is just a Pentium 2 with the SSE instructions added (to compete with 3dnow).

The Pentium 4 is a complete redesign that is tuned for media instructions, and actually runs typical integer/business type instructions SLOWER than a pentium3 at the same clock speed (for multiple reasons).  Its claim to fame is purely the high clock rates.

The Athlon has improved 3dnow, plus a screaming fast floating point and integer unit.  

If you optimize for Athlon or Pentium4, throw compatibility out the window.  Most Athlons are variations of K7, and there are quite a few of them and they are all a bit different to optimize for.  The K8s and AMD 64s are new animals too.  Seems Intels latest generations aren't getting very far.  Oh, and when you see Celeron, its just the "cheaper" version of the same chip (like the old SX designation), and Xeon is a larger cached Server version.

----------

## seppe

I backed up with this command:

```

tar cjpf /20041022-stage4-Helios.tar.bz2 / --exclude=20041022-stage4-Helios.tar.bz2 --exclude=proc/* --exclude=dev/* --exclude=sys/* --exclude=tmp/* --exclude=stage4-errors.log 2>stage4-errors.log

```

I'm going to write it to a DVD, I hope I can restore my system when it breaks someday

----------

## seppe

hmm, it's still busy but I fired up cat stage4-errors.log already, and it showed this:

```

tar: Removing leading `/' from member names

tar: Removing leading `/' from hard link targets

tar: /dev/log: socket ignored

tar: /proc/acpi/event: Cannot open: Device or resource busy

tar: /proc/2/task/2/exe: Cannot readlink: No such file or directory

tar: /proc/2/exe: Cannot readlink: No such file or directory

tar: /proc/3/task/3/exe: Cannot readlink: No such file or directory

tar: /proc/3/exe: Cannot readlink: No such file or directory

tar: /proc/4/task/4/exe: Cannot readlink: No such file or directory

tar: /proc/4/exe: Cannot readlink: No such file or directory

tar: /proc/5/task/5/exe: Cannot readlink: No such file or directory

tar: /proc/5/exe: Cannot readlink: No such file or directory

tar: /proc/6/task/6/exe: Cannot readlink: No such file or directory

tar: /proc/6/exe: Cannot readlink: No such file or directory

tar: /proc/27/task/27/exe: Cannot readlink: No such file or directory

tar: /proc/27/exe: Cannot readlink: No such file or directory

tar: /proc/28/task/28/exe: Cannot readlink: No such file or directory

tar: /proc/28/exe: Cannot readlink: No such file or directory

tar: /proc/39/task/39/exe: Cannot readlink: No such file or directory

tar: /proc/39/exe: Cannot readlink: No such file or directory

tar: /proc/40/task/40/exe: Cannot readlink: No such file or directory

tar: /proc/40/exe: Cannot readlink: No such file or directory

tar: /proc/42/task/42/exe: Cannot readlink: No such file or directory

tar: /proc/42/exe: Cannot readlink: No such file or directory

tar: /proc/41/task/41/exe: Cannot readlink: No such file or directory

tar: /proc/41/exe: Cannot readlink: No such file or directory

tar: /proc/577/task/577/exe: Cannot readlink: No such file or directory

tar: /proc/577/exe: Cannot readlink: No such file or directory

tar: /proc/599/task/599/exe: Cannot readlink: No such file or directory

tar: /proc/599/exe: Cannot readlink: No such file or directory

tar: /proc/606/task/606/exe: Cannot readlink: No such file or directory

tar: /proc/606/exe: Cannot readlink: No such file or directory

tar: /proc/607/task/607/exe: Cannot readlink: No such file or directory

tar: /proc/607/exe: Cannot readlink: No such file or directory

tar: /proc/608/task/608/exe: Cannot readlink: No such file or directory

tar: /proc/608/exe: Cannot readlink: No such file or directory

tar: /proc/8754/task/8754/exe: Cannot readlink: No such file or directory

tar: /proc/8754/exe: Cannot readlink: No such file or directory

tar: /proc/10020/task/10020/fd/3: Cannot stat: No such file or directory

tar: /proc/10020/fd/3: Cannot stat: No such file or directory

```

How can that be? I excluded /proc and /dev, why do I get errors on it then? Could it be that --exclude doesn't handle sub directories or something? Anyone know what's wrong here, and should I stop the tarring and restart with another command? Or isn't this a problem when I extract my stage4 in the future?

----------

## Deranger

Seppe, you have

```

--exclude=proc/*

```

It should be

```

--exclude=/proc/*

```

And so on...

If you don't need/want any downloaded sources to be included, exclude /usr/portage/distfiles/*Last edited by Deranger on Fri Oct 22, 2004 2:30 pm; edited 1 time in total

----------

## seppe

 *Oktane wrote:*   

> Seppe, you have
> 
> ```
> 
> --exclude=proc/*
> ...

 

I tried that first, and I think I would have the same output. I got tar: /proc/acpi/event: Cannot open: Device or resource busy then as well, but I quit it to try it without leading / because tar said 

 *Quote:*   

> 
> 
> tar: Removing leading `/' from member names
> 
> 

 

Besides, I run this command from /, so it doesn't matter, I think?

----------

## Deranger

I'm not sure, but add / in front of them, try to tar your system again and see if there's same errors.

----------

## seppe

Hmm I don't get it, I tried it again with this command:

```

tar cjpf /20041022-stage4-Helios.tar.bz2 / --exclude=/20041022-stage4-Helios.tar.bz2 --exclude=/proc --exclude=/dev --exclude=/sys --exclude=/tmp --exclude=/stage4-errors.log 2>stage4-errors.log

```

And I still get the same errors on /proc and /dev, which should be excluded  :Shocked: 

----------

## gungholady

 *seppe wrote:*   

> Hmm I don't get it, I tried it again with this command:
> 
> ```
> 
> tar cjpf /20041022-stage4-Helios.tar.bz2 / --exclude=/20041022-stage4-Helios.tar.bz2 --exclude=/proc --exclude=/dev --exclude=/sys --exclude=/tmp --exclude=/stage4-errors.log 2>stage4-errors.log
> ...

 

I used the script from gentoo-wiki which I modified just slightly for what I want and it works just fine.

----------

## CRC

What no one has mentioned yet is how to use a "stage4" and use it with a network boot image, such as a pxeboot or grub (grub can optionally be compiled with network drivers to boot kernels over the network, so you can use it as a boot floppy).

If you've ever seen a large datacenter do installs do 500 machines, you'd know they don't pop a CD into each one and manually install the system.  That would be insane.  Gentoo would be much more acceptable for servers if there was more automated support for creating a "stage4" with apache and friends and then blowing onto as many servers as you want.

Anyone want to write a how-to?

----------

## BlinkEye

please continue here if you're interested in my script. i won't update this version anymore.

INDEX

0. What's this all about

1. How it works

2. How to customise

3. The results

4. The script

5. What now

6. Warnings

0. What's this all about

i want to backup my system easily with minimum effort. that's why i wrote this script. the resulting backup file (.tar.bz2) is called a stage4. you're probably a gentoo user and hence know the three stages gentoo uses. this backup will be a stage4 meaning  it will be a full system backup of your current system. 

1. How it works

it's a bash script which uses only "standard" tools, namely "tar", "bzip2" and if you intend to burn the stage4 on a cd you'll probably use "split" too. this means this script works out of the box. it has been modified a lot and i now tried to detect any error/misconfiguration a user might do (wrong $PATH, wrong commands, wrong exclude list, overwriting existing files ...). 

after checking everything's all set up the script will provide you with two options:

```

Backup script v1.1

===================

What do you want to do? (Use CONTROL-C to abort)

(1) Minimal backup

(2) Interactive backup

Please enter your option: 
```

the minimal backup allows you to quickly build a stage4.tar.bz2 without any further fuss, which means

1. it will exclude (i.e. NOT backup) any files/directories listed in the $exclude_custom_list variable

2. result in a minimal stage4

the interactive backup will ask you for any file/directory listed in the $exclude_custom_list variable if you want it to backup. this might result in the same stage4.tar.bz2 if you answer all question with "no" or in a considerably bigger stage4.tar.bz2. the idea of the $exclude_default_list is to put any file/directory in there which is never needed for a minimal full system backup. files/directories listed in $exclude_custom_list are directories which aren't needed either but which may be desirable to be backup-ed too (like /home or /usr/src/). of course opinions about such files/directories differ, so suit yourself.

2. How to customise

as mentioned above there are several variables to customise. the script itself has a few comments which should get you started. you might want to change the location where the stage4.tar.bz2 is put (currently /mnt/backups/stage4), or you might not like the filename (hostname-stage4-datum.tar.bz2), or you might think that the kernel sources is part of a minimal system backup and hence you won't exclude it (and so on) ...

3. The results

the result is IMPRESSIVE. you'll get a stage4.tar.bz2 (for me it's about 800MB from 3.5GB of data) which will be a fully working backup of your system. on my ibm laptop (x40 - with a damn slow harddisk) - it takes about 50 minutes to create the stage4. i recall all those hours i backup-ed my windows installation. the record was 7CD's for a full installation (inkl. all the software you need for the daily use).

4. The Script

```
#!/bin/bash 

# Backup script for Gentoo Linux 

# Author: Reto Glauser aka Blinkeye 

# Mailto: stage4 at blinkeye dot ch

# Date: 23.03.2005 

version=v1.2

# these are the commands we actually need for the backup

command_list="echo tar hostname date split"

# verify that each command we use exists

for command in $command_list; do

 path=`which $command | grep "no $command in"`

 

 if [ ! -x `which $command` -a "$path" ]; then

  echo -e "\n\nERROR: $command not found! Check your commands and/or your \$PATH"

  exit -1 

 fi

done

# options for the tar command 

tarOptions="--create --absolute-names --preserve-permissions --totals --bzip2 --ignore-failed-read --verbose --file" 

# where to put the stage4

stage4Location=/mnt/backups/stage4

# name prefix

stage4prefix=$(hostname)-stage4-`date +\%d.\%m.\%Y`

# these files/directories are always excluded

default_exclude_list="

--exclude=/tmp/* 

--exclude=/var/tmp/* 

--exclude=/lost+found/* 

--exclude=/dev/* 

--exclude=/proc/* 

--exclude=/mnt/* 

--exclude=/sys/* 

--exclude=/usr/portage/*

--exclude=/var/log/*

--exclude=$stage4Location"

# depending on your choice these files or directories will additionally be excluded

custom_exclude_list="

--exclude=/usr/src/*

--exclude=/opt/mathematica

--exclude=/usr/share/smssend

--exclude=/home/*"

# check the folder/files stored in $default_exclude_list exist

for exclude in $default_exclude_list; do

   if [ ! -e "`echo "$exclude" | cut -d'=' -f2 | cut -d'*' -f1`"  ]; then

      echo -e "\n\nERROR: `echo "$exclude" | cut -d'=' -f2` not found! Check your \$default_exclude_list"

   fi

done

# check the folder/files stored in $custom_exclude_list exist

for exclude in $custom_exclude_list; do

   if [ ! -e "`echo "$exclude" | cut -d'=' -f2 | cut -d'*' -f1`"  ]; then

      echo -e "\n\nERROR: `echo "$exclude" | cut -d'=' -f2` not found! Check your \$custom_exclude_list"

   fi

done

# print out the version

 echo -e "\nBackup script $version"

 echo -e "==================="

# how do you want to backup?

echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n

(1) Minimal backup

(2) Interactive backup"

while [ "$option" != '1' -a "$option" != '2'  ]; do

   echo -en "\nPlease enter your option: "

   read option

done

case $option in

1)

   stage4Name=$stage4Location/$stage4prefix-minimal

   final_command="tar $default_exclude_list $custom_exclude_list $tarOptions $stage4Name.tar.bz2 / /var/log/emerge.log"

   ;;

2)

   for folder in $custom_exclude_list; do 

      echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) " 

      read answer 

      while [ "$answer" != 'y' -a "$answer" != 'n' ]; do 

         echo "please enter y or n" 

         read answer 

      done 

      if [ "$answer" == 'n' ]; then 

         default_exclude_list="$default_exclude_list $folder" 

      fi 

   done 

   

   stage4Name=$stage4Location/$stage4prefix-custom

   final_command="tar $default_exclude_list $tarOptions $stage4Name.tar.bz2 /  /var/log/emerge.log"

   ;;

esac

# show what will be done

echo -e "\n* creating the stage4 at $stage4Location with the following options:\n\n"$final_command

# everything is set, are you sure to continue?

echo -ne "\nDo you want to continue? (y/n) " 

read answer 

while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do 

         echo "please enter y or n" 

         read answer 

done 

if [ "$answer" == 'y' ]; then 

   # mount boot 

   echo -e "\n* mount boot" 

   mount /boot >/dev/null 2>&1   

   

   # if necessary, create the stage4Location 

   if [ ! -d "$stage4Location" ] ; then 

      echo "* creating directory $stage4Location" 

      mkdir -p $stage4Location 

   fi

   

   # check whether the file already exists

   if [ -a "$stage4Name.tar.bz2" ]; then  

      echo -en "\nDo you want to overwrite $stage4Name.tar.bz2? (y/n) "

      read answer

      while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

         echo "please enter y or n"

         read answer

      done

      if [ "$answer" == 'n' ]; then

         echo -e "\n* There's nothing to do ... Exiting" 

         exit 0;

      fi

   fi

   

   # do the backup

   time $final_command

   # copy the current world file to the stage4 location

   echo -e "\n* creating stage4 overview $stage4Name.txt"

   cp /var/lib/portage/world $stage4Name.txt >/dev/null 2>&1

   

   # we finished, clean up

   echo "* stage4 is done" 

   echo "* umounting boot" 

   umount /boot

else

   echo -e "\n* There's nothing to do ... Exiting" 

fi 

#Uncomment the following command if you want to split the archive in cd size chunks:

#split --suffix-length=1 --bytes=670m $stage4Name.tar.bz2 "$stage4Name"_ && echo "* splitting is done"
```

5. What now

you now have a stage4.tar.bz2. if in need (or if you use this stage4 for another box) 

1. untar it

```
 tar xvjpf X40-stage4-06.03.2005-minimal.tar.bz2
```

2. rebuild the portage tree

```
emerge sync
```

3. if necessary install a bootloader, adjust the bootloader and your /etc/fstab

6. Warning

although it's not a disaster it's still anoying. if you have a symlink in /boot 

```
# ls -l /boot

total 2588

lrwxr-xr-x  1 root root       1 Jul  7  2004 boot -> .
```

backing up /boot will result in a recursive backing up of boot. this is wasting space and time. the solution is easy, remove the symlink:

```
rm /boot/boot
```

DANGER:  if you haven't configured your bootloader correctly (namely grub), you might not be able to boot your kernel. this results of a wrong configuration, which of course is easily solved. if you don't have a seperate /boot partition (check your fstab) your kernel line should look similar to this one:

```
...

kernel (hd0,0)/boot/linux-2.6.11-ck1/bzImage root=/dev/hda3
```

if you use a seperate boot partition remove "/boot"

```
...

kernel (hd0,0)/linux-2.6.11-ck1/bzImage root=/dev/hda3
```

[EDIT]2005-03-06 Completely rewrote the old script[/EDIT]

[EDIT]2005-03-25 Adjusted the split command [/EDIT]

----------

## tkdfighter

BlinkEye, that script of yours looks really good. I'm going to try it out this evening.  :Very Happy: 

----------

## M@rijn

Can i just exclude the /sys directory?? and don't backup this dir? Because i have errors on reiser4 with tarring the /sys directory.

----------

## BlinkEye

yes, but do it like that: --exclude=/sys/* (btw: peek at my script, i do it too)

----------

## M@rijn

 *BlinkEye wrote:*   

> yes, but do it like that: --exclude=/sys/* (btw: peek at my script, i do it too)

 

Oh damned i see it, your script makes an kind of image of the system, i was making also a script like that  :Smile:  I think i use yours, so i don't have to test it  :Smile: 

----------

## seppe

Hi

I created a stage4 once, and today .. my reiser4 system crashed.

So I restored from my stage4 snapshot, and everything seems to work, except I don't see my runscripts loaded anymore (like "loading sshd    [OK]"), but they load in the background because my services all work without any problems. I just don't SEE them loaded.

Also, I get "Warning: could not open an initial console"

and

"Adding xxxxxxxk swap to /dev/hda3. Initial:-1 extents:1"

Does anyone know how I can fix this?

Apart from this, everything works without problems

----------

## BlinkEye

sounds very familiar. had that one too with reiser4. check the forums - there are some topics around dealing with that issue. i think you need to create some nodes/devices ...

[EDIT] klick me and/or me may help

----------

## Deranger

I did chrooted install (on another partition), and moved it to my system harddrive...I got similar "cannot open initial console" problem and it was caused by an invalid entry on grub.conf  :Embarassed: 

----------

## seppe

it can't be my grub.conf, because I haven't changed that one for ages.

If anyone can point me to the right direction ...

----------

## seppe

Ok, I fixed it ..

see https://forums.gentoo.org/viewtopic.php?p=1718752#1718752 for a solution. Now everyting works without any problems. Thanks again for this great howto, it saved me 2 days of compiling  :Wink: 

----------

## M@rijn

why excluding so many directory's does he rebuild it at hisself???

----------

## BlinkEye

do you mean my exclude directories? no, most of them are dynamically created upon boot time - and some big parts of it (like /var/tmp/portage or /usr/portage are created by portage, might get VERY big and don't need to be backed up because either they are temporary (like /var/tmp/portage or /tmp) or they (/usr/portage get recreated upon an

```
emerge sync
```

ok?

[EDIT] to make it clear - except of an "emerge sync" one doesn't need to rebuild anything [/EDIT]

----------

## M@rijn

Maybe an idea, can you make the script so that he automaticly sync your portage?  :Smile:   :Embarassed: 

----------

## BlinkEye

 *M@rijn wrote:*   

> Maybe an idea, can you make the script so that he automaticly sync your portage?  

 

i don't quite understand - would you like to backup your system with portages structure included? if so, there's no problem - remove the --exlude=/usr/portage command. if you d like that after decompressing your stage4.tar.bz2 it would automatically sync portage again - well, i can't do much about that as the command to decompress the archive is just a tar command (tar -xvjpf stage4.tar.bz2). but if you'd like to automate that too simply execute the following:

```
tar -xvjpf stage4.tar.bz2 && emerge sync
```

----------

## M@rijn

ok thnx

----------

## Greven

I made a few changes to you script Blinkeye

```

#!/bin/bash

# Backup script for Gentoo Linux

# This program is distributed under the terms of GPL version 2.

# Author: BlinkEye

# Date: 03.12.2004

# Comments/Remarks mailto "blinkeye dot gmx dot ch"

# check if /home/* will be backed up too

/bin/echo -n "Do you want to backup /home/* ? (y/n) "

read -t 10 home

while [ "$home" != 'y' ] && [ "$home" != 'n' ]; do

        /bin/echo "please enter y or n"

        read home

done

if [ "$home" == 'n' ]; then

        exclude="--exclude=/home/*"

        stage4Name=$(hostname)-stage4-minimal-`date +\%d-\%m-\%Y`.tar.bz2

else

       stage4Name=$(hostname)-stage4-full-`date +\%d-\%m-\%Y`.tar.bz2

fi

# mount boot

/bin/echo "* mount boot"

/bin/mount /boot >/dev/null 2>&1

# where to put the stage4

stage4Location=/mnt/stage4/

/bin/echo "* Backup stage4 to $stage4Location"

# if necessary create the stage4Location

if [ -d "$stage4Location" ] ; then

        /bin/echo "* directory $stage4Location already exists"

else

        /bin/echo "* creating directory $stage4Location"

        /bin/mkdir -p $stage4Location

fi

# copy the current world file to the stage4 location

/bin/echo "* creating stage4 overview"

/bin/cp /var/lib/portage/world $stage4Location/$(hostname)-stage4-`date +\%d-\%m-\%Y`.txt >/dev/null 2>&1

# options for the archive

tarOptions="--exclude=$stage4Location --exclude=/tmp/* --exclude=/var/tmp/* --exclude=/lost+found/* --exclude=/dev/* --exclude=/proc/* --exclude=/mnt/* --exclude=/sys/* --exclude=/usr/portage/* --exclude=/var/log/* $exclude --create --absolute-names --preserve-permissions --bzip2 --totals --ignore-failed-read --verbose --file"

/bin/echo -e "* creating tar.bz2 archive with the following options:\n"

# create the archive

/bin/echo "tar $tarOptions $stage4Location$stage4Name /"

/bin/echo ""

time /bin/tar $tarOptions $stage4Location$stage4Name /;

/bin/echo -e "\n* stage4 is done"

/bin/echo "* umounting boot"

/bin/umount /boot

```

It's not very big changes.... it just makes direct calles to the program and not alias'.  I had a co-work once alias "mkdir" to "cp .." to watch my face.

----------

## BlinkEye

thanks for the input. i changed a few things too and forgot to adjust them here. so, an update has been made. for the moment i won't reflect your changes/suggestion as i think it's more portable this way, but i'll keep it in mind as it's a cleaner way to do things.

----------

## My_World

Doing the following gives one very disturbing news indeed:

```

tar cjpfv myname.tar.bz2 / --exclude=myname.tar.bz2 --exclude=/proc/* --exclude=/dev/* --exclude=/mnt/*

```

Why I say this? You'll notice the verbose I added and watching the output carefully I get:

```

/mnt/new/home/myworld/.fluxbox/styles/sea/pixmaps/closeufcs.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/

/mnt/new/home/myworld/.fluxbox/styles/Bawn/theme.cfg

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/selected.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/max.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/icon.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/stuck.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/wtitle.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/stick.xpm

/mnt/new/home/myworld/.fluxbox/styles/Bawn/pixmaps/close.xpm

/mnt/new/home/myworld/.fluxbox/styles/Amok/

```

The directories specified is NOT excluded. What on earth am I missing here?

 :Confused: 

----------

## BlinkEye

because the order of the flags matter. as soon as you use -f (--file) and specify the filename any other options is ignored or considered as a file to archive. 

you should use 

```
tar --exclude=myname.tar.bz2 --exclude=/proc/* --exclude=/dev/* --exclude=/mnt/* -cjpvf myname.tar.bz2 /
```

try it, like

```
tar --exclude=myname.tar.bz2 --exclude=/proc/* --exclude=/dev/* --exclude=/mnt/* -cjpvf myname.tar.bz2 /mnt/*
```

which wouldn't return anything as nothing would be backed up because of your exlude options

----------

## gungholady

Greven,

Your script works great. I tried adding to it to cause it to split the created file into chunks to be burned onto cd's. What I tried adding didn't work. 

Here is what I tried using:

#Uncomment if you want to split the tar in cd size chunks

# split the archive to cd size (use: "cat ${archive}.* >> ${archive}" to join the parts)

split --bytes=650000000 ${archive} ${archive}.

echo splitting is done

The only part of it that worked is the echo part. It did not do the split. If someone knows what needs to be changed to work with Graven's script, I would appreciate it.

----------

## BlinkEye

[EDIT] misleading information was posted here due to a false article i read and due to not trying it out myself [/EDIT]

----------

## gungholady

Yes, you can split bz2 files. I do it all the time. The script I use to backup my home files just finished and they split just fine and they can be put back together and untarred just fine.

----------

## gungholady

Splitting of bz2 files is done in the original mkstage4 script found here:

http://gentoo-wiki.com/HOWTO_Custom_Stage4

I would just like to figure out how to modify the split part of it to work with greven's script. I guess I'll just have to experiment with the difference in his script and the one from gentoo-wiki.com.

----------

## gungholady

When added to the end of greven's script after the line "/bin/echo -e "\n* stage4 is done"", the following will split the file to cd size chunks:

split --bytes=650000000 $stage4Location$stage4Name $stage4Location$stage4Name.

echo splitting is done

The 650000000 can be adjusted according to the size of your cd's.

----------

## BlinkEye

i modified my script and changed the split command

```
split --suffix-length=1 --bytes=690m $stage4Location$stage4Name $stage4Location"$stage4Name"_
```

 which results in chunks like

```
X40-stage4-minimal-11-12-2004.tar.bz2_a

X40-stage4-minimal-11-12-2004.tar.bz2_b

...
```

i made some changes which allow you now to select interactively which folder/files you may want to back up (the modified script is still found here).

----------

## Gentree

I've been on R4 for /home and PORTAGE_TMPDIR for about a year with no crashes lost files or anything.

In the last few days I've been having erratic behaviour. Specifically opera crashing out , once when I closed a window , once when I did empty trash in email client.

Opera may be handling the errors very badly but I am starting to suspect the underlying issue is R4.

I had to fsck.rieser4 to rebuild the superblock , then the fs and only then did I have a stable partition but the lost nodes etc meant that I had lost 3/4 of my mail archive.

Now R4 is supposed to be a fully atomic , journalling fs so how is this even possible and why after perfect functioning since Beta?

At some point in the last 2 weeks I have been using 2.6.10-nitro2 and -nitro4 .

I have just gone back to 2.6.9-nitro1 that I used since it came out around sept2004. Fingers crossed.

How many having issue here are using R4 on 2.6.10 (probably nitro*) ?

I dont beleive R4 has gone flakey after a year without a reason.

Spec check , hands up those on recent nitro.  :Cool: 

[EDIT]

Sorry , post went to wrong thread, regard it as [OFF] and heed it if it applies to you.

----------

## Matteo Azzali

Well, since I liked this script a LOT, I've modified this to run in my KDE environment without exiting or opening console. (Think to it like a 

little frontend-ed version for KDE,  (...Kmkstage4 ?  :Smile:  )

It has a progress bar but till now I haven't found a way to syncronize it

with tar  :Crying or Very sad: 

```

Sorry, kdialog seems buggy if used as root. no-way.

```

Tested working even if the progress bar stay under the message dialogs,

feel free to modify, expecially if u can sync progress bar with tar . 

(Lol, I'm trying  :Razz:  )

----------

## Matteo Azzali

 kmkstage4 final

(make your stage4 under KDE, no need to run in console or as root,

it ask for su with kdesu)

Ok,thanks yaneurabeya for the tip, now it uses xterm and runs fine,

without any problem, neither security neither unstopped process, it 

also asks for split the archive. (care, split don't delete the original

big tar)

NOTE: I did not pointed that before cause I thinked it was obvious but....

cause the nature of this script SAFER way to use is to log as root and set:

```
 chown root:wheel kmkstage4

chmod 751 kmkstage4              (chmod 750 also will do...)

```

```

#!/bin/bash

# Backup script for Gentoo Linux

# Author: BlinkEye

# Date: 09.01.2004

# KDE: Mat

# ask for special folders to be backed up too

list="/home /var/log /usr/portage/distfiles"

kdedial=`locate kdialog | grep bin/kdialog`

kdedialb=`dirname $kdedial`

kdedial=$kdedialb"/kdialog"

kdedc=$kdedialb"/dcop" 

kdeforsu=$kdedialb"/kdesu"

#echo "starting kmkstage4 for Kdialog in $kdedial ."

#echo "if its not correct,locate was required"

dcopRef=`$kdedial --progressbar "Configurating" 4`

$kdedc $dcopRef setProgress 1

for folder in $list; do

        $kdedial --title "Kmkstage4" --yesno "Do you want to backup $folder?"

   answer=$?

   if [ $answer = "1" ]; then

                exclude="$exclude --exclude=$folder"

        elif [ "$folder" != "/usr/portage/distfiles" ]; then

                folder=$folder"/*"

                for subfolder in $folder; do

                        $kdedial --title "Kmkstage4" --yesno "Do you want to backup $subfolder?"

         answer=$?

              if [ $answer = "1" ]; then

                                exclude="$exclude --exclude=$subfolder"

                                # stage4 name

                                stage4Name=$(hostname)-stage4-custom-`date +\%d-\%m-\%Y`.tar.bz2

                        fi

                done

        fi

done

if [ "$stage4Name" == "" ]; then

        stage4Name=$(hostname)-stage4-minimal-`date +\%d-\%m-\%Y`.tar.bz2

fi

# mount boot

m1="/bin/mount /boot"

# >/dev/null 2>&1 "

m2="echo \"boot mounted\" "

# where to put the stage4

stage4Location=/mnt/stage4/

# if necessary create the stage4Location

if [ -d "$stage4Location" ] ; then

        $kdedc $dcopRef setLabel "boot mounted ,Backup stage4 to $stage4Location 

   directory $stage4Location already exists

   creating stage4 overview"

   m4="dir $stage4Location"

else

        $kdedc $dcopRef setLabel "boot mounted ,Backup stage4 to $stage4Location

   creating directory $stage4Location

   creating stage4 overview"

        m4="mkdir $stage4Location"

fi

$kdedc $dcopRef setProgress 2

# copy the current world file to the stage4 location

m5="cp /var/lib/portage/world $stage4Location/$(hostname)-stage4-`date +\%d-\%m-\%Y`.txt "

#>/dev/null 2>&1 "

# options for the archive

tarOptions="--exclude=$stage4Location --exclude=/tmp/* --exclude=/var/tmp/* --exclude=/lost+found/* --exclude=/dev/* --exclude=/proc/* --exclude=/mnt/* --exclude=/sys/*--exclude=/usr/portage/* --exclude=/usr/share/smssend $exclude --create --absolute-names --preserve-permissions --bzip2 --totals --ignore-failed-read --verbose --file"

$kdedial --title "Kmkstage4" --yesno "creating tar.bz2 archive with the following options: \

\n tar $tarOptions \n $stage4Location$stage4Name \

\n Do you want to continue?"

answer=$?

# create the archive

$kdedc $dcopRef setLabel "last ones"

$kdedc $dcopRef setProgress 3

if [ $answer = "0" ]; then

   m6="time tar $tarOptions $stage4Location$stage4Name /"

        $kdedial --title "Kmkstage4" --yesno "Do you want to split in CD-size?"

   answer=$?

else

        $kdedial --title "Kmkstage4" --msgbox "Nothing to do.Stage4 not created"

   answer="2"

fi

m7="umount /boot"

#Uncomment the following command

if [ $answer = "0" ]; then

   $kdedc $dcopRef setLabel "Configure Splitting and then go"

   $kdedc $dcopRef setProgress 4

   m8="split --suffix-length=1 --bytes=690m $stage4Location$stage4Name $stage4Location$stage4Name_"

   $kdeforsu -dn --nonewdcop -c "xterm -title Kmkstage4Compression -e \" $m1 ; $m4 ; $m5 ; $m6 ; $m7 ; $m8 \""

   resultsu=$?

   echo "done"

   if [ $resultsu = "0" ]; then

      $kdedc $dcopRef setLabel "splitting is done,enjoy your stage4"

   else

      $kdedc $dcopRef setLabel "without root permission no stage4..."

   fi

   #if you want to split the archive in cd size chunks

elif [ $answer = "1" ]; then

   $kdedc $dcopRef setProgress 4

   $kdeforsu -dn --nonewdcop -c "xterm -title Kmkstage4Compression -e \" $m1 ; $m4 ; $m5 ; $m6 ; $m7 \""

   resultsu=$?

   echo "done"

   if [ $resultsu = "0" ]; then

      $kdedc $dcopRef setLabel "enjoy your stage4"

   else

      $kdedc $dcopRef setLabel "without root permission no stage4..."

   fi   

fi

sleep 2

$kdedc $dcopRef close

```

----------

## schmeggahead

I used this howto to move an existing gentoo installation to raid0 successfully. 

Followed the instructions to the letter to create the tarball, omitting directories, etc.

Set up the raid according to the software raid how to:

https://forums.gentoo.org/viewtopic.php?t=68706

to create the raid.

Then used SystemRescueCD to restore according to these instructions.

Pretty cool.

Only a couple of snags:

1. needed to create proc file system:

```
mkdir -p proc
```

2. forgot to fix fstab to point to the raid drives.

3. forgot to put grub on the mbr of the raid drives. This was challenging because my motherboard considered an extra IDE controller not on the motherboard to be SCSI. Grub numbered them hd2 and hd3 while kernel numbered the hde and hdg. So raidtab worked fine but grub didn't. I also forgot that to put grub in the mbr, you just specify (hd2) instead of (hd2,0).

All in all pretty smooth.   :Cool: 

----------

## BlinkEye

 *schmeggahead wrote:*   

> 1. needed to create proc file system:
> 
> ```
> mkdir -p proc
> ```
> ...

 

if that's true you didn't use my mkstage4.sh, did you? the exlude option

```
--exclude=/proc/*
```

takes care of that problem

----------

## schmeggahead

I know what I did wrong in the manual effort:

```
 --exclude=/proc
```

What a difference a /* makes.

No I didn't use the .sh

I followed the manual steps.

Thanks for the good work.

----------

## JamPS

What I got when I executed BlinkEye's script, my server froze.

Black screen and it dropped all ssh connections... I pressed reset.

I think it had something to do with /dev/pci

----------

## BlinkEye

if you're still going to use it - i just made a few changes/adjustments.

----------

## j-m

BlinkEye, may a suggest an "improvement" to your signature?

```

"grep -v '^#'" /path/to/config/file

```

People will be scared to use the sed command shown there...  :Laughing: 

----------

## BlinkEye

well, i'm scared of it too, that's why it's there so to never forget it - it's not even mine! 

you're suggestion is one i used for quite some time, but it doesn't achieve exactly the same thing. using your grep commands doesn't remove the white spaces or comments like 

```
[whitespace] #foo
```

of course it's a start  :Wink: 

ps: i just noticed that it needs some improvement though  :Shocked: 

----------

## deprave

great post

----------

## Jerri

well, here is my addition... pretty much just a rip off of BlinkEyes post.  but why not.. linux is all about choice right :)

Mostly superficial changes, the thing i didn't like about the original, was clearly defined variables (paths file names, etc.)  other then that, not much different, I cut out stuff i didn't want/need and changed a few things around.  

```
#!/bin/bash

# Full system backup script 

# Usage:

# Just run the script, make any changes or to directories excluded, etc. to the script itself

# 

version=v1.01

# Variable definitions:

# Max archive size / split size

max_file_size=700

split_file_size=695

# Backup Location / file:

backup_location=/mnt/storage/backup/

backup_name=backup-`date +\%d-\%m-\%Y`.tar.bz2

world_file=backup-`date +\%d-\%m-\%Y`.txt

log_file=backup-`date +\%d-\%m-\%Y`.log

# options for the archive

tar_options="--directory / --create --absolute-names --preserve-permissions --totals --bzip2 --ignore-failed-read --verbose --file"

# Excluded directories list (add any extra directories to exclude)

dir_list="

/usr/src

/home/jer/tmp

/home/jer/download

/usr/share/games"

# Default exluded files / directories:

default_exclude="

--exclude=/tmp/* 

--exclude=/var/tmp/* 

--exclude=/lost+found/* 

--exclude=/dev/* 

--exclude=/proc/* 

--exclude=/mnt/* 

--exclude=/sys/* 

--exclude=/usr/portage/* 

--exclude=$backup_location"

# Program prompt

echo -e "\nBackup script $version"

echo -e "===================\n"

# Compile list of excluded folders

for i in $dir_list; do

    exclude_folder="$exclude_folder --exclude=$i"

done

# Display settings: 

#echo -e "Tar options:$tar_options\n"

#echo -e "Excluded folders:\n$exclude_folder $default_exclude\n"

# merge variables:

tar_options="$exclude_folder $default_exclude $tar_options"

# Display tar command:

echo -e "Executing:\n"

echo -e tar $tar_options $backup_location$backup_name "\n"

echo -ne "Do you want to continue? (y/n) "

read answer

# Error checking: 

while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

         echo "please enter y or n"

         read answer

done

if [ "$answer" == 'n' ]; then

    echo -e "\nNothig left to do... Exiting.\n"

    exit 0

fi

# check if file already exists

if [ -a "$backup_location$backup_name" ]; then

        echo -en "\nOverwrite file $backup_name? (y/n) "

        read answer

        while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do

            echo "please enter y or n"

            read answer

        done 

        if [ "$answer" == 'n' ]; then

            exit 0;

        fi

fi

if [ "$answer" == 'y' ]; then

    # mount boot

    mount /boot >/dev/null 2>&1

    # Create archive

    time tar $tar_options $backup_location$backup_name / | tee $backup_location$log_file

    # copy the current world file to the stage4 location

    echo -e "\nCreating backup overview $world_file"

    cp /var/lib/portage/world $backup_location$world_file >/dev/null 2>&1

    # Split if archive greater then max_file_size

    new_file_size=`du -m $backup_location$backup_name | awk '{ print $1 }'`

    if [[ $new_file_size -ge $max_file_size ]]; then

        echo -e "\nSplitting archive..."

        split --suffix-length=1 --bytes="$split_file_size"m --verbose $backup_location$backup_name $backup_location$backup_name

    fi

    # Unmount /boot

    umount /boot

    echo -e "\nBackup is complete.\n"

fi

exit 0

```

----------

## Jerri

[Edit]: changed the script around a bit after testing.  I ran into problems when trying to restore a system, without including /dev in the tar ball.  so i included that, i also added a section to create an iso image of the server tar ball.  I just copied the contents of the livecd, then add the server tar ball, and burn it all on one disk.  works pretty well, I'm working on a restore script, and have been contemplating ripping apart the livecd, so all you have to do is insert the disk, and it will automatically reinstall your whole system..  a sort of bare metal system restor...  we shall see [/edit]

this time, I tailored the script to run on a remote computer, (in my case a server operating in a DMZ)

it will make a backup of the entire system, then transfer the files to another computer (preferable one that is more secure).  I suppose its not really necessary to have periodic, full backups... but then, I have so few services running on my server, that the archive size is around ~150 megs.  

The script comes in two parts:

first part is the actual backup script which will be automatically uploaded to your server.

the second part, is the script that runs the whole deal.

If you want to use it, you will have to change the variables appropriately.

Part 1: (I called this file  backup.server.script)

```
#!/bin/bash

# Full system backup script 

# (Server Side) 

# Variable definitions:

# Backup Location / file:

backup_location=/mnt/backup/   

backup_name=backup-`date +\%d-\%m-\%Y`.tar.bz2  

world_file=backup-`date +\%d-\%m-\%Y`.txt

log_file=backup-`date +\%d-\%m-\%Y`.log

# options for the archive

tar_options="--directory / --create --absolute-names --preserve-permissions --totals --bzip2 --ignore-failed-read --verbose --file"

# Excluded folders list:

list=""

# Default exluded file/folders:

default_exclude="

--exclude=/tmp/* 

--exclude=/var/tmp/* 

--exclude=/lost+found/* 

--exclude=/proc/* 

--exclude=/mnt/* 

--exclude=/sys/* 

--exclude=/usr/portage/* 

--exclude=$backup_location"

# Compile list of excluded folders

for folder in $list; do

    exclude_folder="$exclude_folder --exclude=$folder"

done

      

# merge variables:

tar_options="$exclude_folder $default_exclude $tar_options"

# mount boot

mount /boot >/dev/null 2>&1

   

# Create archive

echo -e "Backing up system...\n"

time tar $tar_options $backup_location$backup_name / > $backup_location$log_file 2>&1

# copy the current world file to the stage4 location

cp /var/lib/portage/world $backup_location$world_file >/dev/null 2>&1

   

# Unmount /boot

umount /boot

echo -e "Backup complete."

exit 0
```

Part 2: (run.backup.script)

```
#!/bin/bash

#

# Backup utlity

#client side

storage_location=/mnt/storage/backup/server/

script_local_location=/root/scripts/

restore_script01=restore-sys

restore_script02=restore-sys-cont

# Server side

server_ip=192.168.3.2

script_remote_location=/root/scripts/

backup_location=/mnt/backup/

# File names

script_name=backup.server.script

backup_name=backup-`date +\%d-\%m-\%Y`.tar.bz2

world_file=backup-`date +\%d-\%m-\%Y`.txt

log_file=backup-`date +\%d-\%m-\%Y`.log

temp_iso=/tmp/newgentoo.iso

# upload backup script to remote computer

scp $script_local_location$script_name root@$server_ip:$script_remote_location$script_name

# run backup script on remote computer

ssh -l root $server_ip  sh $script_remote_location$script_name

# retreive archives

scp root@$server_ip:$backup_location$backup_name  $storage_location$backup_name

scp root@$server_ip:$backup_location$world_file   $storage_location$world_file

scp root@$server_ip:$backup_location$log_file     $storage_location$log_file

# remove backup script from remote computer

ssh -l root $server_ip rm -f "$backup_location"*

# remove backup archives from remote computer

ssh -l root $server_ip rm -f "$script_remote_location$script_name"

#

# Burn archive

#

# Blank CD:

cdrecord dev=ATAPI:0,0,0 blank=fast gracetime=2 > /dev/null 

# Burn archive to livecd:

cp -f $script_local_location$restore_script01 $storage_location

cp -f $script_local_location$restore_script02 $storage_location

cd $storage_location

cd ..

echo -e "Creating ISO image...\n"

mkisofs -quiet -no-emul-boot -boot-load-size 4 -boot-info-table -r -b isolinux/isolinux.bin -c isolinux/boot.catalog -o $temp_iso . > /dev/null

echo -e "Burning image to cd...\n"

cdrecord fs=8m -dao -data dev=ATAPI:0,0,0 speed=10 driveropts=burnfree -overburn gracetime=2 $temp_iso > /dev/null

rm -f $temp_iso

echo -e "\nBack up complete.\n"

exit 0
```

There is a catch to all this, you have to set up public key authentication, so that you can loggin to your server without supplying a password.   for more information about this have a look at these articles:

gentoo keychain guide

OpenSSH key management

hereLast edited by Jerri on Sun Mar 06, 2005 7:55 pm; edited 1 time in total

----------

## BlinkEye

it was about time i rewrote the script. i did a lot of changes. as usual it can be found here

i liked the version print upon calling the script - thanks Jerry!

----------

## Jerri

 *Quote:*   

> the record was 7CD's for a full installation

 

dear me, thats brutal...

----------

## BlinkEye

 *Jerri wrote:*   

>  *Quote:*   the record was 7CD's for a full installation 
> 
> dear me, thats brutal...

 

i dare not to say it: it didn't work afterwards. the big joke was: 7CD's, but NOT bootable. i first had to setup windows ...

----------

## thomash

Hi. I successfully created a stage4 with your script blinkeye.

This is exactly what I've been looking for =)

But I run into some problems when trying to boot from an USB hd.

This is what I did:

---

mkdir /mnt/newsys

mount /dev/sda3 /mnt/newsys

mkdir /mnt/newsys/boot

mount /dev/sda1 /mnt/newsys/boot

cd /mnt/newsys

tar xvjpf ../foo-stage4-19.03.2005-minimal.tar.bz2

---

And I modified my /mnt/newsys/boot/grub/grub.conf to look like this:

---

default 0

timeout 30

splashimage=(hd1,0)/boot/grub/splash.xpm.gz

title=development-sources 2.6.8.1

root (hd1,0)

kernel (hd1,0)/boot/kernel-2.6.8.1 root=/dev/sda3

---

But when powering on my system i get "Operation system not found."

Am i missing something? (also tried (hd0,0) too although this is proboably wrong =)

----------

## BlinkEye

i must admit that everytime i install a stage4 to my external hd i screw up both system (the one on the usb drive and the one on the laptop). somehow i get them both back, so, i know what you are talking about. for me it's always the bootloader, and if that works i may still have a wrong /etc/fstab. at the moment i'm not sure where your problem lies, but there are several things you need to check out:

1. check again your path - is your untar-ed stage4 really on /mnt/newsys/ ?

*  just a note: creating /mnt/newsys/boot should not be necessary

2. what's in your /mnt/newsys/boot directory? sounds like nothing's in there

3. try to install grub again on /dev/sda

4. what exactly happens after booting your system? does grub show up?

5. you may need to append "rootdelay=5" to your kernel line (without the quotes)

6. are you using udev/devfs?

7. what's your /mnt/newsys/etc/fstab look like?

...

----------

## thomash

Hi. the splashscreen doesn't show at all.

The stage4 is really unpacked to /mnt/newsys/ wich is /dev/sda3 and the kernels+grub is in /mnt/newsys/boot wich is /dev/sda1

You say I don't have to create the /mnt/newsys/boot dir, but how can I then specify that /mnt/newsys/boot should be /dev/sda1 before unpacking?

I read something about creating a new initrd though, not sure if this is necessary or not.

I'll check into it atleast.. how do you manage to screw up your current system in your laptop btw?

And I think i'm using devfs hehe.

----------

## BlinkEye

 *thomash wrote:*   

> Hi. the splashscreen doesn't show at all.

 

well, this means that grub isn't installed (properly).

 *thomash wrote:*   

> The stage4 is really unpacked to /mnt/newsys/ wich is /dev/sda3 and the kernels+grub is in /mnt/newsys/boot wich is /dev/sda1

  alright

 *thomash wrote:*   

> You say I don't have to create the /mnt/newsys/boot dir, but how can I then specify that /mnt/newsys/boot should be /dev/sda1 before unpacking?

 sure, forget my comment  :Embarassed: 

 *thomash wrote:*   

> I read something about creating a new initrd though, not sure if this is necessary or not.

 no, this is not at all necessary. if no splashscreen shows, you don't have grub installed (properly). you did install grub explicitly on /dev/sda, did you? 

```
mount -t proc  none /mnt/newsys/proc 

mount --bind /dev /mnt/newsys/dev (might be wrong, check the gentoo doc)

chroot /mnt/newsys/ /bin/bash

env-update

grub

... (installing grub on /dev/sda - see the gentoo doc)
```

 *thomash wrote:*   

> I'll check into it atleast.. how do you manage to screw up your current system in your laptop btw?

 well, somehow if i install grub on the external hd it isn't installed on the laptop anymore (don't know why). as i do not have any disk for my laptop it's quite a problem if i can't boot from it anymore, that's why i do have the external usb hd.

 *thomash wrote:*   

> And I think i'm using devfs hehe.

 in fact shouldn't matter at all. just keep in mind my boot parameter if it won't be able to mount root

----------

## gungholady

Using the script created by BlinkEye, the split command does not work correctly. Here is the error output:

```
Total bytes written: 3584931840 (3.4GiB, 503KiB/s)

real    116m3.419s

user    79m6.556s

sys     2m28.013s

* creating stage4 overview /home/mnt/tarbackup/gungholady-stage4-22.03.2005-custom.txt

* stage4 is done

* umounting boot

split: /home/mnt/tarbackup/home/mnt/tarbackup/gungholady-stage4-22.03.2005-custom: No such file or directory
```

The /home/mnt/tarbackup/gungholady-stage4-22.03.2005-custom.tar.bz2 is being correctly created. The error is is the split part of the script. I can't figure out why it is looking for the bz2 file in /mnt/home/tarbackup/home/mnt/tarbackup instead of /home/mnt/tarbackup where it is located.

----------

## BlinkEye

 *gungholady wrote:*   

> Using the script created by BlinkEye, the split command does not work correctly. Here is the error output:
> 
> ```
> Total bytes written: 3584931840 (3.4GiB, 503KiB/s)
> 
> ...

 

thanks a lot for your input. i'm sorry for the inconvenience, i forgot to adjust the split command. it should be

```
split --suffix-length=1 --bytes=670m $stage4Name.tar.bz2 "$stage4Name"tar.bz2_ && echo "* splitting is done" 
```

the thing is that after the user decided, if he wants to do a minimal or custom backup $stage4Name gets reassigned with a the full path (but without the extension, "tar.bz2" must be added too.

----------

## gungholady

Thanks. It worked correctly this time. I had tried everything I could think of.

----------

## Maedhros

This thread seems now to be a discussion of BlinkEye's script, so please follow up to here: https://forums.gentoo.org/viewtopic-t-312817.html.

----------

