# rsync everything except hardware dependent stuff [SOLVED]

## dan___

Hello!

I want to rsync two systems almost completely.

(I don't want to emerge every package twice.)

The two machines have different hardware. So I need to exclude every 

file and directory which is hardware dependent.

Here is what I can think of. Please tell me what else I should exclude.

```
boot

etc/udev

etc/X11

etc/conf.d/net

etc/init.d

etc/fstab

etc/mtab

etc/modules.autoload.d

etc/modules.conf

etc/modules.d

etc/modprobe.d

etc/modprobe.conf

etc/modprobe.devfs

sys

usr/src

```

I don't know if I also need to exclude dev and proc?

When it works I will try to write a How-To about it. 

So please send comments and suggestions!

Cheers

DanLast edited by dan___ on Sat Apr 14, 2007 9:00 am; edited 1 time in total

----------

## SoylentGreen

 *dan___ wrote:*   

> 
> 
> (I don't want to emerge every package twice.)
> 
> 

 

just btw.. you could create a tbz2file. from "man emerge"

```

tbz2file

              A tbz2file must be a valid .tbz2 created with ebuild <package>-<version>.ebuild package or emerge --buildpkg [cat-

              egory/]<package> or quickpkg /var/db/pkg/<category>/<package>.

```

if the cpu architecture of both machines is identical, that should work?

----------

## jstead1

I usually do something like this when I am not using the system to be copied, such as booting into a different linux on the same machine, or using a knoppix cd.  When you do it this way, you don't need to worry about /dev and /proc (or any of the other differences between a system that is running and one that is shut down).

----------

## dan___

Hello!

 *jstead1 wrote:*   

> booting into a different linux on the same machine, or using a knoppix cd.

 

That's what I am doing, too. I use a Knoppix CD for the rsync. 

But I still have to worry about the config files, the kernel, the kernel modules etc.

How do you handle these things?

Best,

Dan

----------

## dan___

Hello SoylentGreen!

 *SoylentGreen wrote:*   

> just btw.. you could create a tbz2file.
> 
> 

 

If I understand the manpage right, this would save me from compiling everything twice. 

But I still would have to emerge and configure everything on both systems. 

I would prefer to figure out once which parts I need to keep (because they are hardware dependent) 

and then sync the whole rest. I would like to sync everything: software, configurations, data ... 

 *SoylentGreen wrote:*   

> if the cpu architecture of both machines is identical, that should work?
> 
> 

 

Actually they are not. Therefore I use just a generic i686 architecture in my make.conf:

```
CHOST="i686-pc-linux-gnu"

```

Greetings

Dan

----------

## SoylentGreen

 *dan___ wrote:*   

> 
> 
>  *SoylentGreen wrote:*   just btw.. you could create a tbz2file.
> 
>  
> ...

 

well, the "emerge" of an tbz2 file just takes seconds. likewise emerging a bin.

 *SoylentGreen wrote:*   

> if the cpu architecture of both machines is identical, that should work?
> 
> 

 

Actually they are not. Therefore I use just a generic i686 architecture in my make.conf:

```
CHOST="i686-pc-linux-gnu"

```

in this case, a tbz2 should work.

anyway, it was just a suggestion / hint.

no solution is perfect  :Wink: 

----------

## dan___

Hi Guys!

After some trial and error I am now excluding the following files and dirs:

```
etc/udev

etc/X11

etc/conf.d/net

etc/init.d

etc/fstab

etc/mtab

etc/modules.autoload.d

etc/modules.conf

etc/modules.d

etc/modprobe.d

etc/modprobe.conf

etc/modprobe.devfs

sys

usr/src

```

Only the directory /lib remains a problem. 

When I sync it I loose my network interfaces. Or actually: they get moved from eth0 and eth1 to eth2 and eth3. 

I guess eth0 and eth1 get taken by the modules (?) for the eth-cards in my other system. 

These cards don't exist in the second system and therefore fail to start. 

The existing cards are now eth2 and eth3 for which there is no config. So they fail as well. 

Any idea how to fix that?

Best,

Dan

PS: Not syncing /lib is not an option. When I don't sync it, the system breaks.

dbus won't start because it can't find the shared libraries ...

----------

## dan___

Argh! Stupid me!

Of course I don't have to sync the directory /lib.

```
cd /usr/src/linux

make clean

make && make modules_install

```

As far as I could see from the boot messages everything seems to work fine now.

But I'm going to check a few more things ... 

Best,

Dan

----------

## dan___

Ok, there was one more thing: libproc.so somehow got lost or broken. 

It couldn't be loaded during boot.

Fortunately this could be fixed be re-emerging sys-process/procps

(Thanks to dice for pointing out the right package.)

```
emerge --oneshot procps
```

Now everything seems to work.

----------

## dan___

And here comes the summary:

How-To-Sync-Everything-Except-The-Hardware-Dependent-Stuff

The aim of the whole thing is to syncronize two systems so that only one of them has 

to be maintained. For example you could emerge and configure a dozen packages on your first

machine and then just sync the whole system to you second machine ... But since your second 

machine will most likely have different hardware (e.g. could be you laptop) you don't want to sync 

the hardware drivers and configs and the kernel. But you should use the same kernel version on 

both machines. And set the architecture in your make.conf so that both machines will run with it.

(If both machines got different CPU types use something generic like CHOST="i686-pc-linux-gnu".)

Warning:

This is crazy stuff. It might break your system. This how-to works fine for me and my system but it 

might mess up yours. So make sure that you make a backup. Best thing is to create a image of your 

complete harddisk. I use partimage for that. It's a good tool and comes on every Knoppix CD.

Make a backup. 

Make it now. 

If you don't make a backup, don't complain if your system dies, if you loose all your data, or if 

your girlfriend breaks up with you because you're a nerd. You have been warned.

Assume you want to sync from System1 to System2.

As jstead1 said, it's best to sync systems when they are not running.

So boot both systems from a boot CD (e.g. Knoppix). 

Mount all the necessary partitions and make sure they are writable on System2.

I assume you only got one partition and you mount it to /mnt/hda1 on both systems. 

Set up the network connection so that you can ssh from one machine into the other.

Should be easy. If not ask the wwweb.

From now on I'll work on System2. It's also possible to work on System1,

you just have to turn around the rsync command.

Create a file with all the directories which you _don't_ want to sync.

```
emacs exceptions
```

In that file write these directories:

```
boot

etc/udev

etc/X11

etc/conf.d/net

etc/init.d

etc/fstab

etc/mtab

etc/modules.autoload.d

etc/modules.conf

etc/modules.d

etc/modprobe.d

etc/modprobe.conf

etc/modprobe.devfs

lib

sys

usr/src 

```

Now it's time to sync the two systems:

```
rsync -aHPv --delete --except-from=exceptions root@System1:/mnt/hda1/ /mnt/hda1/
```

Clear how it works? If you wanna know what -aHPv means, check the rsync manpage.

Make sure you got the slash / in the end of both paths otherwise rsync is going to be a bitch.

rsync will ask for your passwd and than it will start copying files. This can take some time. Get a beer or a coffee ...

rsync is finished. Congratulations, you successfully broke System2 now. But nevermind, we're going to fix it next.

Reboot System2. The kernel will still work. We didn't even touch it. (Remember excluding /boot and /lib)

So your system should boot but it will throw some error messages. These come from are packages which 

can't find the necessary shared libraries because they got compiled against a different kernel and different 

libraries and we did not sync the directory /lib.

We will now create these libraries by recompiling the kernel. First we have to make sure to remove all 

the old libraries:

```
cd /usr/src/linux

mv .config myconfig

make clean mrproper

mv myconfig .config

```

Now compile the kernel and the modules

```
make && make modules_install
```

You don't have to copy the kernel to /boot because the kernel was ok anyway. We just needed the modules and 

shared libraries which should now be alright, too.

So reboot and see what else is broken. On my system libproc.so couldn't be found. No idea what happened to the 

poor thing but let's get it back:

```
emerge --oneshot sys-process/procps
```

Run repdev-rebuild just to make sure nothing is broken which is not also broken on System1.

```
revdep-rebuild -p
```

Now your system should be successfully syncronized and running. 

Have a nice day!

Dan

PS: Actually it might be enough to exclude /usr/src/linux/.config instead of the whole directory /usr/src. 

Then you might also be able to update your kernel version with the method above. As long as the new kernel 

version still likes the old config file. Which it normally doesn't ...

----------

## dan___

Hmmm, it worked alright a couple of times but yesterday it screwed up my portage on 

the second system. I'll investigate and update the how-to if necessary.

Best,

Dan

----------

## SoylentGreen

thats what i guessed would happen, but didnt want to spoil yours show  :Wink: 

----------

## dan___

Sorry for the confusion. (And sorry if it disappoints you, SoylentGreen .  :Wink:  )

The synchronising worked but unfortunately I messed up portage on the old 

system before I syncronised. So when I syncronised that problem got copied 

to the second system.

(The problem was that I installed TeXLive manually, deinstalled teTeX and 

forgot about the kpathsea library. So I got problems with dependencies etc. 

Fortunately it was easy to fix.)

@SoylentGreen: 

I'm interested in your opinion anyway. Why do you think the sync-ing

could mess up portage? I think in the worst case it would need a 

emerge --sync, a package autocleaning and maybe a revdep-rebuild.

But even these things should be unnecessary since I am sync-ing 

the whole system including portage.

Or am I completely wrong there? If you know anything which I miss 

I would be happy if you share it   :Very Happy: 

Best,

Dan

----------

## SoylentGreen

well, already told ya: "you could create a tbz2file"

----------

## randal1

Hi All,

I have 2 external HD's connected to different laptops.  I wish to mirror the contents of one with the other.

Currently, I am using rsync to do this, with the following command:

```
rsync --progress --ignore-existing -rv /home/HD1 backup@192.168.1.50:/home/HD2
```

this works, but ideally I would like any files that have been deleted from the first to be deleted from the 2nd, which I believe i can achieve with the "--delete" command.  But I would also like to update any files that have been modified from HD1.  Currently, only new files are copied across to the other HD, not existing files that have been updated.  I have had a look at the MAN pages, but cant seem to find what I am after.

Any ideas on the command I need to achieve this?

----------

## nixnut

merged above post here.

Look up

----------

## dan___

Hi randal1,

Sounds as if you are looking for 

```

rsync -aHP --delete 

```

The params:

-a  = -rlDtgop 

-H

-P

recursive; copy symlinks, hardlinks and devices as they are,

preserve owner, group, permissions and timestamp; show progress.

Why did you use --ignore-existing if you want existing files to be updated?

If you want to do more fine-grained syncing back and forth take a look 

at Unison which is a GUI for rsync. 

Best,

Dan

----------

## randal1

thanks for the response.

I tried your suggested command on a small test directory.  Both directories had identical files, but the 2nd HD's files had newer timestamps.  Rsync copied all the files over, but the timestamps of the 2nd HD stayed newer than the ones on the first.  When you say "preserve timestamp", do you mean the original or the updated?

Ideally I would like to have all of the timestamps of the 1st HD copied over, overwriting any newer timestamps of the 2nd HD.  Is this possible?

Many thanks  :Smile: 

----------

## dan___

You want to replace _newer_ timestamps? So if a file got changed on HD2 

you want to replace it with an older version from HD1?  

Actually that should be the default behavior of rsync. Did you use 

the parameter -u (--update skip files that are newer on the receiver) 

by accident?

Or maybe -a includes -u. Try the full -rlDtgop instead of -a.

```

rsync -rlDtgopHP 

```

Dan

----------

## dan___

 *randal1 wrote:*   

> When you say "preserve timestamp", do you mean the original or the updated?
> 
> 

 

Yeah, I mean sender overwrites receiver. 

-a stands for archive. After the rsync -a --delete HD1 HD2

the receiver (HD2) should look exactly like the sender (HD1).

----------

## randal1

Yep, sure I'm not using "-u". I tried your suggested command, and it says it wants to copy no files from the folder in HD1 to HD2, which is strange because I have compared the timestamps of the same file in each and they are different.

Do you think the fact that the timestamp is newer on the destination could be causing this trouble?

Many thanks for your continued help!

----------

## dan___

Are the files actually different or are the only difference the timestamps?

If the file content is the same rsync won't do anything even with different timestamps. 

But if the files are different it should copy them from HD1 to HD2. The file on HD2 will be overwritten 

and the copied file will have the same timestamp as it had on HD1.

Try editing one of the files and see what happens!

Dan

----------

## randal1

Ok, I've been doing some even simpler tests to try and determine what is happening, and this is what I know so far...

I have two directories with identical content on HD1 and HD2.  I deleted a file "dsc00200.jpg" on HD2 (I have a backup elsewhere!), and then ran the command:

 *Quote:*   

> 
> 
> rsync -av --dry-run "/home/aaron/HD1/My Documents/Backup/Phone/dcim/100msdcf/" backup@192.168.1.50:"/home/aaron/HD2/My Documents/Backup/Phone/dcim/100msdcf"

 

using "--dry-run" so that the command tells me what files would be transferred but doesn't actually execute.  I would expect the output to be (at least) the single file "dsc00200.jpg" (if we ignore the timestamp issue at the moment), as this is in HD1's dir but not in HD2's.  But what I am actually seeing is no files want to be copied...

```
building file list ... done

sent 10042 bytes  received 20 bytes  101.13 bytes/sec

total size is 174563425  speedup is 17348.78
```

Can you spot what I am doing wrong?

Many thanks

----------

## dan___

 *Quote:*   

> Can you spot what I am doing wrong? 

 

The only thing I can see is a missing / in the end of the second path. 

rsync is rather picky about the / in the end: either put one in both 

paths or in none of them!

I just tried the the same on my machine and it worked fine. 

```
> mkdir HD1

> mkdir HD2

> cp pic1.jpg HD1/

> cp pic2.jpg HD1/

> rsync -av HD1/ HD2/

building file list ... done

./

pic1.jpg

pic2.jpg

sent 1733505 bytes  received 70 bytes  3467150.00 bytes/sec

total size is 1733078  speedup is 1.00

> rm HD2/pic2.jpg

> rsync -av HD1/ HD2/

building file list ... done

./

pic2.jpg

sent 860890 bytes  received 48 bytes  1721876.00 bytes/sec

total size is 1733078  speedup is 2.01
```

Don't know why it doesn't work for you. Sorry.

----------

## Monkeh

Uh, why not just rsync everything and modify the config files as needed? You'll be rewriting them anyway..

----------

## dan___

 *Monkeh wrote:*   

> Uh, why not just rsync everything and modify the config files as needed? You'll be rewriting them anyway..

 

Yeah. The thread above is rather old. You're right: it's easiest to rsync everything and change the config files afterwards.

Then you can store them in a safe place, and next time you rsync everything you can copy them from this save place or 

modify them again, depending on the changes of the system in the meantime. 

By now I changed to a different approach. I started a fresh install on the machines and used tbz2 files from 

the beginning as SoylentGreen suggested from the very beginning.

The thread just became active again since randal1 asked a question about rsync and an admin merged his thread 

into this one. I don't mind. Unfortunately I couldn't reproduce randal1's problems on my machines yet. No idea why 

rsync isn't working on his setup.

Dan

----------

## randal1

Ah this is driving me crazy!

So as mentioned in my previous post, if I remove a file on HD2 and attempt a resync using the command:

```
rsync -av --dry-run "/home/aaron/HD1/My Documents/Backup/Phone/dcim/100msdcf/" backup@192.168.1.50:"/home/aaron/HD2/My Documents/Backup/Phone/dcim/100msdcf"
```

It wants to copy nothing.  Just to check that the file definitely exists on one and not the other...

On HD1:

```
file dsc00200.jpg

dsc00200.jpg: JPEG image data, EXIF standard 2.2
```

on HD2:

```
file dsc00200.jpg

dsc00200.jpg: ERROR: cannot open 'dsc00200.jpg' (No such file or directory)
```

Now if I add a new file called "dsc00200NEW.jpg" and re-run the command, I see that it DOES want to sync that file...

```
building file list ... done

./

dsc00200NEW.jpg

sent 10048 bytes  received 32 bytes  300.90 bytes/sec

total size is 174563425  speedup is 17317.80
```

So it appears to be half working!

Are there any config/history files that I could try purging? Maybe rsync has some corrupted view of this directory?  Clutching at straws really...

Many thanks  :Smile: 

P.S. with regards to the trailing slash, I think that the method I am using is correct (but please correct me if I have interpreted it wrong  :Smile:  )... from the Rsync MAN page...

 *Quote:*   

> A  trailing  slash  on the source changes this behavior to avoid creating an additional
> 
>        directory level at the destination.  You can think of a trailing / on a source as mean‐
> 
>        ing  “copy  the contents of this directory” as opposed to “copy the directory by name”,
> ...

 

----------

## dan___

Hi randal1

You're right with the trailing / although a slash on both paths works fine, too.

I' using rsync -av /src/foo/ /dest/foo/ all the time and it does what I want. 

Honestly, I have no idea why it doesn't work on your machine.

Maybe it got to do with the username under which you log in to 

the second machine? Did you try the same thing with local directories?

 *Quote:*   

> Are there any config/history files that I could try purging? Maybe rsync has some corrupted view of this directory? Clutching at straws really...
> 
> 

 

I don't know what rsync is actually checking. It might look in some database 

or journal. But whatever that is, it should know when you deleted something.

Check if the two systems are on the same system time. It might cause

strange effects if one system is some hours ahead of the other.

Dan

----------

## randal1

Hi Dan,

First off, thank you so much for your continued support!  I really appreciate you making the effort...

Last night I decided enough was enough and simple synced both drives using the "rsync -aHP" command (thanks Monkeh).  It copied every file again (didn't take too long when both drives were connected locally, and I did it through the night anyway!) and now this morning it is working perfectly.  If I modify a file, only that file gets updated across the network, and If i delete a file on HD2, then that file gets re-synced.

Again, many thanks  :Smile: 

----------

