# two way sync with rsync

## soldstatic

And don't say "unison" please!   :Wink: 

If anyone can help me out, I'd greatly appreciate it.

I am using a NAS box in my house (western digital world book something or other) in my basement, a desktop vista PC, an ARM internet tablet (n770 from nokia), a laptop with vista, a shared host web server, a win xp machine at work, and a coworkers' vista desktop. I want to use my NAS box as a central file synchronization storage thing.

My goal is to have all of my home PCs and the 770 sync to the NAS box, my coworker's machine sync to the webserver, and nightly have the webserver sync with the NAS box. That way everybody will always have the same files. (then later, and I already found some great examples, have rsync create multiple snapshot backups on the NAS box so if something gets messed up I'm not in much trouble).

rsync is great at one way copies, and I know how to do that, but I could use some help creating the perfect rsync command to make a two way sync.

Problem is deletions. If I do a one way sync with --delete and -u, and then do the same sync but switch source / dest around, I will never get brand new files from the first destination on the new machine.

Script I've set up so far:

```

rsync -avz --delete local remote:dir/

rsync -avz --delete remote:dir local

```

granted i throw in the ssh args, but lets ignore that for now.

I know I'm not the first to have this situation, but most threads I read have a bunch of commenters come up and say "use unison". But I don't have unison on my 770, and i don't have it on my webserver, and I know that rsync is tried and true. Also if I try to just do my own local install (assume also I don't run rsync as root from any machine), I know I'll screw it up, and I also just don't really want to install anything else. With SYNC in the name, I'm sure there's a way to make it work!!

Thank you for your time in advance!

 :Confused:  . o O (if I can just get all of my stuff synced everywhere, I be too worried about hard drive failures, then if I can get backups and history of everything, I won't have to worry about accidentally deleting anything!)

----------

## Mr. M

Well, AFAIK rsync has newer been intended for 2 way synchronization, so you should be using unison   :Wink:  . However, I think even if you are using unison for the scenario you described you are likely to get into trouble, as it is likely that you and your co-worker change the same file at some point and you may lose your work accidentaly. I use unison to synchronize multiple computers im working from, but I always make sure to synchronize when I being working and at the end, like this the central repository is newer out of sync. Difficult to do with more than one person.

What I recommend is using a version control system, like subversion or git so that you and your co-worker can work on the same stuff. I'm not sure if you can use ssh+svn (i.e. subversion over ssh) in combination with your NAS box for this, you should investigate this option. You can then sync the web server by checking out the latest version and then use rsync (assuming the files on the webserver don't change).

----------

## soldstatic

hahahaha

Well, I don't want to do subversion because my coworker is a 70 year old man and I don't want to have to install subversion and then explain to him how it works. Besides, we will very rarely (in fact basically never) be working on the same files. For the most part, he acts as an accountant and researcher putting together docs. The sync would streamline our business cause I would get the docs so that I can read them later (i rarely make changes on them, just use them for reference). So I'm not too worried about us both editing one file. I'm worried mostly about when one of us moves or deletes somethings. Say we move a folder from one location to another, depending on the RSYNC we'd get that folder back doing it from the out-of-sync PC and then doing the inverse sync we'd get a copy of that folder in a new location (the destination of the move).

So I'm not worried about losing edits as i would be if we were working on code or something, plus I don't want to have to do svn checkouts from any of the PCs or the tablet, svn always lent itself to getting a new version of one file and a far too old version of another one with me, and it would probably screw us up somehow.

I'm using SyncToy by microsoft on my desktop and laptop to sync with the NAS on some samba shares, it might be able to work over FTP I don't know. Maybe try mapping an FTP drive from the coworker's location. Then I will also set him up on more of his PCs with synctoy as well. But I still have the problem of syncing the contents of the NAS box and the webserver nightly.

So my checklist of syncs:

laptop <-> NAS  (synctoy) X

desktop <-> NAS (synctoy) X

coworker <-> webserver (synctoy) ?map ftp drive?

corworker <-> other company PCs (synctoy)

770 <-> NAS (? rsync / unison / svn ?) ?

NAS <-> webserver (? rsync / unison / svn ?) ?

Once I get everything syncing with the NAS box correctly including the webserver, I will probably just use rsync to do the backups. I found a great example that explained it, and I will keep a few 10min backups, a few hourly, a few daily, a few monthly, and one yearly for the last 3 years. Sounds like it's going to take a ton of space, and we do have a huge amount of data, but we really are only adding to the data instead of changing stuff, so I don't think it will actually end up taking up an inordinant amount of space.

----------

## Mr. M

For the rsync backups, look at the link-dest option, as described here http://www.mikerubel.org/computers/rsync_snapshots/. We use this to backup a 600GB RAID array to a 1TB disk nightly, as long as not too many things change we can keep backups for one week.

----------

## soldstatic

yea that's exactly what I plan to do.

So, nobody else has any suggestions to make rsync work for a two way sync??

----------

## soldstatic

well sadly i tried to install unison, but the package manager for the western digital my book world edition (linux based box I want to run everything off of) doesn't have it. So i dunno. Guess I need to compile it by hand or something?

----------

## flybynite

It's hard to tell exactly what you need but do look at rdiff-backup.

http://www.nongnu.org/rdiff-backup/features.html

It uses rsync to make an (almost) exact copy of a dir on a remote machine.  The part that makes it interesting is it keeps diff's to make it possible to roll back changes to the dir like a version control system would.  It uses rsync to only send the changes across the network.  You would only need to copy, mount or rsync the dir to other boxes skipping one hidden dir that contains the version history info.

It can easily work over ssh and transfer from arbitrary boxes, like from box a you can backup box b to box c.  Restoring would be just as easy.

Its a python script that calls librsync so it it very portable.

----------

## soldstatic

ok i'm looking at rsnapshot project. Looks very promising. And the config file helped me set it up so that rsnapshot will run these commands for me:

```
[root@AndyBook BACKUP]# rsnapshot -t hourly

echo 6482 > /var/run/rsnapshot.pid

/shares/internal/BACKUP/expor.sh

mkdir -m 0755 -p /shares/internal/BACKUP/hourly.0/

/opt/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \

    /shares/internal/MUSIC /shares/internal/BACKUP/hourly.0/.

touch /shares/internal/BACKUP/hourly.0/

```

However, this is creating directory structure i don't want. I'd like to see /shares/internal/BACKUP/MUSIC/something.mp3 but instead I'ms eeing /shares/internal/BACKUP/MUSIC/shares/internal/BACKUP/MUSIC/something.mp3.

Any ideas??? What command would I need to run to get it to look right??

There are a couple of options in the rsnapshot config file, namely snapshot root and then the arguments you set up for your particular backup directories, just can't figure out the right combination of these I think. But I've tried almost everything.

Thanks in advance! -Andy

----------

## ppurka

Try to see whether there is an option in rsnapshot to turn off the -R flag of rsync. That is the argument required to provide to rsync to create such (relative) directory structures.

If you can't turn it off, try to see if you can provide --no-R as an option.

----------

## soldstatic

brilliant that worked perfectly. 

Thank you!

----------

## soldstatic

OK When I first ssh in and run rsnapshot, I get an error that says 

 *Quote:*   

> /opt/bin/perl: can't load library 'libperl.so'

 

I know that perl is installed via the world book package manager (ipkg) which is located somewhere like /opt/bin/perl but that there is also the default installation that is seperate from the ipkg version at /bin/perl

ANy ideas? I can run 

 *Quote:*   

> export LD_LIBRARY_PATH=/opt/lib:/opt/lib/perl5/5.8.8/arm-linux/CORE

 

and it will work for the rest of that ssh session, but that isn't going to help when I try to put this in crontab...  :Sad: 

----------

