# HOWTO:Local Rsync Mirror

## flybynite

If you have a couple of gentoo boxes on a LAN be a good gentoo netizen and setup a local rsync mirror.  You can sync the server box across the net and then sync all your other machines faster across the LAN.  This not only allows gentoo to grow faster by not abusing mirrors, but your LAN is way faster than your internet connection!!

I recommend you also set up a local package cache for downloaded package files, ebuild and HOWTO is here  :Smile:  https://forums.gentoo.org/viewtopic.php?t=173226

Let me start by saying there is conflicting info in the forums and gentoo docs about how to setup a local rsync mirror.  The howto at http://www.gentoo.org/doc/en/rsync.xml is for OFFICIAL public mirrors!  It can confuse a user wanting a LOCAL LAN mirror!  There is an app-admin/gentoo-rsync-mirror ebuild that is currently outdated and possibly insecure!!  If you think that ebuild has been updated and the problems fixed you should check the difference between it and this howto to see if it has been updated to include all fixes and features!

You really only need a config file to get up and running, but this version adds features and security.

Every gentoo box has rsync installed because it's part of the base system.  Lets add the parts that make it a server.

1.  /etc/rsyncd.conf

```

uid = nobody

gid = nobody

use chroot = yes

#limit access to private LAN's

hosts allow=192.168.0.0/255.255.0.0 10.0.0.0/255.0.0.0

hosts deny=*

max connections = 15

pid file = /var/run/rsyncd.pid

motd file = /etc/rsync/rsyncd.motd

#This will give you a separate log file

#log file = /var/log/rsync.log

#This will log every file transferred - up to 85,000+ per user, per sync

#transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

#If you need this, UPGRADE portage please!

#[gentoo-x86-portage]

#this entry is for compatibility

#path = /usr/portage

#comment = Gentoo Linux Portage tree

#exclude=distfiles/

[gentoo-portage]

#modern versions of portage use this entry

path = /usr/portage

comment = Gentoo Linux Portage tree mirror

exclude = distfiles/

```

2.  /etc/init.d/rsyncd

```

#!/sbin/runscript

# Copyright 1999-2002 Gentoo Technologies, Inc.

# Distributed under the terms of the GNU General Public License, v2

or later

# $Header: /var/www/www.gentoo.org/raw_cvs/gentoo/xml/htdocs/doc/en/rsync.xml,v 1.26 2004/05/03 12:30:06 neysx Exp $

depend() {

need net

}

# FYI: --sparce seems to cause problems.

RSYNCOPTS="--daemon  --timeout=300"

start() {

ebegin "Starting rsync daemon"

start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 15 --exec /usr/bin/rsync -- ${RSYNCOPTS}

eend $?

}

stop() {

ebegin "Stopping rsync daemon"

start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid

eend $?

} 

```

3.  /etc/rsync/rsyncd.motd

```

Welcome to My Local Rsync Mirror!!

```

Then start rsyncd

```

# /etc/init.d/rsyncd start

```

rsyncd should be started at boot - add to default runlevel

```

# rc-update add rsyncd default

```

Now you have a local server up and running.  You must point your client boxes to this new server.  

Edit /etc/make.conf on the client boxes and change the SYNC option to your new server.

The SYNC section should look like this :

```

#SYNC="rsync://rsync.gentoo.org/gentoo-portage"

SYNC="rsync://YourMirrorHere.com/gentoo-portage"

```

Replace YourMirrorHere.com with your server's Name or IP address.

Remember to open port 873 on the rsync server box or any firewall between your clients and the rsync server box.

Then kick back and admire your work, knowing your client boxes can sync faster over the LAN and your saving Gentoo/yourself alot of bandwidth!!

NOTE 1: My /etc/init.d/rsyncd above has --nicelevel  set to 15.  This purposely slows down the rsync server so you won't notice other boxes syncing - This is what you want if you spend time logged in on the server box.  If you only use your rsync server as a server, set the nicelevel to 0 to make the rsync server run at normal speed (change the line with  --nicelevel 15 to --nicelevel 0 in the /etc/init.d/rsyncd above.  You could also set the nicelevel to max 19 to slow it down even more ).

NOTE 2: The default script installed with rsync is similiar but omits nicelevel, and timeout.   You could use the default script and and add "--timeout=300" to /etc/conf.d/rsyncd , but that would still leave out setting the nicelevel which can only be done by editing the file.  By replacing the script with my verson, I've kept all the changes in one file, which I find easier when merging future updates......

UPDATE* Latest versions of rsync support bandwidth limiting. Large shared installations might consider this option.

  --bwlimit=KBPS    limit I/O bandwidth; KBytes per second

If you need to setup this local rsync server on a non-gentoo box, this will sync your data with the official gentoo rsync servers:

```

#!/bin/bash

RSYNC="/usr/bin/rsync"

OPTS="--quiet --recursive --links --perms --times --devices --delete --timeout=300"

#Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org

#SRC="rsync://rsync1.us.gentoo.org/gentoo-portage"

#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:

SRC="rsync://rsync2.de.gentoo.org/gentoo-portage"

DST="/space/gentoo/rsync/"

echo "Started update at" `date` >> $0.log 2>&1

logger -t rsync "re-rsyncing the gentoo-portage tree"

${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1

echo "End: "`date` >> $0.log 2>&1 

```

Let me say again, most of this info is taken from the Official Howto BUT modified for LOCAL use.  The gentoo-rsync-mirror ebuild is designed for OFFICIAL PUBLIC mirrors only.

Version 1.8

--safe-links no longer needed or supported in daemon mode

added note about bwlimiting

Version 1.7

Added script note

Version 1.6

Updated rsync.conf location

added non-gentoo

Version 1.5

Added firewall port

Version 1.4

Cleaned up [gentoo-x86-portage]

section in case someone actually needs it

Version 1.3

Added trailing / to 'exclude distifles'

Version 1.2

Added nicelevel note

Version 1.1

added hosts allow/deny

Version 1.0

Initial PostLast edited by flybynite on Sat Aug 19, 2006 8:24 pm; edited 10 times in total

----------

## barran

Why did you make your own init-script? There is one included in the rsync ebuild and it is working smoothly for me also having a local mirror I made myself.

You can adjust settings in /etc/conf.d/rsyncd.

I'm just curious.

----------

## dhurt

The init script he used was from the original Gentoo howto. 

Is the main difference between your method and the method posted here:

https://forums.gentoo.org/viewtopic.php?t=59134

Is that you use the Gentoo Documentations init script and you use host blocking?  Why was the previous setup insecure? Thanks for the update.

----------

## senectus

woohoo it works fine  :Smile:  thanks...

Now to get the package cache working  :Wink: 

----------

## flybynite

 *barran wrote:*   

> Why did you make your own init-script? There is one included in the rsync ebuild and it is working smoothly for me also having a local mirror I made myself.
> 
> You can adjust settings in /etc/conf.d/rsyncd.
> 
> I'm just curious.

 

I guess there are two or three reasons for the init script change.

1. Mine uses the start-stop-daemon, which is the official gentoo standard.   I believe the default rsync  init script will be changed to use the start-stop-daemon whenever a developer can spare some time to fix it.

2.  My script has some nice features I wanted to make this a sweet install for users, like setting the --nicelevel which the default script can't do.

3.  I copied it exactly as the official howto suggests!!

----------

## flybynite

 *ender2431 wrote:*   

> 
> 
> Is the main difference between your method and the method posted here:
> 
> https://forums.gentoo.org/viewtopic.php?t=59134
> ...

 

Yes, my init script is different, read the above post for why mine is better.

My config is also more secure!!

An official gentoo rsync server was compromised a while ago.  Part of the hack was kernel related, part was rsync's fault.

The main problem with the older howto you mention is it contains:

#uid = nobody

#gid = nobody

use chroot = no 

which is the part that makes it insecure due to a known rsync exploit.  A fix is in the works but I don't think it is in portage yet.

Look at the latest rsync ebuilds and you will see this warning:

 *Quote:*   

> 
> 
> 	ewarn "Please make sure you do NOT disable the rsync server running"
> 
> 	ewarn "in a chroot.  Please check /etc/rsync/rsyncd.conf and make sure"
> ...

 

This warning is also in the Official Howto:

 *Quote:*   

> 
> 
> For security reasons, the use of a chrooted environment is required!
> 
> 

 

I added the Hosts Allow/Deny because it is appropriate and another level of security for a LAN only server.

----------

## cogitate

the deny should come before the allow

```
# Deny all

hosts deny *

# Allow from internal

hosts allow 192.168.0.0/255.255.255.0 10.0.0.0/255.0.0.0

```

I'm not sure if it really matters, but I got a deny error first time I tried it.

----------

## flybynite

 *cogitate wrote:*   

> the deny should come before the allow
> 
> ```
> # Deny all
> 
> ...

 

I believe your are incorrect because your config syntax is wrong.  You forgot both the '=' as in 'hosts deny=', 'hosts allow=',  and have one too many 255's in your netmask!!!

Try to copy the config again.......

----------

## dhurt

The netmask he is using is just more restrictive.  It allows only IPs on the 192.168.0.X Class C Subnet.  The 255.255.0.0 is a Class B subnet.  do you really have more than 254 computers/ip's in use on your network?  :Wink: 

----------

## flybynite

I'm afraid you missed the point that because of the syntax error, it doesn't matter what the netmask is because his options are being ignored.  He is left with the defaults which are to allow anybody and that's why it's working for him now .....

Since he didn't change the 10.x.x.x netmask and yes, neither he, nor I,  have 2^24 (16,777,216) computers - I guessed he just copied them wrong and didn't  change the netmask on purpose  :Smile: 

----------

## dhurt

 *flybynite wrote:*   

> [quote="and have one too many 255's in your netmask!!! 

 

Yeah, I realize his syntax error, I was just referencing this comment.  The only reason I mentioned something is that is how I have mine set (minus the "=" typo) and it works.  Thanks for your work in this project here flybynite and with your portage web cache system.  I am amazed at how beautiful it works here on my lan.    :Very Happy:   Cheers!!!

This is just awesome (8.50 M/s):

```

# emerge -f openoffice

>>> emerge (1 of 1) app-office/openoffice-1.1.1-r1 to /

>>> Downloading http://gentoo.mirrors.pair.com/distfiles/OOo_1.1.1p1_source.tar.bz2

--20:59:36--  http://gentoo.mirrors.pair.com/distfiles/OOo_1.1.1p1_source.tar.bz2

           => `/usr/portage/distfiles/OOo_1.1.1p1_source.tar.bz2'

Resolving xenophobia... 192.168.0.20

Connecting to xenophobia[192.168.0.20]:8080... connected.

Proxy request sent, awaiting response... 200 OK

Length: 192,388,698

100%[====================================>] 192,388,698    8.50M/s    ETA 00:00

20:59:59 (7.85 MB/s) - `/usr/portage/distfiles/OOo_1.1.1p1_source.tar.bz2' saved [192388698/192388698]

```

----------

## KpR2000

Hi,

I have configured my server (700Mhz) with the above hints. 

emerge sync works fine at my client computer. But I did not get full speed over a full duplex 100 Mb connection:

wrote 226 bytes  read 437 bytes  442.00 bytes/sec

Can I do speed improvements?

Greetings

----------

## Suicidal

 *Quote:*   

> The netmask he is using is just more restrictive. It allows only IPs on the 192.168.0.X Class C Subnet. The 255.255.0.0 is a Class B subnet. do you really have more than 254 computers/ip's in use on your network? 

 

The netmask for my entire network is 255.252.0.0 or /14 but it is comprised of over 40 subnets spanning 3 class b subnet and alomst 3000 hosts. It really doesnt atter as long as his hosts are within the masks range.

Personally on my home subnet i dont use 192.168.*.* because that is what a hacker would expect to find. I use a range within the 10.*.*.* range in case they are able to get past my firewall and nat. It would make it much more difficult to find a host within that range compared to a simple class c range.

----------

## Satori80

 *KpR2000 wrote:*   

> Hi,
> 
> I have configured my server (700Mhz) with the above hints. 
> 
> emerge sync works fine at my client computer. But I did not get full speed over a full duplex 100 Mb connection:
> ...

 

First, that's too small an amount of data to accurately judge the speed of the sync.

Secondly, unless there is something noticeably wrong with your LAN configuration, your hard drive will be the bottleneck here, not the network.

If you like, you can look into caching the distfiles on the server if you have an abundance of RAM. Do a search on proxy and cache, as there have been several posts about it in this section.

The only other thing you could do is upgrade to a faster drive or even use a RAID to improve speed, but I doubt on a one user system it'd be worth the effort and money unless you already notice that the disk(s) is/are slow in day to day use.

----------

## Satori80

I want to be sure I'm clear on this. Using this configuration one would use emerge sync to update the mirror?

----------

## dhurt

Yes, and then use the same command to replicate to properly configured clients  :Smile: 

----------

## Satori80

Okay, I've set up everything as per the instructions here (afaik), emerge sync on the machine running rsyncd. All goes well there. Changed my main desktop to sync off my LAN mirror and i get this:

```
>>> Starting retry 1 of 3 with rsync://<snip ip-address>/gentoo-portage

>>> checking server timestamp ...

<snip my motd>

receiving file list ...

link_stat "metadata/timestamp.chk" (in gentoo-portage) failed: No such file or directory

0 files to consider

client: nothing to do: perhaps you need to specify some filenames or the --recursive option?

rsync error: some files could not be transferred (code 23) at main.c(653)

>>> retry ...

```

I've gone over the configs several times, yet obviously I'm missing something. Any ideas?

EDIT: Maybe I wasn't clear enough in my 1st question... one would use emerge sync to get the portage tree for rsyncd? Not an rsync command?

----------

## dhurt

Sounds like you are not sharing the portage directory correctly on the server, could you post your /etc/rsync/rsyncd.conf?

Responce to your edit question, yes you run the "emerge sync" command on all your computers.  Just run your server computer first and then emerge sync on the clients.

----------

## Satori80

```
# Copyright 1999-2004 Gentoo Technologies, Inc.

# Distributed under the terms of the GNU General Public License v2

# $Header: /home/cvsroot/gentoo-x86/app-admin/gentoo-rsync-mirror/files/rsyncd.conf,v 1.4 2004/05/02 22:58:16 mholzer Exp $

uid = nobody

gid = nobody

use chroot = no

max connections = 20

pid file = /var/run/rsyncd.pid

motd file = /etc/rsync/rsyncd.motd

transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

#[gentoo-x86-portage]

#this entry is for compatibility

#path = /opt/gentoo-rsync/portage

#comment = Gentoo Linux Portage tree

[gentoo-portage]

#modern versions of portage use this entry

path = /etc/portage

comment = Gentoo Linux Portage tree mirror

exclude = distfiles

```

----------

## Satori80

From /var/log/messages:

```

Jun  6 19:33:59 ns1 rsyncd[3041]: rsync: name lookup failed for ip-address deleted: Name or service not known

Jun  6 19:33:59 ns1 rsyncd[3041]: rsync on gentoo-portage/metadata/timestamp.chk from UNKNOWN (ipaddress deleted)

Jun  6 19:34:00 ns1 CRON[3043]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Jun  6 19:34:12 ns1 rsyncd[2587]: rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(249)

Jun  6 19:34:17 ns1 rsyncd[3133]: rsyncd version 2.6.0 starting, listening on port 873

Jun  6 19:34:23 ns1 rsyncd[3136]: rsync: name lookup failed for ip-address deleted: Name or service not known

Jun  6 19:34:23 ns1 rsyncd[3136]: rsync on gentoo-portage/metadata/timestamp.chk from UNKNOWN (ip-address deleted)
```

It would seem this goes back to a DDNS problem I never got solved. :/

Also, when I log into the server it says it's name is "ns1.(none)" wich is another issue i've never been able to sort out. 

Thanks a lot for your help. I'll have to get on this other stuff 1st apparently.

----------

## dhurt

Your config is wrong as well, or should be:

This should not be correct in your rsync config:

```

path = /etc/portage 

```

but:

```

path = /usr/portage

```

Or whatever you have listed in your /etc/make.conf under:

```

PORTDIR=/usr/portage

```

----------

## Satori80

lol! Damn. Figures it was something simple and stupid like that. 

Thanks again.

----------

## KpR2000

 *Quote:*   

> 
> 
> First, that's too small an amount of data to accurately judge the speed of the sync.
> 
> 

 

Another test shows still such a bad result.

 *Quote:*   

> 
> 
> Secondly, unless there is something noticeably wrong with your LAN configuration, your hard drive will be the bottleneck here, not the network.
> 
> 

 

ok, hdparm -tT /dev/hdd3 sais:

/dev/hdd3:

 Timing buffer-cache reads:   240 MB in  2.00 seconds = 119.72 MB/sec

 Timing buffered disk reads:   50 MB in  3.11 seconds =  16.10 MB/sec

It's a 2" HD... But should it not enough for the rsync action?

 *Quote:*   

> 
> 
> If you like, you can look into caching the distfiles on the server if you have an abundance of RAM. Do a search on proxy and cache, as there have been several posts about it in this section.
> 
> 

 

I think it is not a good idea to cache, because my ram is limited to 256Mb

About the ftp protocol I get full speed out of the machine. What is wrong here?

Thx in advance[/quote]

----------

## flybynite

KpR2000:

The easiest fix for the name lookup failures in your logs is to list the ip's and hostsnames in /etc/hosts.

I noticed that you seem to be comparing your rsync server speed with someones distfile cache speed in this thread.  Two different things.

Now that you have your config file fixed, what speeds are you getting?

----------

## CarpJA

I too am having speed issues... Downloading the file list from my local mirror takes several minutes as opposed to less than a minute for an internet mirror.  Has anyone ecountered this and solved it?

----------

## KpR2000

 *flybynite wrote:*   

> 
> 
> The easiest fix for the name lookup failures in your logs is to list the ip's and hostsnames in /etc/hosts.
> 
> 

 

The ip's for the rsync server and all other pc's in my network are listed there.

I have also not used an hostname in /etc/make.conf for the SYNC variable:

SYNC="rsync://192.168.3.84/gentoo-portage"

 *Quote:*   

> 
> 
> I noticed that you seem to be comparing your rsync server speed with someones distfile cache speed in this thread.  Two different things.
> 
> Now that you have your config file fixed, what speeds are you getting?
> ...

 

Thats my current state:

```

receiving file list ... 

1 file to consider

timestamp.chk

          32 100%   31.25kB/s    0:00:00  (1, 100.0% of 1)

Number of files: 1

Number of files transferred: 1

Total file size: 32 bytes

Total transferred file size: 32 bytes

Literal data: 32 bytes

Matched data: 0 bytes

File list size: 32

Total bytes written: 226

Total bytes read: 437

wrote 226 bytes  read 437 bytes  442.00 bytes/sec

total size is 32  speedup is 0.05

```

and the collected packages counter is really slow in contrast to the "internet sync". > like the previous answer by "CarpJA"

Greetings

----------

## dhurt

 *KpR2000 wrote:*   

> 
> 
> ```
> 
> receiving file list ... 
> ...

 

This is way to little data to get the speed from.  The time stampfile contains just this:

```

Thu Jun 10 15:06:57 UTC 2004

```

Downloading from the lan or the internet even over a 56K modem on something this small will be the same.  That is not the bottleneck.  The problem with trying to see a speed increase on the emerge sync is that you will not see one.  You are transfering lots of very small files.  This is NOT a bandwidth intensive process.  Look at my results from the main portage tree sync.  As you can see the internet sync is faster.  Maybe by about 5 seconds out of 40.  Nothing big.  Also the amount of data transfered is 300K which is tiny.

```

<Internet>

------------------------------------------------------------

Number of files: 85256

Number of files transferred: 182

Total file size: 69887332 bytes

Total transferred file size: 341443 bytes

Literal data: 341443 bytes

Matched data: 0 bytes

File list size: 1944462

Total bytes written: 3825

Total bytes read: 2083098

wrote 3825 bytes  read 2083098 bytes  46897.15 bytes/sec

total size is 69887332  speedup is 33.49

<Local Mirror>

------------------------------------------------------------

Number of files: 86006

Number of files transferred: 180

Total file size: 69896735 bytes

Total transferred file size: 340442 bytes

Literal data: 340442 bytes

Matched data: 0 bytes

File list size: 2036955

Total bytes written: 3785

Total bytes read: 2177599

wrote 3785 bytes  read 2177599 bytes  37937.11 bytes/sec

total size is 69896735  speedup is 32.04

```

This process I found is HIGHLY dependent on server load.  I tried syncing while the main server was caching the portage files and the process was  slow as can be.  I know that rsyncing causes major load on the server.  Not just disk activity, but also computational activity. So the bottleneck can be somewhere else, probably the speed of the computer.  The gentoo servers are usally dual processor servers optimized to be a rsync mirror and your desktop/local server is probably not and so the process will be slower.  But it will not be an unbearably longder time, the actual rsync time is very quick and caching the portage tree takes much longer than the rsync process anyway.  The reason for creating a local mirror is not so that it is faster, as you can see mine was slower.  But to reduce load on the gentoo servers.  You really do not need to have two copies of the same information downloaded.  That is wasteful.  If you want a speed increase Take a look here:

https://forums.gentoo.org/viewtopic.php?t=173226&highlight=

This http-replicator and allows you to cache all the portage files and will then serve them up at lan speeds  :Very Happy: 

----------

## KpR2000

You are right with your arguments. I will give http-replicator a try.

Thx

----------

## mxc

Is it possible to set the client up to fallback to an external rsync server if it cannout find the file it needs on the local server? I have an adsl connection with a cap limit. I often need to install machines over night and I would prefer the machine to finish compiling than save bandwidth in this case.

thanks

----------

## dhurt

You are confusing an rsync mirror and a package mirror.  The rsync mirror which this post is about will not have any packages in it.  It allows you to:

```

# emerge sync 

```

on just one machine and then replicate that effort to other machines on the lan to reduce the load on the Gentoo mirrors.  It just syncronizes /usr/portage, but excludes /usr/portage/distfiles and /usr/portage/packages.  So there are no files that are skipped unless you have a funky setup.

I think you are refering to setting up a local package mirror which you would want to use http-replicator which is another part to the system. It basically caches all the files that you have downloaded for building purposes locally.  If it cannot find a file it then it downloads it from the internet.

A link to it is about 3 posts above.

----------

## mxc

Thanks KillBill,

In the one post I found the poster had set up a rsync 'link' to the portage/distfiles directory. Would just sinking this with another machine not mean that I have all the files the other has and there will only be a need to download ones which I don't already have?

Would rsyncing the distfiles dir skip some important step that emerge needs?

I will look into setting up the http proxy as a longer term solution later.

thanks

----------

## dhurt

The problem with the rsync solution is that there is not a fall back if the package is not on the main server.  A poor solution because if you are upgrading alot of packages and it cannot download the file halfway through it, the ebuild will fail.  You then have to change your mirror and download the file manually, or go to your server, download the file manually, change back your mirror so it is pointed at your local server.  Finally continue the ebuild.

I know I used it for about 2 months.  It was a pain to keep up in the long run and not transparent at all.

http-replicator on the other hand is very seemless.  It is a proxy between you and the internet just for the purposes of getting distfiles.  It does not mess with your traffic in anyother way.  How it works, is all requests for files come through the proxy now.  If it has the file locally, it serves them up at LAN speeds.  If it does not have the file,  it fetches the file locally to the proxy and sending it to the requesting machine at the same time.  It is very seemless.  So if you have the file on your proxy it, comes in at LAN speeds, if not it comes in at the speed of your connection.  I have been using it for 2-3 weeks now and it is excellent.

----------

## seringen

Well, other than a stupid carriage return error in a config file, everything worked immediately and beautifully.

To give people an idea about what would be typical performance for most people here's an example from my network:

First the rsync server over ssh, a VIA Nehemiah computer

```
# hdparm -tT /dev/hda

/dev/hda:

 Timing buffer-cache reads:   520 MB in  2.01 seconds = 258.10 MB/sec

 Timing buffered disk reads:  122 MB in  3.06 seconds =  39.89 MB/sec
```

Now the connecting computer, a PIII Laptop with a slow, normal harddrive

```
# hdparm -tT /dev/hda

/dev/hda:

 Timing buffer-cache reads:   416 MB in  2.00 seconds = 207.51 MB/sec

 Timing buffered disk reads:   54 MB in  3.12 seconds =  17.33 MB/sec

```

Over fast ethernet it gets

```
39833.50 bytes/sec
```

All in all not bad and without any optimizations of any sort, and it really is a good thing to take some of the weight off of the main mirrors - it's easy to forget how heavy rsync is on servers.

----------

## Cetanu

Is there any reason to keep portage for unofficial mirror outside server's /usr/portage directory? 

I am asking because I installed app-admin/gentoo-rsync-mirror package today and portage is kept in separate directory by default (/opt/gentoo-rsync/portage/). I use configuration with portage keep in /usr/portage for half of year and I haven't experienced any problems yet...

----------

## dhurt

Works great with the directory /usr/portage/.  Maybe on the server configuration they like to mount the /usr directory read only until update times and storing this in opt would allow them to do this and still have an upto date mirror.

----------

## flybynite

New HOWTO version 1.2 !

I added a note about the rsync daemon nicelevel that my /etc/init.d/rsyncd script sets on starting.  This applies only if you use my script on your machine.

My script sets the nicelevel to a lower priority (15) than normal (0) because I spend time logged in on my rsync server box and use it as a normal desktop.  If you do also, leave it set as is.  If you only use your rsync server as a server go ahead and set the nicelevel to 0 to give normal priority to make rsync run at normal speed.

----------

## JSharku

Just a quick note on packages and distfiles; it's better to put the following in your rsyncd.conf:

```

# excluding packages is optional, if you don't use --buildpkg you don't need it

exclude = distfiles/ packages/

```

instead of

```

exclude = distfiles packages

```

NOTE THE TRAILING /'s

If you don't add the slashes  rsync will exclude anything ending in either distfiles or packages, not just those directories. Not that big a deal you might say, were it not that every /usr/portage/profiles/<specific profile>/ directory has a file in it called packages which portage uses to determine what to build when you bootstrap or emerge system. Those files get deleted by rsync on the client machines if you don't add the trailing slashes, resulting in rebuilds, rebootstraps, resyncs and tons of frustration... at least it did for me until I finally figured this out.

Sharku

----------

## flybynite

 *JSharku wrote:*   

> Just a quick note on packages and distfiles; it's better to put the following in your rsyncd.conf:
> 
> ```
> 
> # excluding packages is optional, if you don't use --buildpkg you don't need it
> ...

 

I see your point about what a trailing / does and you are correct, but I'd bet you're doing this for the wrong reasons and you don't need it either!!

First, you're correct about the trailing slash in the exclude pattern ensuring it only excludes directories and not files.  I've updated my howto just to make it clear what is being excluded, but it probably doesn't matter if any user changes their config.

The reason it doesn't matter is were dealing with the SERVER.  The exclude in the SERVER config makes it impossible for a client to TRY and get distfiles (or packages, in your config) by rsync.

But portage will NOT request those files!!!!!

Look at file:/usr/lib/portage/bin/emerge for rsync_flags and you'll find that the portage CLIENT sets rsync options that automatically skips distfiles, local, and packages.

```

         "--exclude='distfiles/*'",   # Exclude distfiles from consideration

         "--exclude='local/*'",       # Exclude local     from consideration

         "--exclude='packages/*'",    # Exclude packages  from consideration

```

So to wrap this up:

1.  Gentoo's portage automatically skips distfiles, local, and packages when syncing so you don't have to exclude these in the SERVER config, and they won't ever appear on clients when you 'emerge sync'.

2.  Excluding distfiles in the SERVER config only serves to prevent anyone from abusing the server using  their own rsync command.  It is possible to create your own rsync request that would try to suck down all of distfiles from the public rsync servers.  Excluding distfiles on the SERVER prevents a user from doing this.   I left this in as protection for those running my local rsync server in a college campus, for example.

However, if you're running a semi public server on a gentoo box with alot of packages and your afraid someone might try to craft an rsync command to get all your packages, exclude distfiles/ packages/ per JSharku's example above.

----------

## JSharku

When I first set up my local rsync server, an emerge sync would try to pull in the distfiles and packages, so I added that line to my rsyncd.conf, which worked at the time (portage 2.0.4x, 1-1.5 years ago ) so I kept it in there. I didn't know it had been added to portage, so I kept the line thinking it was necessary. It's only very recently that I discovered it was messing with emerge system, but I still didn't know that the exclude line itself had become obsolete.   :Embarassed: 

Sharku

----------

## flybynite

No problem, Its hard to keep up with all the changes.  Thanks for helping make the syntax clear in the config.

----------

## _sparks_

try turning logging off

/etc/rsync/rsyncd.conf:

```

#This will log every file transferred - up to 85,000+ per user, per sync

transfer logging = no

```

speeds up things in my configuration by a factor of 100 or soLast edited by _sparks_ on Fri Jun 25, 2004 12:05 pm; edited 1 time in total

----------

## fvant

My local rsync server seems to sync in blocks of 200 files only. Where as the internet download filecounter can barely be followed, rsync from my local server steps slowly in steps of 200

CPU on the server i rsync form is not busy and rsync process only uses 2.3%, HD use DMA

----------

## Marwin

Take your samba-server and make a directory that you call 'distfiles'.

Share it and make the clients mount it at /usr/portage/distfiles.

And Wolla! You've got a shared distfiles  :Smile: 

----------

## quill18

 *fvant wrote:*   

> My local rsync server seems to sync in blocks of 200 files only. Where as the internet download filecounter can barely be followed, rsync from my local server steps slowly in steps of 200
> 
> CPU on the server i rsync form is not busy and rsync process only uses 2.3%, HD use DMA

 

Ditto on this.  Very similar performance.  Lots of spare CPU, bandwidth, and harddrive speed but terrible throughput.

Made sure that the hostnames are setup properly, and tried it with the original startup script as well as the one posted above.

----------

## flybynite

It appears some users are using other rsync.conf file and startup scripts and are having problems.  Someone even posted a bad rsync.conf in this thread!!

The reason I posted this HOWTO is to eliminate the junk floating around!!

Everyone check that you are using the exact config and startup scripts in the HOWTO!!  That will eliminate many problems!!

----------

## Nekkrist

For everyone having speed related issues with your sync'ing, this is probably not a network problem, configuration problem, or anything of the sort.  It is probably simply an aspect of computer hardware.

The reason the rsync server's appear to be so fast is that all they do all day is offer syncing services.  Your local mirror, however, does not do this all day, in fact it probably very rarely is sync'd against.

Since the inner workings of the rsync algorithm are somewhat detailed, if you are interested, read http://samba.org/~tridge/phd_thesis.pdf (the rsync author's PhD thesis which includes a few chapters on rsync).

Otherwise, the basic result is that the rsync protocol operations are cached by the CPU cache of the main rsync mirrors, so that they don't actually need to be performed every single time.  If you happened to be the very first person to sync against a main server after it was turned on, you would see very similar results to your own server.  Your own server does not have these operations in the CPU cache since when you sync, that is likely the first time it has been sync'd against since its update.

If you have three or four computers, let one sync to the server, then after that one has completed, do another sync, chances are it will be a bunch faster than your previous findings.

----------

## flybynite

Thanks for some more info Nekkrist!

There are many things to consider about your rsync speed:

1.  CPU/Memory  - your old Pent II 233mhz isn't going to be as fast as an official rsync server such as crane.gentoo.org with it's Dual 1.7GHZ Xeon's and 2GB ram.

2.  Filesystem/Disk Speed - Rsync has to consider about 85,000 small files in many dir's.  Put your /usr/portage on a fast disk with a filesystem that has high small file performance.

3.  Disk Cache - The second rsync will be faster than the first.

4.  Logging - My config has logging turned off because every client rsync will generate 85,000+ lines in the log file!  

5. More....

----------

## dmitrio

I have copied this HOWTO, with permission of flybynite, to gentoo-wiki.com 

http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror

If you see anything that should be added or changed, feel free to do so. 

Thank you for a great HOWTO.

----------

## flybynite

I appreciate that dmitrio, the wiki should help get the word out!!

I also submitted the howto's for possible inclusion in Gentoo Weekly Newsletter as suggested by monkeywrench on the http-Replicator thread https://forums.gentoo.org/viewtopic.php?t=173226

----------

## dmitrio

 *flybynite wrote:*   

> I also submitted the howto's for possible inclusion in Gentoo Weekly Newsletter as suggested by monkeywrench on the http-Replicator thread https://forums.gentoo.org/viewtopic.php?t=173226

 

Thank you for good HOWTO

please look at 

http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

If you see anything that should be added or changed, feel free to do so.

----------

## flybynite

never heard anything back about the newsletter...

----------

## drakos7

Great little HOWTO. I am thinking about providing this to my university as we are now charged per GB of external traffic. Not that syncing transfers much but...

One question from brain dead me: How do I go about getting my new local rsync server to sync to a main mirror on a regular basis. I assume it is accomplished using a cron command, but could someone please send me an actual cron command to do it? My thought is to put this in my crontab file:

```
0       0,3,6,9,12,15,18,21       *       *       *       emerge sync  >/dev/null 2>&1

 
```

So that it syncs every 3 hours (I hope! Is this right?) instead of the primary mirror's 2x/hour.

Is every 3hr too often?

----------

## flybynite

Setting up this rsync mirror on a campus is a great idea!!  You should also setup a cache for packages if you really want to save some traffic:

https://forums.gentoo.org/viewtopic.php?t=173226

Your users will want to use the cache because it is much faster!  

There is only one other thing you must do.  The gentoo rsync guidelines at http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3 suggest syncing more than 1-2 times a day for an individual user is bad.  Since your setting up a private rsync mirror for many users, you may need to  coordinate your use with an official mirror admin to prevent being banned.  If an official mirror admin thinks your an individual user and are abusing the rsync system, you could get your ip banned for a day.

The reason I wrote this howto is because the gentoo rsync mirror doc was out of date and confusing for local rsync mirror configuration.  Now that the official doc has been updated, you should also check it: http://www.gentoo.org/doc/en/rsync.xml

Just remember the only section to apply is section 4 -  Setting up your own local rsync mirror

----------

## drakos7

thx flybynite. I saw your other posting for caching packages while I was looking up this procedure. That is my next step.

As to syncing often, I may move it to every 4 or 5 hours. I am not strictly an "individual user" and the rotation of mirrors helps somewhat. Besides, how often do people really need to sync? Ya know? 

It is definitely not as bad as where I found my cron example. The guy had his machine set to sync every hour!  :Evil or Very Mad:  Rather inconsiderate I do believe. And it was posted to these forums!   :Shocked: 

----------

## afabco

At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host?

----------

## flybynite

 *afabco wrote:*   

> At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host?

 

This has been discussed before, look for more info on the boards. Here is the short version....

There are numerous security and other problems including non existent file locking plus nfs is slower than local disk access.  Portage can't share some of the directories under /usr/portage without possible corruption problems.  

However, If you own/control all machines and/or you totally trust all users of all machines and all others on the network, and you only mount specific directories which aren't machine specific, and you don't edit files that would fubar other machines, and/or you don't dare emerge sync from more than 1 machine at a time and/or you dont dare emerge/download a package from more than 1 machine at a time , and you don't dare etc etc .... it can be done as long as your still comfortable with the remote possibility of file corruption.

The only reason I can think of to nfs share /portage would be if you had a really small disk on a box.  As long as you have the disk space, the local rsync mirror is a simple, safe, and the highest preformance option.  You can even share it with your friends at a college, work, or gentoo install party.

----------

## afabco

OK, that's kinda what I thought.  Thank you for replying

Ignoring the fact that I'm very dubious about trusting NFS  (usually a lot more trouble than it's worth, and 9 times out of 10 I give up in disgust and use smbmount and then there's the squash flakiness and....and...)   :Rolling Eyes: 

But when it does work, most of my complaints are handled with the 'soft' mounting option....

Isn't /usr/portage supposed to be identical no matter where it is?  Except for /usr/portage/distfiles of course.  So the only machine that would ever emerge sync is the hosting machine; all the rest would only do emerge <pkg> or emerge -uD world?

There is the speed thing of course, but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out).  I -thought- I took care of that by just setting up a 10 G partition on my central machine and then nfs sharing it all around.

I'll have a look around the board for other discussion.  Any pointers?

----------

## jussen

i want to link my uni server for emerge but it always fails is it possible that the server only have the files online but not an sync server installed??

```
bash-2.05b# emerge sync

>>> starting rsync with rsync://131.246.120.27/pub/linux/gentoo-portage...

>>> checking server timestamp ...

@ERROR: Unknown module 'pub'

rsync: connection unexpectedly closed (41 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(189)

>>> retry ...

```

----------

## flybynite

 *afabco wrote:*   

> 
> 
> but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out)

 

The portage tree is really about 600mb.  This is the part that is needed by gentoo and what you get when you emerge sync.

All else is space needed only temporarily and can be deleted.  Gentoo doesn't do this automatically for various reasons.  Many users don't even know what they can delete.

Try it yourself:

```

du  -c -h --exclude=distfiles /usr/portage 

```

If you don't have disk space for 600mb on a box then it might be worth the hassle of trying to share /usr/portage and losing portage capabilities now or in the future.

Realize that we can encourage developers to make portage more friendly and work sanely with multiple machines sharing /usr/portage.

There is work being done to make it safe to run two emerge's on one box and were closer today than last year,  but for now it is not safe to even run two emerge's on one box - nevermind over a nfs share.

----------

## micmac

Thanks man! Your howto is most useful! Great idea!

Laters

mic

----------

## cbradney

Is this still applicable now? Portage and rsync have been updated since June. IE, is the ebuild for the rsync server solely for a non local network?

----------

## flybynite

Good question.... Could you install it and compare it to this howto?

I don't know if the ebuild is still for official mirrors.  I did look at the local rsync howto at:

http://www.gentoo.org/doc/en/rsync.xml

and it is much improved.

----------

## cbradney

I've followed the gentoo rsync server howto and it seems to work just fine here. Theres a section at the bottom for local LAN stuff. 

Now.. I've also posted on the wiki talk page re the http replicator.. i get an error:

 readline43-004

 readline43-005

 Checking authenticity and integrity of new files...

 Searching for ebuilds's ....

 Done!

 Found 16435 ebuilds.

 Extracting the checksums....

 Missing digest: net-mail/gml-0.5

 Done!

 Verifying checksum's....

 /usr/portage/distfiles/gdbm-1.8.0.tar.gz

 Traceback (most recent call last):

   File "/usr/bin/repcacheman", line 198, in ?

     if t[0]:

 KeyError: 0

----------

## flybynite

fixed:

download an updated repcacheman:

http://www.updatedlinux.com/replicator/portagefix/repcacheman

see the original thread for the background and the fix instructions:

https://forums.gentoo.org/viewtopic.php?t=173226&start=175

----------

## cbradney

Hey.. cool! Seems to be working having done an emerge -f of a package! Now all I need is something I really want to emerge  :Smile: 

Thanks heaps for this.. any idea about getting it into portage?

----------

## flybynite

The ebuild has been submitted.  I don't know any reason for the delay....

----------

## anderlin

I have a rsync-mirror on my LAN, set up following the instructions in this thread, and I want to include some extra ebuilds not in portage. How can that be done?

----------

## flybynite

Good question... 

I don't know of any portage feature that will help you do this.  Doesn't mean it doesn't exist...

You could define another rsync source dir for /usr/local/portage so any machine could separately sync your local ebuilds to their /usr/local/portage....

----------

## anderlin

That's a possible solution. Another is to replace (alias?) emerge sync on the server with a command that syncs the tree, for then to copy the extra ebuilds into it. Both solutions work, but I don't find them very elegant... Some built in feature would have been nice.

----------

## packman

 *Marwin wrote:*   

> Take your samba-server and make a directory that you call 'distfiles'.
> 
> Share it and make the clients mount it at /usr/portage/distfiles.
> 
> And Wolla! You've got a shared distfiles 

 

I can assure you - you'll lose a lot of speed with this. Building the dependency cache on a lot of small files on a network drive? Not a good idea imho  :Smile: 

For distfiles - that's smth else (however, I just use squid for this with a rather large cache  :Wink: )

----------

## flybynite

I've trie squid a while ago.  The problems I had include:

1.  Squid names files "lskajdifoa98834o53l2kdsl" so they are not usable by another program.

2.  Squid isn't made for binary files.  One example is http://gentoo.oregonstate.edu/distfiles/kde-3.3.1.tar.bz2 is not the same to squid as http://some.other.mirror/distfiles/kde-3.3.1.tar.bz2 even though it is the same file.....

3. Squid wants to remove cached files by time.  Problem is kde-3.3.1.tar.bz2 is current for months/years? till kde is upgraded, squid still wants to remove files by time not version.

4. You can't add files to the cache yourself - like from an install CD...

I could go on and on...

Just try http-replicator....

https://forums.gentoo.org/viewtopic.php?t=173226

----------

## Boris27

I have a portage server for my local lan.

It's a rsync mirror (with the help of this thread), and exports its distfiles via samba. Distfile sharing like that is pretty nice, if you control all the systems.

If you don't, flakyness can occur. Say you're downloading kdebase-3.3.1 and someone else wants to install it too. Their portage client thinks the file is there (it exists), and starts extracting the partial file. It will fail. Not nice. If you update all the boxes, you know what is downloading where, so no duplicate installs are started.

----------

## flybynite

 *Boris27 wrote:*   

> Distfile sharing like that is pretty nice, if you control all the systems.
> 
> If you don't, flakyness can occur.

 

The question is why would you make your system flaky and slow?  Yes, if you always remember exactly how to do updates and portage never changes your probably OK.  A better solution exists...

Using http-replicator is fast and bulletproof!!   You can update ALL your boxes at the same time or at any time without worry and do so knowing that they will finish faster!!

https://forums.gentoo.org/viewtopic.php?t=173226

----------

## Gentree

I have got my second box to sync off the main system but I want to rsync my overlay files as well.

I added a [overlay] section to  /etc/rsync/rsyncd.conf that seems correct:

```
rsync rsync://linbox/overlay 
```

this returns a list of the directories on linbox.

```
rsync rsync://linbox/overlay/* . 
```

This only tries some of the files and fails with permission errors.

Why does this work on one structure but not on the other?

Thanks for any advice.

 :Cool: 

----------

## flybynite

 *Gentree wrote:*   

> I have got my second box to sync off the main system but I want to rsync my overlay files as well.
> 
> I added a [overlay] section to  /etc/rsync/rsyncd.conf that seems correct:
> 
> ```
> ...

 

Here is what I can think of at the moment.

You have to deal with both client and server levels of rsync and portage.

Rsync SERVER has it's own set of default security features and options, such as --safe-links that my init script uses.  If you don't use my init script what options does your sever init script use?

The rsync CLIENT has options set by portage.  You'll have to dig through portage to find them.  Those options do things like skip certain files and ignore certain dir's.

The options could be the cause, or it could just be the file permissions?

----------

## fourhead

Great how-to I'm setting this up ony my virtual Debian server right now. I copied my local /usr/portage to /var/portage on the Debian server and edited the config files accordingly. My only question now is - how do I rsync the portage tree that is in /var/portage on the Debian box? There's no "emerge sync" of course. I'm doing this service mainly for a few friends and some people on my university, so I was thinking I could sync the tree on the Debian box perhaps two or four times a day, and me & my friends can sync to this server. But again, how do I sync the Debian box with an "real" official portage mirror?

Tom

----------

## Gentree

man rsync !

I am pretty sure you have rsync on the debain system  so just look at /etc/make.conf to see the rsync command used by gentoo and adapt the paths etc. to do the same thing on your Debox.

HTH  :Cool: 

PS also do regular tar of both the portage tree and distfiles if you have room. This will be a great help to you when portage rsyncs with broken packages and deletes the working ones from your system's portage.  :Evil or Very Mad: 

----------

## flybynite

This script below should get you started.  I took this from the official gentoo howto on rsync at http://www.gentoo.org/doc/en/rsync.xml

The original reason I wrote this howto is because that official howto was confusing and gave bad info for home/casual users.  I created this howto and suggested changes to the official howto.  Last I looked, most of the problems with the official howto are fixed.  Many users could still be confused so my howto is still relavent.

I would also recommend setting up http-replicator.  Read all about it and know ~15,000 of your fellow enthusiasts use it also!

https://forums.gentoo.org/viewtopic.php?t=173226

```

#!/bin/bash

RSYNC="/usr/bin/rsync"

OPTS="--quiet --recursive --links --perms --times --devices --delete --timeout=300"

#Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org

#SRC="rsync://rsync1.us.gentoo.org/gentoo-portage"

#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:

SRC="rsync://rsync2.de.gentoo.org/gentoo-portage"

DST="/space/gentoo/rsync/"

echo "Started update at" `date` >> $0.log 2>&1

logger -t rsync "re-rsyncing the gentoo-portage tree"

${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1

echo "End: "`date` >> $0.log 2>&1 

```

----------

## lysergicacid

anyone know how id setup the box i just setup to be my rsync mirror to also be my distfiles server plz ? , i suppose i could just mount the distfiles folder over nfs into the portage tree on the comps i wana update but dont wana do it that way

----------

## flybynite

 *lysergicacid wrote:*   

> anyone know how id setup the box i just setup to be my rsync mirror to also be my distfiles server plz ?

 

The absolute best way is to use http-replicator which will do what you want plus has many more benefits:

https://forums.gentoo.org/viewtopic.php?t=173226

----------

## user808

Nice, thanks for the guide... I was having problems untill I figured out that the rsync server runs on port 873 and had to poke a hole in firewall.

----------

## flybynite

 *user808 wrote:*   

> Nice, thanks for the guide... I was having problems untill I figured out that the rsync server runs on port 873 and had to poke a hole in firewall.

 

Thanks!  The point about opening the port is a good one, I've added it to the HOWTO.

----------

## flybynite

Ymerej pointed out to me that gentoo has made a few changes to the default scripts and wondered if the scripts in this howto are out of date.

This howto is still current and will work fine.  I will update this howto to use the new "style", but there won't be any operational differences so feel free to use this howto without worry  :Smile: 

----------

## SchrodingerPenguin

Lots of thanks to flybynite for a useful and easy-to-implement howto!

I have just followed the instructions as given at the beginning of this thread, and right at this instant my brother's laptop is syncing from mine.  This will certainly save doubling up download time from the internet.

I had everything work fine, and I copied the config files exactly as given.

Thanks again for a great bit of help.[/b]

----------

## jleejj

Why does the rsync daemon run with root priviledges.  When I do a:

```
ps -ef | grep rsync
```

I get:

```
root      3134     1  0 11:15 ?        00:00:00 /usr/bin/rsync --daemon --safe-links --timeout=300
```

Shouldn't I see this process owned by "nobody" as per the rsyncd.conf file settings?  Also, is there a way to check that a process is really running inside a chroot?  I honestly don't know that much daemons so if I am confused feel free to set me straight, but I am concerned about the security of the current setup.

----------

## flybynite

 *jleejj wrote:*   

> Why does the rsync daemon run with root priviledges.  When I do a:
> 
> ```
> ps -ef | grep rsync
> ```
> ...

 

These are questions probably better asked on an rsync mailing list.  I'm not an expert on rsync, I just use it  :Smile: 

But I was curious and I get this:

```

 $ ps aux | grep rsync

root      6674  0.0  0.1   1712   496 ?        Ss   Feb19   0:00 rsync --daemon

nobody   24262 41.2  1.7   8184  6680 ?        R    00:44   0:06 rsync --daemon

tom      24273  0.0  0.1   1432   420 pts/2    S+   00:45   0:00 grep rsync

 $     

```

The difference is the daemon starts as root but changes into "nobody" when a user actually connects.  Only root can change into another user so that makes sense.  In my example, You see the program running as "nobody" because a user is actually connected.....

Verifying the chroot, I don't know how to do..  But here is the man page info:

 *Quote:*   

> 
> 
> use chroot
> 
> 	      If  "use	chroot"	 is  true, the rsync server will chroot to the
> ...

 

This man page is for a newer version of rsync.  Older versions don't have "use chroot" default to true, that is why it's still in the config.

So thats what I find.  Rsync runs as root initially in order to be able to change to "nobody" and run in a "chroot" when a user is actually connected, just like its supposed to...

----------

## transitbus

I have to confess I have been bad! I have 10 clients I was updating by going to official mirrors. Thanks for making this tutorial, I have beeen meaning to do this forever but never got around to it. Now I can do my part to relieve the load.

----------

## Jimmy Jazz

Hello,

i'm trying to use a rsync server in order to distribute distfiles to internal gentoo clients

It works great for gentoo portage tree but not for the distfiles tree  :Sad: .

i added the following lines in the /etc/rsync/rsynd.conf file,

```

[gentoo-portage]

path = /usr/portage

comment = Gentoo Linux Portage tree mirror

exclude = distfiles/

[gentoo-packages]

#For distributing Portage packages (distfiles) to internal clients

path = /usr/portage/distfiles

comment = Gentoo Linux Packages mirror

```

and for every clients, i added in /etc/make.conf,

```

GENTOO_MIRRORS="rsync://MY_LOCAL_SERVER_NAME/gentoo-packages

```

what have i done wrong ?

Jj

----------

## flybynite

 *Jimmy Jazz wrote:*   

> Hello,
> 
> i'm trying to use a rsync server in order to distribute distfiles to internal gentoo clients
> 
> what have i done wrong ?
> ...

 

Just using the wrong tool for the job  :Smile: 

Check out my other post on http-replicator.  So many advantages, seriously....

https://forums.gentoo.org/viewtopic-t-173226-highlight-.html

----------

## Jimmy Jazz

@flybynite

i really thought rsyncd would do the job as well.

Also, i will read your post and install it  :Wink: 

Thx for your howtos

Jj

----------

## Jimmy Jazz

Hello,

by the way i 'm using rsyncd with xinetd to make it even more secure, it works well and it is fast enough  :Smile: 

stop rsyncd

```

rc_update del rsyncd

/etc/init.d/rsyncd stop

```

restrict the access to your server

```

#cat /etc/xinetd.conf

# Sample configuration file for xinetd

defaults

{

        only_from      = localhost YOUR_LOCAL_NETWORK

        instances      = 60

        log_type       = SYSLOG authpriv info

        log_on_success = HOST PID

        log_on_failure = HOST

        cps            = 25 30

}

includedir /etc/xinetd.d

```

add rsyncd file

```

# cat /etc/xinetd.d/rsyncd

# default: off

# xinetd service description for rsyncd

service rsyncd

{

        disable         = no

        socket_type     = stream

        wait            = no

        user            = root

        server          = /usr/bin/rsync

        protocol        = tcp

        port            = 873

        bind            = YOUR_SERVER_IP_INTERFACE

        cps             = 60 10

        server_args     = --daemon --safe-links --timeout=300 

        nice            = 5

        instances       = 8

        type            = UNLISTED

}

```

xinetd needs to read the new configuration

```

kill -HUP XINETD_PROCESS

```

check if portmap is not restricted to listen localhost only

```

#cat /etc/conf.d/portmap

# Listen on localhost only by default

#PORTMAP_OPTS="-l"

```

Jj

----------

## Bob P

Thanks for posting this helpful guide.  In following it, I very easily set-up an RSYNC server on my home LAN a few days ago, and did my part to decrease the load on the gentoo mirrors.  :Wink: 

I'd like to point out one discrepancy that I've noticed between the guide and the current location of rsyncd.conf on my computers.  It seems that the default location of the rscyncd.conf files on my Gentoo boxes is different from the location suggested by the guide:

The guide states that the configuration files are located at /etc/rsync/rsyncd.conf.  On both my 2004.3 and 2005.0 boxes there is no /etc/rsync directory, and the rsyncd.conf file is located at /etc/rsyncd.conf.  After taking that discrepancy into account, the process of creating an rsync server works fine with the guide as it is written.

My RSYNC server has been running fine for a few days now.  I've decided to rebuild the Gentoo installation on the PC that I was using as the rsync client because of a separate software project I'm working on.  The rsync client seemed to have worked for a brief period, but now appears broken and issues the following error message:

```

receiving file list...

115784 files to consider

rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(249)

rsync: connection unexpectedly closed (2722197 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(189)

>>> retry...

```

then rsync reiterates for the specified number of attempts, and issues this error:

```

!!! Rsync has not successfully finished.  It is recommended that you keep

!!! trying or that you use the 'emerge-webrsync' option if you are unable

!!! to use rsync due to firewall or other restrictions. This should be a

!!! temporary problem unless complications exist with your network

!!! (and possibly your system's filesystem) configuration.

```

what seems most odd about this situation was that i have not made any changes to the rsync server.  the only thing that i had changed was to reinstall gentoo on a client box upon which rsync had worked fine before. 

it seems that i have the rsync settings properly configured in  make.conf on the client PC:

```

SYNC="rsync://<lan-IP-address>/gentoo-portage"

RSYNC_RETRIES="3"

RSYNC_TIMEOUT=180

```

so i am at a loss to explain why the rsync appears to be broken.  i had considered that i may have encountered a timeout problem, because the client box is slow.  so i've increased the timeout values on both the client and the server to a generous 600 seconds.  unfortunately, that hasn't improved the situation.

diskspace does not appear to be a problem.

i'd appreciate it if anyone has a helpful idea.  thanks.

----------

## Jimmy Jazz

@Bob P

 *Quote:*   

> 
> 
> ```
> 
> receiving file list...
> ...

 

Hello,

that seems more a server side problem who has certainly disconnected since  :Smile: .

Have you increased your --timeout option to 300 or more ?

Look after the /etc/conf.d/rsyncd file or /etc/xinet.d/ directory if your are using xinetd and try to modify the --timeout for a more appropriate value.

Jj

----------

## Bob P

 *Jimmy Jazz wrote:*   

> that seems more a server side problem who has certainly disconnected since .
> 
> Have you increased your --timeout option to 300 or more ?
> 
> Look after the /etc/conf.d/rsyncd file or /etc/xinet.d/ directory if your are using xinetd and try to modify the --timeout for a more appropriate value.
> ...

 

yes, i thought i had mentioned that i've tried increasing the server and client timeouts to a generous value of 600.

what is really strange is that the server works fine with the other boxes on the network that are rsyncing.  one client box (2005.0) is having problems.  i've rebuilt rsync on the server and the effected client with no change in results.  i'm still trying to figure out what the error messages mean.  :Confused: 

----------

## Jimmy Jazz

@Bob P

 *Quote:*   

> 
> 
> yes, i thought i had mentioned that i've tried increasing the server and client timeouts to a generous value of 600.
> 
> 

 

Sorry i was absent-minded  :Smile:  and i thought you were using xinetd and in that case rsyncd.conf isn't used at all. You need to add it manually for the service.

The strange thing is that rsync caught an INT signal.

Could you trace the process with strace ? 

```

ps aux | grep rsync

strace -pRSYNC_PID

```

And perhaps, try ethereal as well.

Jj

----------

## Bob P

well, i've solved the problem.  it turns out that there was a corrupted baselayout ebuild in the portage tree.  i'm not absolutely certain that this was the problem, as i used the shotgun approach of updating the ebuild while rebuilding the entire installation with the --emptytree command.  baselayout was the only ebuild that changed, and the net result is that the "rsync over LAN" problem magically disappeared.  it seems that the problem was indeed on the client and not the server.  thanks for your help.

one thing that i would point out that may be worth updating in the guide:  the location of /etc/rsync/rsyncd.conf is now /etc/rsyncd.conf.

thanks again!  :Cool: 

----------

## bino

Bob P,

How do I do an emptytree? I'm having the same problem you had with a local rsync mirror. I googled a bit but can't find anything...

Thanks for any help!

Bino

----------

## zecora

```
Warning: unexpected read size of -864025635 in map_ptr

rsync: connection unexpectedly closed (2776821 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(189)

End: Thu Jun 30 20:10:45 CDT 2005

```

I get that msg when i try to tail and see the progress.  What is that @#$%?

Here is my /etc/rsync.conf

```
uid = nobody

gid = nobody

use chroot = yes

#limit access to private LAN's

hosts allow=192.168.2.1/255.255.255.0 10.0.0.0/255.255.0.0

hosts deny=ALL

max connections = 15

pid file = /var/run/rsyncd.pid

motd file = /etc/rsyncd.motd

#This will give you a separate log file

#log file = /var/log/rsync.log

#This will log every file transferred - up to 85,000+ per user, per sync

#transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

#If you need this, UPGRADE portage please!

#[gentoo-x86-portage]

#this entry is for compatibility

#path = /usr/portage

#comment = Gentoo Linux Portage tree

#exclude=distfiles/

[gentoo-portage]

#modern versions of portage use this entry

path = /usr/portage

comment = Gentoo Linux Portage tree mirror

exclude = distfiles/
```

Here is my /etc/init.d/rsync

```
#!/sbin/runscript

# Copyright 1999-2002 Gentoo Technologies, Inc.

# Distributed under the terms of the GNU General Public License, v2

or later

# $Header: /var/www/www.gentoo.org/raw_cvs/gentoo/xml/htdocs/doc/en/rsync.xml,v$

depend() {

need net

}

# FYI: --sparce seems to cause problems.

RSYNCOPTS="--daemon --safe-links --timeout=300"

start() {

ebegin "Starting rsync daemon"

start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 0 -$

eend $?

}

stop() {

ebegin "Stopping rsync daemon"

start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid

eend $?

}

```

I am just wondering why it wont finish?  Plz help

----------

## bino

I just tried it today and it fixed the io.c problem...

So I guess people should try a reboot on the client and perhaps re-running emerge sync on the server (my server does this automatically at night)

HTH

bino

----------

## zecora

 *bino wrote:*   

> I just tried it today and it fixed the io.c problem...
> 
> So I guess people should try a reboot on the client and perhaps re-running emerge sync on the server (my server does this automatically at night)
> 
> HTH
> ...

 

How does it do it automatically?  Plz do tell.

----------

## bino

Sorry, I'll make it clearer. My server has a crontab that runs emerge sync in the middle of the night. So I was trying to explain that the sync run in the night might have fixed the problem. Hope that helps

Bino

----------

## chratnox

First off, very nice howto. However, I'm not gonna use it since it nearly kills my p4 2.6 (read: apache/mysql get slow because of it). 

Anyway, when you're speaking about security, wouldn't it be better to run it as an other user then 'nobody'?

A snippet from the gentoo Security handbook: (Here)

 *Quote:*   

> 
> 
> #It is not a good idea to use nobody or nogroup - 
> 
> #for every service not running as root 
> ...

 

Just my 2 pennies  :Wink: 

----------

## piyo

 *JSharku wrote:*   

> Just a quick note on packages and distfiles; it's better to put the following in your rsyncd.conf:
> 
> ```
> 
> # excluding packages is optional, if you don't use --buildpkg you don't need it
> ...

 

Just a FYI, but this forum thread "emerge depclean says "You have no system list..."" shows what happens when there are no "/".

----------

## CarlUman

I'm working on this but when I start the process it doesn't write to the .pid file.  So when I try /etc/init.d/rsyncd stop it will not stop because there is no pid file.  I created the /var/run/rsyncd.pid file but got the same results.  Shouldn't I be able to see a rsyncd process in top?  Maybe for some reason the process isn't even starting even though there is no error on startup.

Thanks for any help

Carl

----------

## CarlUman

I tried rsync --deamon and nothing happens.  Nothing in ps or top.  What is up?

----------

## vico

hello every body  :Smile: 

i have a question about the config , if the server architecture and the rest of the boxes are different , 

do  i have to change my make.conf ? or something else.

----------

## flybynite

 *vico wrote:*   

> hello every body 
> 
> i have a question about the config , if the server architecture and the rest of the boxes are different , 
> 
> do  i have to change my make.conf ? or something else.

 

Not if your following the howto!

----------

## BertsInn

A recent portage update to rsync has changed both the /etc/rsyncd.conf and /etc/init.d/rsyncd files.  No surprise that this caused failures.  What is the best way to return to working state?  Just a matter of copying back your examples?

----------

## Bob P

you definitely need to restore your rsyncd.conf file if it was overwritten by etc-update following an emerge.  i think you should probably be okay using the new init.d file.

----------

## Salemixu

The Rsync Mirro has been working for quite some time now, but today i hasn't worked, it says this when i sync from the client:

```

>>> starting rsync with rsync://192.168.1.2/gentoo-portage...

>>> checking server timestamp ...

rsync: connection unexpectedly closed (0 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(189)

>>> retry ...

```

I redid what it says in this tutorial, so the configuration files are the same, and i have put two copies of rsyncd.conf, one in /etc and the other on /etc/rsync. Then i have noticed that happens the same as CarlUman, the rsync process is not loaded never, no matter if i run it by init.d or manuall by command line.

I have updated portage and updated the rsync package of the server side and still it doesn't load  :Sad: 

Any hints?

----------

## flybynite

The rsync troubleshooter says:

http://samba.anu.edu.au/rsync/issues.html

Q: Why does my transfer die with something like the following error?

rsync: error writing 4 unbuffered bytes - exiting: Broken pipe

rsync error: error in rsync protocol data stream (code 12) at io.c(463)

or

rsync: connection unexpectedly closed (24 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(342)

A: This error tells you that the local rsync was trying to talk to the remote rsync, but the connection to that rsync is now gone. The thing you must figure out is why, and that can involve some investigative work.

If the remote rsync is a daemon, your first step should be to look at the daemon's log file to see if it logged an error explaining why it aborted the transfer.

Beginning with version 2.6.3, rsync now does a better job of retreiving the error from the remote process than older versions of rsync, so you may wish to try upgrading and see if you now get sent the error message from the remote rsync.

There are several common causes for a remote rsync process going away:

    * The destination disk is full (remember that you need at least the size of the largest file that needs to be updated available in free disk space for the transfer to succeed).

    * A network error caused the connection to be dropped.

    * The remote rsync executable wasn't found.

    * Your remote-shell setup isn't working right or isn't "clean" (i.e. it is sending spurious text to rsync).

----------

## Salemixu

Now i tried again without modifying anything more, and it works :/

Perhaps it is fixes because of the reboots i have made in some days.

----------

## gronlund2

Hi.

I'm setting up a cvs backup/mirror. I need to temporarily disable cvs access on the remote host that is to be backed-up since a check-in during rsync may leave the back-upped repository in an inconsistent state.

So I need to execute a remote command in my back-up script (to disable cvs on the remote machine). Judging from the manuals it's a real pain to do this with ssh. If I generate keys and distribute the public key, I still need to supply a pass-phrase. And using ssh-agent seems like a real mess to me. 

Is there any easy way to execute a remote command in a script (i.e. automatically supply a password)?

Can I run rsync on the to be backed-up server instead? (No need to execute the remote command then.)

Grateful for your help.

/G2

----------

## thecooptoo

I did this 

```
grenada etc # ls -la |grep rsync

drwxr-xr-x    2 root root     4096 Feb 26 21:31 rsync

lrwxrwxrwx    1 root root       17 Mar  4 10:19 rsyncd.conf -> rsync/rsyncd.conf

```

and then set up the server 

```
grenada etc # grep ^[a-z] /etc/rsyncd.conf

uid = nobody

gid = nobody

use chroot = yes

max connections = 20

pid file = /var/run/rsyncd.pid

motd file = /etc/rsync/rsyncd.motd

transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

path = /opt/gentoo-rsync/portage

comment = Gentoo Linux Portage tree

path = /opt/gentoo-rsync/portage

comment = Gentoo Linux Portage tree mirror

exclude = distfiles

hosts allow=192.168.0.0/255.255.0.0 10.0.0.0/255.0.0.0

hosts deny=*

grenada etc #        

```

when i connect from the client 

```
dads distfiles # emerge sync

>>> starting rsync with rsync://192.168.0.1/gentoo-portage...

>>> checking server timestamp ...

This is rsync[number].[country].gentoo.org.

receiving file list ...

link_stat "metadata/timestamp.chk" (in gentoo-portage) failed: No such file or directory

0 files to consider

client: nothing to do: perhaps you need to specify some filenames or the --recursive option?

rsync error: some files could not be transferred (code 23) at main.c(653)

>>> retry ...

```

ive got onthe client 

```
dads distfiles # cat /usr/portage/metadata/timestamp.chk

Sat Mar  4 10:36:59 GMT 2006

dads distfiles #

dads distfiles # grep ^[SYNC]  /etc/make.conf

CHOST="i686-pc-linux-gnu"

CFLAGS="-march=athlon -O3 -pipe"

CXXFLAGS="${CFLAGS}"

SYNC="rsync://192.168.0.1/gentoo-portage"

```

and on the server FWIW 

```
grenada etc # cat /usr/portage/metadata/timestamp.chk

Sat Mar  4 10:38:33 GMT 2006

grenada etc #

```

----------

## flybynite

 *thecooptoo wrote:*   

> 
> 
> path = /opt/gentoo-rsync/portage
> 
> comment = Gentoo Linux Portage tree
> ...

 

Your not using the info from this howto.  I lost track of this thread for a while and yes, it does need some updating.

Is the server a gentoo box?  If so, the portage dir probably isn't located in /opt/gentoo-rsync.  If it isn't a gentoo box, the portage tree probably isn't in /opt/gentoo-rsync either.

----------

## radr

i been successfully using this since early 2k5.  never failed me

one sync to rule them all (heh)

and space saved on a 23 machine LAN = priceless

for reference my local sync mirror has a gentoo install that has lasted since late 2k1  (anyone remember pre 1.4 gentoo?)

----------

## ulises.dumount

I follow all the steps in this post but my problem is that i have a LAN and http-replicator working in a pc at this LAN, but all the PCs are working with a proxy to surf websites, and when i need to do a "emerge sync" i get this error:

emerge sync

>>> starting rsync with rsync://192.168.1.72/gentoo-portage...

>>> checking server timestamp ...

bad response from proxy - HTTP/1.1 502 Proxy Error ( El puerto SSL (nivel de sockets seguro) especificado no está permitido. El servidor ISA no está configurado para permitir peticiones de SSL de este puerto. La mayoría de los exploradores de Web utilizan el puerto 443 para peticiones de SSL.  )

rsync: failed to connect to 192.168.1.23: Success

rsync error: error in socket IO (code 10) at clientserver.c(88)

>>> retry ...

It seems like it want to connect to internet allways when i try to sync, but the proxy in this LAN didn't give me acces to internet via SSL in that port or something like that.

Any ideas??

----------

## flybynite

 *ulises.dumount wrote:*   

> 
> 
> It seems like it want to connect to internet allways when i try to sync, but the proxy in this LAN didn't give me acces to internet via SSL in that port or something like that.
> 
> Any ideas??

 

Is this error when trying to sync the box running the local rsync server or is it when a client tries to connect to your local rsync server?

If it is a client, then it looks to me that all requests are going to the other proxy.  You need to change the settings so that the rsync request doesn't go through the proxy.  I don't know anything about how your proxy is configured, but there should be a way to turn off the proxy for requests that stay local to your LAN.

If it is the box running the local rsync server, then can you emerge sync normally through your internet proxy?  I'm suggesting that the proxy your going through to the internet just won't allow you to emerge sync because of the port it uses.  I've run into that before and you need to look into web-rsync which uses the http port to sync the local rsync server box.  Then you can have clients connect to the local rsync server on your lan in the normal fashion.

----------

## ulises.dumount

Yes, this error is when a client tries to connect to my local rsync server. All the pcs are behind a proxy and are configured to go through it, all the clients are configured to emerge packages from my http_replicator in my pc. But the problem is when the clients want to sync because they want to go to internet to sync. In my http replicator i do it with "emerge-webrsync" and it goes through the proxy well. But i don't want all my clients to use the emerge-webrsync all the time they want to actualize!!!

Any ideas??

----------

## flybynite

How did you configure the clients to use the proxy?

Try this on the clients:

```

export RSYNC_PROXY=""

emerge sync

```

----------

## ulises.dumount

export RSYNC_PROXY="192.168.1.23"

emerge sync

>>> starting rsync with rsync://192.168.1.72/gentoo-portage...

>>> checking server timestamp ...

invalid proxy specification: should be HOST:PORT

rsync error: error in socket IO (code 10) at clientserver.c(8 :Cool: 

>>> retry ...

The same problem.

----------

## ulises.dumount

I found a partial solution!!!  This is it:

Always when i need to update the clients i need to do three steps:

1- export RSYNC_PROXY=""

2- emerge sync

3- export RSYNC_PROXY="yourproxy.com"

And the emerge works great with a http replicator working in the other pc of my LAN. I only need to configure the clients to not proxy the ip of my http replicator, but for now i do a scrip with these 3 steps in every pc client of my LAN.

Thanks a lot!!

----------

## melinux

Is there a problem with the latest rsync?

When I start it I get this error:

 * Starting rsyncd ...

rsync: --safe-links: unknown option (in daemon mode)

(Type "rsync --daemon --help" for assistance with daemon mode.)

rsync error: syntax or usage error (code 1) at options.c(857) [client=2.  [ !! ]

My rsync is the following:

net-misc/rsync-2.6.8-r2  USE="xinetd -acl -build -ipv6 -static"

It seems the option --safe-links has been removed. Does anyone know the best way to fix this in the init script- besides removing the --safe-links and timeout params of course (which I am temporarily doing)? I am using the init script from this thread.

----------

## KB7OEB

Do real portage mirrors use ssh while syncing?  I'm thinking its running encryption and causing the slow down.

----------

## zxy

@melinux

I have the same problem with the same version of rsync.

----------

## flybynite

 *melinux wrote:*   

> Is there a problem with the latest rsync?
> 
> When I start it I get this error:
> 
>  * Starting rsyncd ...
> ...

 

--safe-links is no longer needed or supported in daemon mode.  I've updated the howto to reflect this.  Just remove --safe-links and it should work fine.

----------

## katafitos

I have a problem with the Local Rsync Mirror

When I test sync with emerge --sync I get this

```
>>> Starting rsync with rsync://192.xxx.xxx.xxx/gentoo-portage...

>>> Checking server timestamp ...

rsync: read error: Connection reset by peer (104)

rsync error: error in rsync protocol data stream (code 12) at io.c(609) [receiver=2.6.9]

```

I can't figure out what's the cause of this. Any idea?

----------

## shrike

katafitos,

This is a long shot but check the RSYNC_TIMEOUT value in /etc/make.conf. Default is 30. I recall similiar error msgs when I had that set to a value of 5.

shrike

----------

## katafitos

 *shrike wrote:*   

> katafitos,
> 
> This is a long shot but check the RSYNC_TIMEOUT value in /etc/make.conf. Default is 30. I recall similiar error msgs when I had that set to a value of 5.
> 
> shrike

 

Thank you very much shrike!!   :Very Happy: 

that did the trick!!

----------

## Guybrush

For all with a local rsync mirror who have problems syncing and get the error message "rsync: on remote machine: --no-specials: unknown option":

I had a problem syncing my local portage mirror with the portage tree. It seems that the meaning of the rsync-parameter --devices has changed as of rsync version 2.6.7. Just add "--special" to the list of rsync parameters and it should work again. I also have filed a bug report. Hope that helps.

----------

## carpman

Hello, am getting following error on starting deamon

```

/etc/init.d/rsyncd start

 * Starting rsync daemon ...

rsync: --timeout=300: unknown option (in daemon mode)

(Type "rsync --daemon --help" for assistance with daemon mode.)

rsync error: syntax or usage error (code 1) at options.c(878) [client=2.6.9]     

```

```

cat /etc/init.d/rsyncd

#!/sbin/runscript

# Copyright 1999-2004 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: /var/cvsroot/gentoo-x86/net-misc/rsync/files/rsyncd.init.d,v 1.5 2007/02/23 11:33:59 uberlord Exp $

depend() {

        use net

}

# FYI: --sparce seems to cause problems.

RSYNCOPTS="--daemon  --timeout=300"

start() {

ebegin "Starting rsync daemon"

start-stop-daemon --start --quiet --pidfile /var/run/rsyncd.pid --nicelevel 15 --exec /usr/bin/rsync -- ${RSYNCOPTS}

eend $?

}

stop() {

ebegin "Stopping rsync daemon"

start-stop-daemon --stop --quiet --pidfile /var/run/rsyncd.pid

eend $?

}

```

cheers

----------

## zappatized

I have been trying to set up a local (personal) mirror for some time. There are a lot of guides but for now I'm just looking for a simple solution that I can implement right away. It was fairly straightforward to follow the official guide for local portage tree syncing, but trying to mirror distfiles proved more elusive. To save others the trouble of sorting through numerous forum posts I'd like to share my setup culled from those posts. This setup requires no packages beyond what is included in the base system.

```
# /etc/rsyncd.conf

uid = nobody

gid = nobody

use chroot = yes

munge symlinks = yes

# Limit access to private LAN (substitute with 10.0.0.0/24 depending on router, check with '$ route')

hosts allow = 192.168.1.0/24

max connections = 15

# This line is required by the /etc/init.d/rsyncd script

pid file = /var/run/rsyncd.pid

motd file = /etc/motd.rsyncd

# This provides a separate log file

log file = /var/log/rsync.log

# This will log every file transferred - up to 85,000+ per user, per sync

#transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

# Simple example for enabling your own local rsync server

[gentoo-portage]

   path = /usr/./portage

   comment = Funtoo Portage tree

   exclude = /distfiles /packages

# For distributing Portage sources (distfiles) to internal clients

[gentoo-distfiles]

   path = /usr/./portage

   comment = Gentoo Distfiles mirror

   include = /distfiles

```

No changes were made to /etc/init.d/rsyncd. /etc/conf.d/rsyncd contains the following options:

```
RSYNC_OPTS="--quiet --recursive --links --perms --times --devices --delete --nicelevel 15"

```

Start the rsyncd daemon on the local server and add to default runlevel. Edit make.conf for client computers:

```
SYNC="rsync://<localserver ip or name>/gentoo-portage"

GENTOO_MIRRORS="rsync://<localserver ip or name>/gentoo-distfiles ..."

```

I haven't set up a binhost yet but if I do I would start by duplicating the rsync distfiles module for the packages dir and then configure PORTAGE_BINHOST on the client(s) and PORTAGE_BINHOST_HEADER_URI on the server.

----------

## msst

Hmm, I am considering to set up a local mirror, but I do not want to run an additional daemon on my mini-box. Theres already sshd on it and all is set up.

How do I use rsync over ssh without setting up rsyncd? If I use the rsync program to access my mini-box, then it already copies things over using ssh. What to do to make emerge use this mechanism?

----------

## pgu

This is 10 years old. Is it still valid? Or are there easier ways obtain a local sync cache today?

----------

## Ant P.

There are a bunch of other options, like:

Share $PORTDIR over NFS, though this doesn't address the fact your initial rsync from upstream is still dangerously insecure.

Use webrsync and a caching HTTP proxy (which would also work for distfiles downloads).

Use one of the many git mirrors of the portage tree (or switch to Funtoo), and run a local git-daemon

"Easier" is always going to be subjective though.

----------

## steveL

distfiles is best done over sftp, ime; I don't recall the details but I used to have that setup for my old laptop, with the user allowed write access in order to save off new downloads.

Then rsync over ssh, separately, for portage tree.

----------

