# Local Mirror, storing the files needed for local clients...

## JohnerH

Hello,

My question is, 

There is a way to have a main server within the lan syncing all the other clients when it comes to portage,

But is there a way that we could have the same server hosting the files needed for the clients and fetching files needed for the clients when they get updated?

I.e.: client need foo file, server doesn't have the file, fetches it, then client gets foo file from server.... And so on with all the clients within the lan.

I've been looking for the answer to this within the forum but didn't find any... 

Sort of mini hosting server, 

I was thinking of doing this purely for bandwidth purposes... 

Thank you in advance,

J

----------

## Hu

Such a thing is certainly possible.  Three approaches come to mind.  One is supported almost transparently by Portage, but may have some performance degradation associated with it.  Another should have minimal degradation, but does not guarantee that you will never redownload the same package.  The third avoids degradation and redownloads, but is considerably more complex.

The first approach is to keep one copy of /usr/portage/distfiles and export it over NFS or CIFS.  This guarantees that when one client downloads it, all clients see it.  It requires that all machines have read/write access to the share.  Depending on your trust model, you may not like this.  I think there is minimal security risk since Portage validates checksums before using the file, but this is something to consider before using this approach.  This approach also carries some performance degradation, since the tarballs are fetched over NFS when they are unpacked.  The degradation is probably not significant in a LAN.

The second approach is to keep a master copy on one server and use frequent rsync runs from cron to synchronize the files to all the clients.  The rsync task would also need to copy any new downloads to the server.  This ensures that the files are local for unpacking, which avoids the potential degradation mentioned above.  However, if one client downloads the package, then another client needs the package before the next rsync run copies it over, then second client will redownload the package.  Also, this approach makes it rather difficult to get rid of old source tarballs, since you need to delete them from all systems at once.  Otherwise, the rsync run will repopulate from a peer.

The third approach involves setting up a caching proxy on the server and configuring the clients to fetch through it.  This provides local copies to clients on an as-needed basis, which should avoid both the degradation of non-local files and the redownload risk of the rsync approach.  Finally, this approach has the downside that it may not correctly cache packages which do not use source tarballs.  For example, MythTV ebuilds have a habit of using Subversion to update to a particular revision.  This means that no source tarball exists, and the caching proxy might refuse to cache the files since caching version control messages is usually a bad idea.

If you like any of these ideas and want more detail on how to configure them, just ask.

----------

## alex.blackbit

i believe what you want is just prevent clients from downloading one file more than one file because this is pointless, right?

there are two things you can do.

* if you have a 100G drive on your shelf, setup a gentoo mirror, so you get rid of the dups  :Laughing:  just a little joke, sorry.

* you can setup a NFS or CIFS or other network filesystem server that you mount as $PORTDIR/distfiles or maybe as $PORTDIR on your clients.

----------

## alex.blackbit

wow, Hu, your response was even faster and MUCH more detailed. how bad   :Crying or Very sad: 

----------

## JohnerH

The mirror is option is quite temptingly good (and yes I actually do have 100GB spare)...How to start?

Next option, I don't have any internal security issues (domestic LAN), so the NFS option would seem to be the more appropriate, specially considering that I have

everything hardwired. 

So gentleman how would I go about setting these 2 options up? 

J

----------

## Hu

For the mirror option, I think the best way to do it would be to configure all the clients to use an HTTP proxy.  Point the http_proxy at the server, and run net-proxy/squid on the server.  Configure Squid to cache content, and everything should just work from there.

For the NFS option, add a line to /etc/exports to allow clients to mount /usr/portage/distfiles:

/usr/portage/distfiles    192.168.0.0/24(rw,sync,no_root_squash,no_subtree_check)

Run exportfs -a to update the NFS server's records of what is exported.

On the clients, configure them to mount /usr/portage/distfiles from the server.  You can do this either using a static entry in fstab or by configuring an automounter.  A static entry is easier, but the automounter is considerably more flexible.  For fstab:

server:/usr/portage/distfiles      /usr/portage/distfiles      nfs             rw,auto,nodev,noexec,_netdev,hard,intr                     0  0

See man nfs for details on how the options affect the mount.  Caution: if you set it to auto, then the mount is automatically brought up at boot time.  This may be undesirable if you expect to boot the client while the server is unresponsive.  On the other hand, noauto will prevent an automatic mount at boot, but requires that you manually mount the directory before you have access to the files on the server.

For automounting, you need CONFIG_AUTOFS4_FS enabled in the kernel and the net-fs/autofs package to provide the userspace daemons.  Edit auto.master to enable NFS automounting.  For your purposes, you might prefer to write a new automount entry just for the server, rather than try to modify the shipped file.  The shipped file is very nice for doing mount point autodiscovery in conjunction with the /net automount point.  It is less suited for automounting where the only variable is availability of the server.

If you are new to NFS, there are several services you need to enable and start at boot.  Failure to do so causes problems ranging from mounts not working to mounts working very slowly.  On the server, rc-update add nfs default.  On the clients, rc-update add nfsmount default.  On both, rc-update add portmap default.

I think this is all you need, but I do not configure new NFS networks very regularly.  If I have missed some step, something should break.  In that case, please post back with details of what I missed and, if you have found a solution to it, what you had to do beyond what I mentioned here.

----------

## alex.blackbit

of course squid is a working and quite elegant option, because this way only the really needed packages are beeing downloaded.

if you still think about setting up a complete mirror, this should help you.

----------

## JohnerH

 *alex.blackbit wrote:*   

> of course squid is a working and quite elegant option, because this way only the really needed packages are beeing downloaded.
> 
> if you still think about setting up a complete mirror, this should help you.

 

Followed that guide completely borked my system and had to do a reinstall...

The crunch continues...

J

----------

## alex.blackbit

i made MANY mistakes in the early days but i never had to do a real reinstall. what happened? or better: what have you done?

----------

## JohnerH

 *alex.blackbit wrote:*   

> i made MANY mistakes in the early days but i never had to do a real reinstall. what happened? or better: what have you done?

 

Well, did,

```

rsync /usr/portage/ /mirror/gentoo-portage --exclude="distfiles" -t --recursive -P

```

And it started complaining about lack of space.

But I reckon I didn't mount the 2 SCSI drives I had reversed for it right.

I did it again today and it was fine, now I'm tackling the process of downloading the whole distfiles from a mirror.

using...

```

Prometeu ~ # rsync distro.ibiblio.org::gentoo /mirror/gentoo -t --recursive -P --verbose --delete --delete-after --links -q

```

Do you know a better way of this this?

And another question why is it that in the wiki page it makes you 

get proftpd? I don't see it being used anywhere...

----------

## bunder

 *JohnerH wrote:*   

> Do you know a better way of this this?

 

confirm that GENTOO_MIRRORS is set in make.conf and running emerge -feva world...  the -f is for fetchonly.  that way you aren't downloading unneeded distfiles.   :Wink: 

cheers

----------

## JohnerH

 *bunder wrote:*   

>  *JohnerH wrote:*   Do you know a better way of this this? 
> 
> confirm that GENTOO_MIRRORS is set in make.conf and running emerge -feva world...  the -f is for fetchonly.  that way you aren't downloading unneeded distfiles.  
> 
> cheers

 

but I would have to set a local distfile variable on make.conf (to /mirrors/gentoo) for that to work no?

And remember this for 3 machines not just the server...

----------

## bunder

 *JohnerH wrote:*   

>  *bunder wrote:*    *JohnerH wrote:*   Do you know a better way of this this? 
> 
> confirm that GENTOO_MIRRORS is set in make.conf and running emerge -feva world...  the -f is for fetchonly.  that way you aren't downloading unneeded distfiles.  
> 
> cheers 
> ...

 

since you're mounting /usr/portage/distfiles over nfs, it shouldn't be a problem to set it on all your machines... only thing that needs to be pointed to a local mirror would be the SYNC variable, if you're running a local rsync server (which is a very good idea if you have multiple machines on the same network)

take a look at mine:

```
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo"

SYNC="rsync://rsync/gentoo-portage"

PORTDIR_OVERLAY="/usr/local/portage"

```

cheers

----------

## think4urs11

the most easy way (biased opinion) still is to use http-replicator for this kind of task.

A central machine does a emerge --sync once per day, serves /usr/portage to internal clients via rsync and utilizies http-replicator for the distfiles.

As time goes by the cache of http-replicator fills up with all distfiles needed by your clients and nothing else.

----------

## JohnerH

 *Think4UrS11 wrote:*   

> the most easy way (biased opinion) still is to use http-replicator for this kind of task.
> 
> A central machine does a emerge --sync once per day, serves /usr/portage to internal clients via rsync and utilizies http-replicator for the distfiles.
> 
> As time goes by the cache of http-replicator fills up with all distfiles needed by your clients and nothing else.

 

And how would you set this up ?

Note: I'm not sharing things over NFS...

----------

## think4urs11

i've used HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0) back in time (i.e. 2004); since then this works without any issue.

----------

