# TIP: Compressing portage tree using squashfs

## adsmith

Here is a quick overview of how to compress the portage tree using squashfs.  I won't cover every little detail, as I presume your are comfortable configuring/compiling your kernel, using emerge, and dealing with simple bash scripts.

Why?

Wow, the portage tree is huge!    It takes around 500MB, which is a huge chunk of the storage space on many systems.     Now (unless you're crazy), you only sync your portage tree once a week or so.   In the mean time, it is essentially a read-only filesystem.  

Thus, we get the idea to put it in a compressed file system.  squashfs (http://squashfs.sf.net) gives us this option.   By following this howto, you'll have a compressed portage tree which only takes 25MB, and is also quite fast.

How?

Getting SquashFS

KERNEL

You need a kernel with supports squashfs and loop-back file systems, either compiled-in or as modules. 

kernel config:

```

Device Drivers -> Block Devices -> Loopback device support <M>

File systems  ->  Miscellaneous Filesystems  -> SquashFS <M>

```

NOTE: Squashfs is not yet in the vanilla-sources kernel, but the patch is very easy to do yourself.  Otherwise, gentoo-sources contains this patch already, so you may use that.  

Of course, recompile your kernel, fix grub, and reboot into it.

TOOLS

Next, you need to emerge the squashfs tools:

```

emerge squashfs-tools

```

PORTAGE CONFIG

Since squashfs is a read-only filesystem, we need to put distfiles somewhere else.  I chose /var/tmp/distfiles:

/etc/make.conf

```

DISTDIR="/var/tmp/distfiles"

```

MAKING THE FILESYSTEM

Suppose you have a current live portage tree at /usr/portage, and that you have already moved distfiles to $DISTDIR listed above.  The basic command to squash portage is this:

```

rm /usr/portage.sqsh

mksquashfs /usr/portage /usr/portage.sqsh -check_data

```

After running this command, you'll have a squashed copy of your portage tree in the file /usr/portage.sqsh.  

Now, what do we do with it?

Well, we want to mount this squashed file to /usr/portage.

To prepare for the next step, you need to move the live portage tree out of the way.  I suggest doing this for now, so you have a backup in case you decide to abort this procedure.

```

tar cvzf /usr/portage-backup.tar.gz /usr/portage

rm -rf /usr/portage/*

```

SYSTEM CONFIG

If you compiled loop and squashfs as modules, you need to load them at boot time.

/etc/modules.autoload.d/kernel-2.6

```

loop

squashfs

```

For now, you can just modprobe them.

You also need to edit /etc/fstab so that the new filesystem is mounted properly.

/etc/fstab

```

/usr/portage.sqsh    /usr/portage     squashfs     ro,loop     0 0 

```

Finally, to mount your shiny new compressed portage tree, do 

```

mount /usr/portage

```

That's the basic idea.  

Updates??

Ah, but the portage tree doesn't sit still forever!  At some point, you have to update it!  How do we do this?

Obviosuly, since the /usr/portage filesystem is read-only, a simple "emerge --sync" won't work.

There are various ways to do this.    

UPDATING WITH EMERGE

Here is a script which will go through the update process.  A serious downside of this method is that you'll need the full 500MB available for a short bit of time, since you can unly sync an uncompressed copy.   Here is a simple script which goes through the steps.  I'm sure it can be improved, but you get the idea.

/usr/local/sbin/emerge-sync-squash.sh

```

#!/bin/bash

## Where things go:

PORTDIR=/usr/portage

PORTSQSH=/usr/portage.sqsh

PORTTMP=/var/tmp/portage-tmpcopy

## First, make sure the squashed protage is mounted

mount -o remount $PORTDIR &> /dev/null || mount $PORTDIR

## make sure there is no old tmp copy in the way

rm -rf $PORTTMP

## Make an uncompressed copy 

cp -ra $PORTDIR $PORTTMP 

## sync to the uncompressed copy

FEATURES="-fixpackages" PORTDIR="$PORTTMP" emerge --nospinner --sync

## uncomment re-make the "q" database, if you use it.  (portage-utils)

#PORTDIR=$PORTTMP  q -r

## squash! and remount

umount $PORTDIR

rm -f $PORTSQSH

mksquashfs $PORTTMP $PORTSQSH -check_data

mount $PORTDIR

## cleanup

rm -rf $PORTTMP

##  Make sure local databases are up to date

emerge --nospinner --metadata 

update-eix

```

That's it.  As you can see, the idea is straightforward, but you need to full disk space available somewhere, if only for 5 minutes.

UPDATING FROM SOMEONE ELSE

If you don't have that 500M available (if you did, then this wouldn't be a very useful exercise!), then the only option is to only get compressed copies.  Eventually, perhaps, it will be possible to produce a squashfs directly from a tarball, but this is not yet possible.  So, you have to get the squashed image directly.

For a short while, I'll provide pre-squashed copies, updated every day or two, at 

http://www.math.duke.edu/~adsmith/portage.sqsh

However don't expect these to last forever.  If my sysadmin complains about traffic, I'll have to take them down.

To update with these pre-squashed copies, simply to

/usr/local/sbin/emerge-sync-squash2.sh

```

#!/bin/bash

umount /usr/portage

wget http://www.math.duke.edu/~adsmith/portage.sqsh -O /usr/portage.sqsh

mount /usr/portage

emerge --metadata

update-eix

```

I'll think about how to distribute these using binary diffs, but for now, they're only 25MB, so it's not big deal.

WHAT ABOUT SHARING THE TREE??

If you want to share the tree over a LAN, you have to be a bit more careful.  NFS doesn't play with squashfs, so you have to use network block devices (config kernel and emerge nbd)  to export the /usr/portage.sqsh image file to your other systems.  

basically, you do this:

server:

```

nbd-server <port> /usr/portage.sqsh

```

client:

```

modprobe nbd

nbd-client <server> <port> /dev/ndb0

mount /dev/nbd0 /usr/portage

```

Enjoy!

UPDATE:  A few pages ahead, several people are reporting success by using unionfs over squashfs to make a RW portage tree, which can then be re-squashed after a sync.  I personally have not had success with this, but there are good instructions ahead.

[edit]: changed -noappend to 'rm file', so mksquashfs doesn't make portage a subdirectory.Last edited by adsmith on Sat Mar 25, 2006 3:25 pm; edited 2 times in total

----------

## frostschutz

 *adsmith wrote:*   

> Why?
> 
> Wow, the portage tree is huge!    It takes around 500MB, which is a huge chunk of the storage space on many systems.

 

500MB is huge  :Question: 

I'm somewhat unsatisfied by this description. Hard disks are so huge nowadays (they broke like the 100GB barrier ages ago) that it seems hardly imaginable that 500MB would pose that much of a problem. The only application which would be more limited in terms of disk space that I can think of, would be a virtual server. And for a Gentoo installation/update on those I would go about in a completely different manner.

So can you give a real world example where SquashFSing the Portage tree would actually be useful? It has so many downsides (hard to update, requires a lot of free space anyway during update), plus the Portage tree isn't even the biggest part about Gentoo (my distfiles usually grows bigger than that, as well as the work directory during a compile).

If you need the free space only while you are not updating your system, and you are only updating like once a week, and you don't mind to download a tarballed tree then, you could just delete the whole thing (and thus achieve a even better compression ration than SquashFS without all the hassle).  :Laughing: 

----------

## adsmith

Sure -- a good question. 

 Nowadays, when many desktop machines are amd64 mid-towers with 500 GB of SATA drives, we forget that not all machines have the same flexibility.      Sometimes we can't or don't want to buy a new drive for an older machine which is otherwise still useful.  

Case in point:  Many small ("legacy") laptops have 8, 10 or 12G drives.  Basic system tools take up to 2GB; A window manager and X programs take another 1 or 2GB.  Saving 0.5GB can be crucial!  (As for distfiles, I clean it after every emerge, so it's never any bigger than a few dozen MB)

Such a machine (like mine!) might only be updated every several months, when there is a day it can connect to a LAN with fast distcc hosts.   However, a recent portage tree still has to be kept on the laptop in order for various system utilities to function properly.

So, Yes, this is a very specific application which isn't of much benefit for many people, but in some cases, I've found it to be quite useful.  

Another point:  In some sense, when slow hard drvies are in use, this obviates the need for putting the portage tree on a seperate reiser partition, et cetera,  for speed.  The whole 25MB tree can usually fit in RAM all at once, and squashfs is really quite fast, so basic emerge functions are pretty darn quick, even with slow hardware.  This is a sort of half-assed, cock-eyed way of getting the databased portage tree we all hope for some day.     :Wink:    well, okay, hardly.....

----------

## adsmith

Also, note that in a parallel forum, one of our fellow gentooers seems to haver patched squashfs to read directly from tar.gz;  hence, (once it really works) you can just grab the binary-diff portage snapshot and run from there.  No extra space needed!

https://forums.gentoo.org/viewtopic-p-2873343.html#2873343

----------

## kuku

what do you think about using unionfs to make the squashed image writable , then sync it and create new squashed image (you dont need the extra space  to unpack it every time you sync)

----------

## adsmith

That's an interesting idea!  I tried playing with unionfs a little bit for NFS purposes.

I'll give this a try later this afternoon.

----------

## adsmith

Okay, I've tried the unionfs idea.  

The plan is this

1. start with a squashfs image, and mount it at /usr/portage-ro

2. mkdir /usr/portage-rw

3. mount -t unionfs -o dirs=/usr/portage-rw=rw:/usr/portage-ro=ro  none /usr/portage

4. emerge --sync

5. remake squashfs image and remount.

This would be really slick if it worked, but I keep getting kernel Oopses on step 4.  Is there a known problem with using rsync with unionfs??

----------

## makomk

 *adsmith wrote:*   

> Also, note that in a parallel forum, one of our fellow gentooers seems to haver patched squashfs to read directly from tar.gz;  hence, (once it really works) you can just grab the binary-diff portage snapshot and run from there.  No extra space needed!
> 
> https://forums.gentoo.org/viewtopic-p-2873343.html#2873343

 

Unfortunately, that's not exactly how it works; I wasn't terribly clear. (Sorry about that - I was half-asleep at the time.) Basically, I patched mksquashfs to generate a squashfs filesystem directly from a portage tarball. Unfortunately, due to the way I did it, you have to use uncompressed tarballs (~200Mb), otherwise it takes forever. (Of course, emerge-delta-webrsync chucks uncompressed tarballs around anyway!) If I understand how squashfs works correctly, it should be possible to do it efficiently from compressed tarballs, but it'd probably require rewriting mksquashfs (not fun)*.

On the other hand, my solution works now - I'm running portage from a squashfs generated using it at the moment. I've got a modified version of emerge-delta-webrsync called emerge-delta-squashfs which runs the modified mksquashfs instead of tarsync - quite neat actually. I don't have anywhere to host the patches, but anyone who's interested should feel free to PM or otherwise contact me, and I'd be happy to mail them (7K total).

Basically, you install emerge-delta-squashfs and the modified mksquashfs, set up your system as above (except don't emerge squashfs-tools, and replace /usr/portage.sqsh with /var/delta-webrsync/portage.squashfs everywhere), then run emerge-delta-squashfs every time you want to sync portage. Free space requirements are the same as emerge-delta-webrsync, minus the 500Mb /usr/portage.

* Besides, it might be easier and better to find a way of generating deltas between squashfs filesystems.

----------

## adsmith

 *Quote:*   

> 
> 
> ... (once it really works) ...
> 
> 

 

Yeah, I got that picture.  I guess I didn't convey it to others very clearly.

I'd love to play with the patches.    You can guess my email address from the weblink I provide above, and I could host them there as well, if others want to play.

----------

## adsmith

Ah! Yet another way to do it!

We can use the "--compare-dest" option in rsync.  To do this, simply add a corresponding line around line number 2434 of /usr/lib/portage/bin/emerge: e.g.,

```

            "--compare-dest='/usr/portage-ro'",

```

if the squashfs is mounted at /usr/portage-ro

it's still a little shakey, but it looks promising.

----------

## makomk

 *frostschutz wrote:*   

> So can you give a real world example where SquashFSing the Portage tree would actually be useful? It has so many downsides (hard to update, requires a lot of free space anyway during update), plus the Portage tree isn't even the biggest part about Gentoo (my distfiles usually grows bigger than that, as well as the work directory during a compile).
> 
> If you need the free space only while you are not updating your system, and you are only updating like once a week, and you don't mind to download a tarballed tree then, you could just delete the whole thing (and thus achieve a even better compression ration than SquashFS without all the hassle). 

 

Not if the reason you need the extra 500Mb space is to *build* the packages in (some of them are pretty hefty - the main thing I had/have space problems with was emerges). Also, part of the reason I like my current experimental setup is that I just have to run emerge-delta-squashfs (and hope it works) instead of running emerge-delta-webrsync like I did before - easy.

----------

## mdeininger

That's quite a nice one really, and I'd like to back you guys up on the space thing, the computer I have at work only has like 8 gigs of hd and saving 500 megs of disk space really can be crucial  :Smile: 

albeit i got around that slightly differently, I did a normal install, copied everything to our server and made the server create squashfs images that i put on my machine's harddrive with dd. then i made the machine boot with an initrd that mounts the read-only image and creates three ramdisks, one for /tmp and two for /etc and /var. it then combines the /etc and /var from the image with the ramdisks using unionfs so that every change goes to the ramdisk. works like a charm, the image is 700mb including gnome and some office stuff and i get lotsa disk space for my /home. Updates are rather fast too, since i can just chroot on the thing on the server, so if you're tight on disk space, you could try this method instead of only compressing /usr/portage  :Wink: . It might be me but i also think it's faster than using reiserfs, at least on that old box...

----------

## electrofreak

My laptop only has a 2GB hard drive, so most definately 500MB helps. Unfortunately, the portage tree doesn't actually take up 500MBs, but only like 100-150MB. 'du .hs /usr/portage' isn't very accurate. However, even saving about 100MB still helps. But I don't really want to do this if it still needs the space to sync and such. so when the unionfs idea works, I may go ahead and do that.

----------

## adsmith

While du doesn't reflect the size of the actual files, it does show how much disk space is allocated for those files.  From the perspective of wasted disk space, it's correct.

Between the websync script mentioned above, and some other ideas, we're working on better sync/update methods.

----------

## Cappo

When I try to mount the squashfs partition, I get this error in the system log:

SQUASHFS error: zlib_fs returned unexpected result 0xfffffffd

SQUASHFS error: Unable to read cache block [1e85440:90f]

SQUASHFS error: Unable to read inode [1e85440:90f]

SQUASHFS error: Root inode create failed

Any idea what this means and how to fix it?

----------

## Cappo

Nevermind. After restarting my computer, the error message went away.

----------

## lazy_bum

Is it possible to do the same thing with kernel sources? (-:

----------

## adsmith

errr.. how would you build them if the directory is read-only????

----------

## lazy_bum

 *adsmith wrote:*   

> errr.. how would you build them if the directory is read-only????

 

Well, my kernel is about 2 months old (or even more). Only few modules, almost everything build-in... so that's why i wonder about "squashing" kernel (it's ~300 mb, a lot of small files).

----------

## adsmith

I don't understand what you mean at all.  

You can always delete the /usr/src/linux-* sources if you so desire, since that's only needed for building the kernel. The main kernel image, /boot/kernel-*, is already bz2-compressed.  I'm not sure if the modules in /lib/modules/* are compressed or not, but if they aren't, then they can't be.

At the very least, you can "cd /usr/src/linux" and "make clean" to clean up everything but the original source.

----------

## yoyo

Good tips imho.

But don't forget the "PKGDIR" which, like DISTDIR, needs to be mounted  with read/write access.

My 0.02 cents.

----------

## bibi.skuk

 *adsmith wrote:*   

> I don't understand what you mean at all.  
> 
> You can always delete the /usr/src/linux-* sources if you so desire, since that's only needed for building the kernel. The main kernel image, /boot/kernel-*, is already bz2-compressed.  I'm not sure if the modules in /lib/modules/* are compressed or not, but if they aren't, then they can't be.
> 
> At the very least, you can "cd /usr/src/linux" and "make clean" to clean up everything but the original source.

 

But in order to build modules, and some apps, you need to have a /usr/src/linux/... maybe only the config file, but i don't know exactly.

----------

## electrofreak

About compressing the linux sources: I actually tar my current sources up and always unmerge older sources. When I come across something that needs them to build their stuff, then I simply untar them, let it build that delete them (leaving the tar there still). Yeah, it's a little extra work.

If we could get a compressed portage tree that can be written to and updated as if it wasn't compressed, then, yeah, it'd be completely awsome to do the same thing with the linux sources.

----------

## frank_einstien

Hi guys

How do I get use squashfs without recompiling my kernal? Just asking...

Thanks

----------

## drwook

Short answer is you don't  :Wink: 

Seriously though, compiling a kernel isn't a black art or anything.  Give it a go if you haven't before.

----------

## lazy_bum

 *frank_einstien wrote:*   

> How do I get use squashfs without recompiling my kernal? Just asking...

 

Why not? You can add a module and then...

```
make modules && make modules_install
```

PS. squashed my portage and it works fine. With my 300 MHz on a home router i haven't noticed big slowdown (well "emerge sync" takes about... half an hour), and ~100 mb is nice cmp to >500 mb. Nice tip, thx.

----------

## electrofreak

 *lazy_bum wrote:*   

>  *frank_einstien wrote:*   How do I get use squashfs without recompiling my kernal? Just asking... 
> 
> Why not? You can add a module and then...
> 
> ```
> ...

 

I do stuff like that all the time with modules. Works fine.

As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method??

----------

## Liviu

Alright! Really nice guide!

I got only around 6 gig for my linux partition on my Laptop (60G Harddrive).

I use my Home-Made P3-533 Gentoo Router to make my Squashed Portage Snapshots! It really doesn`t take too long.. perhaps 15 mins ?!?

Works perfectly for me so i don`t need to make one from a tar-snapshot or sync it and then squash it.

In my opinion this whole idea is very nice! I will distribute a copy of the .sqsh too .. I`ll just configure my cron...........*workinprogress*

----------

## lazy_bum

 *electrofreak wrote:*   

> 
> 
> As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method??

 

Used the guide method. Router is online all the time so the emerge-sync-squashed is a cron job in the middle of the night.

----------

## bld

 *frank_einstien wrote:*   

> Hi guys
> 
> How do I get use squashfs without recompiling my kernal? Just asking...
> 
> Thanks

 

NOTE: The squashFS patcheset is available for 2.6.13 and below. Hence I managed to rename the kernel version inside the patch. You should do this if your kernel is above 2.6.13 .. if something is not clear just ask.

# cd /root

# uname -r

2.6.14.2

# wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz

# tar zxvf squashfs2.2-r2.tar.gz

# cd squashfs2.2-r2/linux-2.6.13

# sed 's/2.6.13/2.6.14.2/g' squashfs2.2-patch  > squashfs2.2-ready-patch

# cp  squashfs2.2-ready-patch /usr/src/linux/ && cd /usr/src/linux && patch -p1 <  squashfs2.2-ready-patch

done

choose, loop and squashFS as described below.. then type "make modules && make modules_install" inside the /usr/src/linux dir.. and you're all set.. 

Then follow the steps described above  :Smile: 

----------

## Liviu

Okay Folks !

A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):

http://home.arcor.de/liviufl/portage.sqsh/

I got 1 Gigabyte of Traffic for that webspace. It should be enough for around 33 Downloads  :Wink: 

have fun !

----------

## emj

Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all. 

I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.

Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.

adsmith: I don't know why you got a kernel oops while testing unionfs. Perhaps it's because you didn't set the read/write layer to a tmpfs? Shouldn't matter, but... Or maybe it's just a bug in the code in your kernel... maybe try another kernel? unionfs has to be relatively stable, ordinarily, since most livecd are using it now as a fully filetree overlay, sooo... But there, it's a tmpfs overlay, again, so maybe it just likes that better. tmpfs is fine, really, since the changes accumulated during emerge syncs seem to only amount to a few megabytes of data in the ramdisk directory, which is cleared immediately after the sync, soo.

Anyway, after some testing, could you put this script into the main post? Thanks.

```

#!/bin/bash

#

#myemerge_sync.sh - script to emerge --sync from a squashfs

#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location

#emerge --sync's, then re-squashes portage and remounts

#also runs an update-eix to update eix database

#

#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.

#I use the ArchCK ebuilds (search the gentoo forums) to get the above in the kernel.

#Also, remember that it's very important to have DISTFILES set somewhere

#other than /usr/portage/distfiles

#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf

#

#Preparatory steps to running this the first time are as follows:

#

#emerge squashfs-tools

#mkdir /images

#mksquashfs /usr/portage /images/usr.portage.sqfs

#mv /usr/portage /usr/portage_

#mkdir /usr/portage

#echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab

#mount /usr/portage 

#

#...then try running this script. When you confirm that everything's working you

#can rm -rf /usr/portage_ to remove the old uncompressed portage tree, or restore

#it if the script doesn't work for you.

#

#~Ev

#

#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.

#100M should suffice unless you've failed to sync for a very, very long time.

#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.

#It will be unloaded when done sync'ing.

RAMSIZE="100M"

IMAGEDIR="/images"

RAMDISKDIR="$IMAGEDIR/ramdisk"

#######################

mkdir -p $IMAGEDIR &> /dev/null

mkdir -p $RAMDISKDIR &> /dev/null

rm -rf $RAMDISKDIR/*

#only proceed if squashfs already mounted..

if [ "`cat /proc/mounts | grep squashfs | grep \/usr\/portage\ `" ]

then umount $RAMDISKDIR &> /dev/null

mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR

echo "** tmpfs of $RAMSIZE mounted on: $RAMDISKDIR"

mount -t unionfs -o dirs=$RAMDISKDIR=rw:/usr/portage=ro none /usr/portage

echo "** unionfs overlay mounted over /usr/portage"

emerge --sync

update-eix

mv -f /images/usr.portage.sqfs /images/usr.portage1.sqfs

mksquashfs /usr/portage /images/usr.portage.sqfs

umount -dl /usr/portage &> /dev/null

umount -dl /usr/portage &> /dev/null

umount -dl $RAMDISKDIR &> /dev/null

mount -t squashfs -o ro,loop /images/usr.portage.sqfs /usr/portage

fi

```

Last edited by emj on Tue Jan 10, 2006 1:31 am; edited 1 time in total

----------

## electrofreak

looks like a nice script. I will try it soon (when I have time)

----------

## opentaka

great howto, this helped my router server with 5GB HDD. yes, 500MB is huuuge  :Very Happy: 

----------

## bld

 *emj wrote:*   

> Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all. 
> 
> I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.
> 
> Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then  resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.
> ...

 

While trying your script I got those errors:

 *Quote:*   

> 
> 
> mkstemp "/usr/portage/sys-apps/baselayout/.Manifest.l5EAf8" failed: Read-only file system        1317 100%    0.00kB/s    0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r1.ebuild.WlejER" failed: Read-only file system       18930 100%    0.00kB/s    0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r2.ebuild.bvd3yB" failed: Read-only file system       19000 100%    0.00kB/s    0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.14.ebuild.SuXJZl" failed: Read-only file system       18962 100%    0.00kB/s    0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.12.0_pre13-r1.ebuild.ZvcoW6" failed: Read-only file system

 

Are you sure that we need to mount the fs as read only?

----------

## chojin

al this looked very appealing to me so I tried it right away.. 

It all seemed to work alright until I tried the above myemerge-sync.sh script.

The script seems to mount the tmpfs and the unionfs 

and while the scripts is running I see this in df -h:

```
 # df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/sil_afabafddahff5

                       19G   11G  6,7G  62% /

udev                  252M  292K  252M   1% /dev

/dev/mapper/sil_afabafddahff6

                      9,2G  8,3G  443M  96% /home

/dev/mapper/sil_afabafddahff8

                      184G  175G     0 100% /mnt/backup

/dev/mapper/sil_afabafddahff9

                      143G  131G  5,3G  97% /mnt/storage

/dev/mapper/sil_afabafddahff2

                       19G   11G  8,6G  55% /mnt/windows

none                  252M     0  252M   0% /dev/shm

none                  252M     0  252M   0% /dev/shm

/images/usr.portage.sqfs

                      127M   28M   99M  22% /usr/portage

filekeeper:/mnt/storage

                      113G  106G  1,9G  99% /mnt/filekeeper

tmpfs                 100M  1,2M   99M   2% /images/ramdisk

none                  127M   28M   99M  22% /usr/portage

```

and rsync starts, without problems.. doing some files and suddenly

```
.....

dev-libs/libdaemon/libdaemon-0.8.ebuild

         980 100%    0.00kB/s    0:00:00

dev-libs/mpfr/ChangeLog

        1231 100%    0.00kB/s    0:00:00

dev-libs/mpfr/Manifest

         980 100%    0.00kB/s    0:00:00

dev-libs/mpfr/files/2.2.0/patch06

         835 100%    0.00kB/s    0:00:00

dev-libs/mpfr/files/2.2.0/patch07

        1651 100%    0.00kB/s    0:00:00

dev-libs/mpfr/files/digest-mpfr-2.2.0_p7

         120 100%    0.00kB/s    0:00:00

metadata/cache/media-tv/

metadata/cache/media-video/

metadata/cache/net-analyzer/

metadata/cache/net-dialup/

metadata/cache/net-dns/

metadata/cache/net-firewall/

metadata/cache/net-fs/

metadata/cache/net-ftp/

metadata/cache/net-im/

metadata/cache/net-irc/

metadata/cache/net-libs/

metadata/cache/net-mail/

metadata/cache/net-misc/

rsync: connection unexpectedly closed (3238834 bytes read so far)

rsync error: error in rsync protocol data stream (code 12) at io.c(189)

>>> retry ...

>>> Starting retry 1 of 3 with rsync://81.223.20.162/gentoo-portage

>>> checking server timestamp ...

This is mirror.inode.at, your local friendly mirror in the neighbourhood.

Connection sponsored by inode. See http://www.inode.at/

Have a look at http://mirror.inode.at/ for further information and statistics on the server.

Please contact mirror@inode.at in case of problems.

*** Checksumming (-c) is disabled, so don't use it ***

Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162

receiving file list ... 

1 file to consider

timestamp.chk

          32 100%    0.00kB/s    0:00:00

Number of files: 1

Number of files transferred: 1

Total file size: 32 bytes

Total transferred file size: 32 bytes

Literal data: 32 bytes

Matched data: 0 bytes

File list size: 32

Total bytes written: 223

Total bytes read: 542

wrote 223 bytes  read 542 bytes  510.00 bytes/sec

total size is 32  speedup is 0.04

This is mirror.inode.at, your local friendly mirror in the neighbourhood.

Connection sponsored by inode. See http://www.inode.at/

Have a look at http://mirror.inode.at/ for further information and statistics on the server.

Please contact mirror@inode.at in case of problems.

*** Checksumming (-c) is disabled, so don't use it ***

Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162

receiving file list ... 

133128 files to consider

./

app-admin/

app-admin/conky/

app-admin/conky/files/

app-admin/denyhosts/

app-admin/gentoo-bugger/

app-admin/gentoo-bugger/files/

app-admin/modlogan/

app-admin/modlogan/files/

app-admin/superadduser/

app-admin/superadduser/files/

app-admin/usermin/

app-admin/usermin/files/

app-admin/webalizer/

app-admin/webalizer/files/

app-admin/webalizer/files/2.01.10/

app-admin/zope-config/

app-admin/zope-config/files/

app-arch/

app-arch/bsdsfv/

app-arch/bsdsfv/files/

app-arch/sharutils/

app-arch/sharutils/files/

app-backup/

app-backup/sarab/

app-backup/sarab/files/

app-benchmarks/

app-cdr/

app-cdr/k9copy/

app-cdr/k9copy/files/

app-crypt/

app-crypt/heimdal/

app-crypt/heimdal/files/

app-crypt/johntheripper/

app-crypt/johntheripper/files/

app-dicts/

app-dicts/aspell-et/

app-dicts/aspell-et/files/

app-doc/

app-doc/doxygen/

app-doc/edox-data/

app-doc/xmltoman/

>>> retry ...

>>> Starting retry 2 of 3 with rsync://62.197.40.130/gentoo-portage

>>> checking server timestamp ...

,----------------

|  Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!

|  

|  For info on Gentoo Linux, see <http://www.gentoo.org/>.

|  

|  For more on this server, see <http://vegetable.org.uk/>.

|  For server-admin policy, see <http://gentoo.vegetable.org.uk/>.

|  

   Please note: an excessive rate of connections to this server does not

   benefit anyone - a couple of times per day is normally sufficient. A 

   small amount of profiling is undertaken to determine potential abusers 

   and rate-limiting will be enforced if needed.

receiving file list ... 

1 file to consider

timestamp.chk

          32 100%    0.00kB/s    0:00:00

Number of files: 1

Number of files transferred: 1

Total file size: 32 bytes

Total transferred file size: 32 bytes

Literal data: 32 bytes

Matched data: 0 bytes

File list size: 32

Total bytes written: 223

Total bytes read: 699

wrote 223 bytes  read 699 bytes  1844.00 bytes/sec

total size is 32  speedup is 0.03

,----------------

|  Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!

|  

|  For info on Gentoo Linux, see <http://www.gentoo.org/>.

|  

|  For more on this server, see <http://vegetable.org.uk/>.

|  For server-admin policy, see <http://gentoo.vegetable.org.uk/>.

|  

   Please note: an excessive rate of connections to this server does not

   benefit anyone - a couple of times per day is normally sufficient. A 

   small amount of profiling is undertaken to determine potential abusers 

   and rate-limiting will be enforced if needed.

receiving file list ... 

133128 files to consider

./

app-admin/

app-admin/modlogan/files/

app-admin/superadduser/files/

app-admin/usermin/files/

app-admin/zope-config/files/

app-arch/

app-arch/bsdsfv/files/

app-benchmarks/

app-cdr/

app-crypt/

app-crypt/heimdal/files/

app-dicts/

app-doc/

io timeout after 358 seconds - exiting

rsync error: timeout in data send/receive (code 30) at io.c(109)

io timeout after 180 seconds - exiting

rsync error: timeout in data send/receive (code 30) at io.c(109)

```

And from that point I see this with ps -A |grep rsync:

```
$ ps -A |grep rsync

 8698 pts/0    00:00:02 rsync

 8765 pts/0    00:00:00 rsync <defunct>
```

THen when shutting down the pc, it can't unmount the loopback filesystems, complains a lot, even lets me use the emergency maintenance console (or press Ctrl-D to continue)...probably because rsync is keeping the filesystem busy and doesn't want to get killed.

Does anybody know why this is happening?

----------

## adsmith

For some reason, the email notification hadn't been telling me about these recent posts, so I missed this discussion.

Thanks for the help with unionfs.  I'll try it out again shortly.  If it works, I'll toss your script back up in the main post.

----------

## emj

chojin:

I can only presume that perhaps you don't have enough free, unswapped, ram available. Either that, or your unionfs is bugged somehow. did "dmesg" give any errors? What kernel are you using? I'm using 2.6.14-archck5. Haven't tried it with any other. I'd be surprised if unionfs is broken on many kernels... I suspect it's just the same code copy in the various new kernels...

It would be useful to check and see the output of df -h during the bugged part of the emerge --sync .. that could let us know how full the ramdisk is. Oh, and a free -m also, so we can see the ram situation.

I'm operating with a gig of ram, just for reference... but at the end of a normal emerge --sync the ramdisk for the unionfs overlay is only using 11mb, soo...

bld:

The fact that it's complaining that the file system is read-only is probably an indication that you don't have unionfs support enabled / working. Try checking to make sure it's enabled in the kernel, and/or modprobing it if you selected it as a module.

I suppose I could add something to the script that does "grep /usr/src/linux/.config -ie UNIONFS | grep "yes"" or something...

Anyway, thanks for testing! If it's working for some, post and say so  :Razz:  I'm continuing to use the script and it's working great.

~Ev

----------

## chojin

I have a total 512mb Ram and 2Gb of Swap so it would surprise me if I ran out of ram/swap with this script.

As you can see in the output of df -h in my previous post only 1.2M was taken on the ramdisk at the moment rsync started to act funny.

I'm running kernel 2.6.14-gentoo-r5-mh2 #9 PREEMPT but I didn't find anything about unionfs in the kernel config itself. I had to emerge sys-fs/unionfs-1.0.14 (which is masked ~x86) and this gave me the unionfs module, which I load in /etc/modules.autoload/kernel-2.6 together with the squashfs and loop module.

this is the output of free -m while rsync was crashing

```
$ free -m

             total       used       free     shared    buffers     cached

Mem:           503        495          7          0        153        103

-/+ buffers/cache:        239        264

Swap:         1906          0       1906

```

and df -h at the same moment

```
$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/sil_afabafddahff5

                       19G   12G  6,0G  66% /

udev                  252M  292K  252M   1% /dev

/dev/mapper/sil_afabafddahff6

                      9,2G  8,5G  277M  97% /home

/dev/mapper/sil_afabafddahff8

                      184G  175G     0 100% /mnt/backup

/dev/mapper/sil_afabafddahff9

                      143G  131G  5,3G  97% /mnt/storage

/dev/mapper/sil_afabafddahff2

                       19G   11G  8,6G  55% /mnt/windows

none                  252M     0  252M   0% /dev/shm

/images/usr.portage.sqfs

                      127M   29M   99M  23% /usr/portage

filekeeper:/mnt/storage

                      113G  106G  1,9G  99% /mnt/filekeeper

tmpfs                 100M  1,9M   99M   2% /images/ramdisk

none                  127M   29M   99M  23% /usr/portage

```

As you can see, I have plenty of free swap space and only 1.9M used on the ramdisk...

----------

## adsmith

I had a kernel panic yet again, about half-way through the sync.  My r/w image was just a directory on the HD, so no risk of running out of space. I'm using vanilla 2.6.14.2.  I'll look into it later this week, but I'll be busy for a few days.

----------

## emj

chojin:

Oh, ok, sorry. I thought that your previous df -h was at the point where the sync went buggy.

Hmmm, weird. I'll switch to a few different kernels and see how it works then. Oh, and I'll try the unionfs in portage. Maybe the problem's there. There's gotta be some buggy unionfs code somewhere. I'll post later tonight with results.

adsmith:

The kernel panic may be 'cause your r/w image was a directory on the HD... it should work, of course, but who knows? I'll give that a try too, if I have time.

~Ev

----------

## emj

Well, I tried to set up the overlay and whatnot on my epia router, but to get squashfs, I ended up going with the ArchCK kernel again (that's what I'm using on my desktop as well), because mm-sources and regular ck-sources don't have that patch included, and I was too lazy to patch a kernel myself. I added squashfs and unionfs and proceeded.

Anyway, my router worked fine as well.

I just did: 

emerge squashfs-tools

mkdir /images

mksquashfs /usr/portage /images/usr.portage.sqfs

mv /usr/portage /usr/portage_

mkdir /usr/portage

echo "/images/usr.portage.sqfs       /usr/portage     squashfs     ro,loop     0 0" >> /etc/fstab

mount /usr/portage

then: myemerge_sync.sh

And I didn't have any errors. I'll edit my original post to include those preparatory steps.

~Ev

----------

## lazy_bum

 *Liviu wrote:*   

> Okay Folks !
> 
> A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):
> 
> http://home.arcor.de/liviufl/portage.sqsh/
> ...

 

It would be nice to tell when the snapshot from "daily" is turning to old, dead and buried... synced with this and i have ~20 packets to "downgrade"...

----------

## Boris27

 *frostschutz wrote:*   

>  *adsmith wrote:*   Why?
> 
> Wow, the portage tree is huge!    It takes around 500MB, which is a huge chunk of the storage space on many systems. 
> 
> 500MB is huge 
> ...

 

My server only has a 10 GB disk, so every byte is welcome. I even looked into changing the blocksize on my EXT3 and having it create less inodes, which take 128 bytes apiece.

----------

## attrezzo

MY laptop has a little 3 gig hard drive and I'm getting around the problem by nfs mounting those partitions when I need to update or do a compile.  /usr/portage & /usr/src 

Just throw an extra partition on a bigger box with more hd space and let nfs do all the work when you have time at your home network. 

Often I can't do any installs when I'm on the road anyway cause the proc is so stinkin slow it simply takes too long.  I usually end up trying to find bin versions and grabbing those.

----------

## Phlogiston

I hope  my guide about different filesystems for portage does fit here quite well.

Discussion on gentoo forums can be found  here

Thanks for any visit.

Phlogiston

----------

## adsmith

But you only work with conventional filesystems, not any compressed filesystems like squashfs. 

You should do this benchmark with squashfs.  I've found it to be extremely fast... empty metadata update in exactly 2 minutes with a 1GHz laptop with a slow hard drive.  The "du -h" test, which is silly in this case, takes <2 seconds.

Also, to what extent does the "emerge -S" test depend on the filesystem for /var/cache/edb, eh?

In any case, that test gave 1min29sec on this machine.

----------

## Gergan Penkov

I have a little bit different partitioning scheme and using unionfs to create the squashfs on the fly with emerge --sync.

I have distfiles and the unionfs-rw-overlay on a standalone partiotions, like this in fstab:

```
/dev/hda5               /var/snapshot           ext2            noatime                 0 0

/var/portage.sqsh       /var/portage    squashfs        ro,loop                 0 0

/usr/portage           /usr/portage    unionfs dirs=/var/snapshot=rw:/var/portage=ro,delete=whiteout   0 0

/dev/hda6               /usr/portage/distfiles          ext3            noatime                 0 0

```

So no need to change the distdir in make.conf, you simply need more partitions - but this is good as the fragmentation is smaller.

modules needed as follows

```
loop

squashfs

unionfs

```

/etc/conf.d/squash_portage

```
#!/bin/bash

PORTAGE_METADIR="/var"

PORTAGE="/usr/portage"

#don't change these values, unless you know what are you doing

PORTAGE_DISTDIR="${PORTAGE}/distfiles"

PORTAGE_SQUASH="${PORTAGE_METADIR}/portage"

PORTAGE_UNIONFS="${PORTAGE_METADIR}/snapshot"

SQUASH_FILE="${PORTAGE_METADIR}/portage.sqsh"

SQUASH_TEMPFILE="${PORTAGE_METADIR}/portage_snapshot.sqsh"

```

/etc/conf.d/local.stop - we need this to properly umount the partitions on shut-down

```
# /etc/conf.d/local.stop

# This is a good place to unload any misc.

# programs you started above.

# For example, if you are using OSS and have

# "/usr/local/bin/soundon" above, put

# "/usr/local/bin/soundoff" here.

if [[ -e /etc/conf.d/squash_portage ]] ; then

    source /etc/conf.d/squash_portage

    umount "${PORTAGE_DISTDIR}"

    umount "${PORTAGE}"

    umount "${PORTAGE_SQUASH}"

else

    eend $? "failed to import squash_portage"

fi

```

and here is the script, which I use to update the portage:

```
#!/bin/bash

#tests if the config exists

if [[ -e /etc/conf.d/squash_portage ]] ; then

    source /etc/conf.d/squash_portage

else

    echo "failed sourcing /etc/conf.d/squash_portage"

    exit 127

fi

#tests if the dirs exist

if ! [[ -d "${PORTAGE_METADIR}" && -d "${PORTAGE_DISTDIR}" && -d "${PORTAGE}" && \

    -d "${PORTAGE_SQUASH}" && -d "${PORTAGE_UNIONFS}" && -n ${SQUASH_TEMPFILE} ]] ; then

    echo "!!! the directories do not exist check your fstab"

    echo "!!! or some of the following variables are not defined"

    echo "!!! PORTAGE_DISTDIR PORTAGE PORTAGE_SQUASH PORTAGE_UNIONFS"

    echo "!!! PORTAGE_METADIR SQUASH_TEMPFILE"

    exit 127

fi

#test if the file is in place

if ! [[ -e "${SQUASH_FILE}" ]] ; then

    echo "!!! the SQUASH_FILE is not defined or the file does not exist, aborting"

    exit 127

fi

#remove stale temp, if exists

if [[ -e "${SQUASH_TEMPFILE}" ]] ; then

    echo "removing stale ${SQUASH_TEMPFILE}"

    rm -f "${SQUASH_TEMPFILE}"

fi

echo "emerge syncing"

torify /usr/bin/emerge --sync

echo "emerge --sync ready..."

echo "building the squashfs image"

umount "${PORTAGE_DISTDIR}"

/usr/bin/mksquashfs "${PORTAGE}" "${SQUASH_TEMPFILE}" -check_data

umount "${PORTAGE}"

umount "${PORTAGE_SQUASH}"

rm -fvR "${PORTAGE_UNIONFS}"/*

rm -f "${SQUASH_FILE}"

mv "${SQUASH_TEMPFILE}" "${SQUASH_FILE}"

mount "${PORTAGE_SQUASH}"

mount "${PORTAGE}"

mount "${PORTAGE_DISTDIR}"

echo "ready..."

```

The first squashfs image must be created with the normal method.

Strangely enough the squashfs creation is a lot faster in this way probably, because it could read faster  :Smile: 

----------

## adsmith

Which kernel are you using?

I always failed to get the unionfs patch working on vanilla kernels.

I'm curious if this has been fixed in the move from 2.6.14 to 2.6.15

----------

## Gergan Penkov

I'm using the standalone unionfs from the portage (unionfs-1.1.3), it's not patched in the kernel.

I don't know which kernels have this, (gentoo is not patched), but I have seen it somewhere earlier, probably in nitro.

----------

## yoshi314

combined with cdb-portage tweak this thing is a killer :] 

i used to keep my portage on reiser4 loopfile. and i use cdb. it was very fast. but squashfs metadata update with cdb is ~3x faster than that :] 

(not mentioning the update-eix, with metadata setting for portage cache  :Very Happy: )

i wish it had support for overlays as well  :Very Happy: 

oh, i use squashfs 3.0 from archck-sources 2.6.16-archck1. work s like a charm :]

----------

## Gergan Penkov

Well, I don't see any problems running the overlays on squashfs/unionfs, I already think how it will be better to partition here for this to work good.

----------

## yoshi314

 *Gergan Penkov wrote:*   

> Well, I don't see any problems running the overlays on squashfs/unionfs, I already think how it will be better to partition here for this to work good.

 no, the problem was in my config. i had portage in /usr/portage/maintree and overlays in /usr/portage/overlays and whole /usr/portage was on a reiser4 loopback. 

so i converted the entire loopback into squashfs, but the script expected to have portage only on that squashfs image. so my overlays got erased :/

----------

## xming

I am testing jffs2 for /usr/portage, it's +/- 75MB so 3x bigger than squasfs but it's rw. Jffs2 + cdb is like portage of 2000.

----------

## adsmith

Cool!

I had tried jffs2 before, but I never got it working particularly well.

Any tips?

----------

## xming

ok I will do that once I have things sorted out

----------

## Da Fox

I've updated emj's original script to include some error checking and error messages. It's probably still far from a pretty way to make Bash scripts, but it seems to work  :Wink: 

Before using you should set RAMSIZE, IMAGEDIR and RAMDISKDIR (in the script) to apropriate values. I haven't really tested this script yet, but it probably should work. It lets you know exactly what it is doing and when something goes wrong you (should) get a semi-descriptive error message.

I found that if you umount /usr/portage with the '-l' option (--lazy) the '-d' (clear loopdevice) option loses it functionality, thus filling up your loop devices. Uhm, so don't use that option :p

The script:

```
#!/bin/bash

#

#myemerge_sync.sh - script to emerge --sync from a squashfs

#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location

#emerge --sync's, then re-squashes portage and remounts

#also runs an update-eix to update eix database

#

#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.

#I use the ArchCK ebuilds (search the gentoo forums) to get the above in the kernel.

#Also, remember that it's very important to have DISTFILES set somewhere

#other than /usr/portage/distfiles

#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf

#

#Preparatory steps to running this the first time are as follows:

#

#emerge squashfs-tools

#mkdir /images

#mksquashfs /usr/portage /images/portage.sqfs

#mv /usr/portage /usr/portage_

#mkdir /usr/portage

#echo "/images/portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab

#mount /usr/portage

#

#...then try running this script. When you confirm that everything's working you

#can rm -rf /usr/portage_ to remove the old uncompressed portage tree, or restore

#it if the script doesn't work for you.

#

#~Ev

#

# Given a thorough make-over by Da Fox

#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.

#100M should suffice unless you've failed to sync for a very, very long time.

#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.

#It will be unloaded when done sync'ing.

RAMSIZE="128M"

IMAGEDIR="/var/ftp"

RAMDISKDIR="/var/tmp/ramdisk"

#######################

echo -n Checking for a place to store the SquashFS image...\ 

if mkdir -p ${IMAGEDIR} &> /dev/null ; then

  echo found ${IMAGEDIR}

  echo -n Checking for a place to mount the ramdisk...\  

  if mkdir -p ${RAMDISKDIR} &> /dev/null ; then

    echo found ${RAMDISKDIR}

    echo -n Cleaning ${RAMDISKDIR}...\ 

    if rm -rf ${RAMDISKDIR}/* ; then

      echo done

      echo -n Checking if /usr/portage is a mounted SquashFS...\ 

      if cat /proc/mounts | grep squashfs | grep "\ \/usr\/portage\ " &> /dev/null ; then

        echo ok, it is.

        #Since the ramdisk is part of the unionfs we should make sure that the union

        #isn't active beforehand, otherwise we'll get a lot of stale mounts for ${RAMDISKDIR}

        if cat /proc/mounts | grep unionfs | grep "\ \/usr\/portage\ " &> /dev/null ; then

          #If the unionfs is already mounted on /usr/portage we can't just remount it

          #Try to umount it || die

          echo \*\* Warning: /usr/portage is already mounted as an UnionFS \*\*

          echo -n \ \ \ attempting to umount /usr/portage...\ 

          if umount -d /usr/portage &> /dev/null ; then

            echo ok. Proceeding as usual.

          else

          #Something bad has happened

            echo \ \ \ Failed!

            echo \ \ \ This really shouldn\'t happen. Please make sure /usr/portage is unmounted

            echo \ \ \ before running this script again.

            exit $?

          fi

        fi 

        if umount -d ${RAMDISKDIR} &> /dev/null ; then

          echo Unmounted existing ramdisk from ${RAMDISKDIR}

        fi

        echo -n Mounting ramdisk of ${RAMSIZE} on ${RAMDISKDIR}...\ 

        if mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR &> /dev/null ; then

          echo ok

          echo -n Mounting UnionFS overlay on /usr/portage...\ 

          if mount -t unionfs -o dirs=${RAMDISKDIR}=rw:/usr/portage=ro none /usr/portage &> /dev/null ; then

            echo ok

            #Now we can finally sync the tree :D

            if which eix &> /dev/null ; then 

              echo -n Now running eix-sync, please standby...\ 

              if eix-sync &> /dev/null ; then

                echo done

              else

                echo failed!

                echo An error occurred while running eix-sync.

                echo Please make sure that eix-sync works first!

                exit $?

              fi

            else

              echo \*\* Warning: You are running this script, but are not using eix \*\*

              echo \*\* eix is a really nice tool and you should emerge it asap. \;\) \*\*

              echo -n Now running emerge --sync, please standby...\ 

              if emerge --sync &> /dev/null ; then

                echo done

              else

                echo failed!

                echo An error occurred while running emerge --sync.

                echo Please make sure that emerge --sync works first!

                exit $?

              fi

            fi

            #create a backup of the old image, user *should* have write permissions

            echo -n Creating temporary backup of ${IMAGEDIR}/portage.sqfs...\ 

            if mv ${IMAGEDIR}/portage.sqfs ${IMAGEDIR}/portage.sqfs.bak &> /dev/null ; then

              echo ok

            else

              echo failed!

              echo Is your ${IMAGEDIR}/portage.sqfs missing?

              exit $?

            fi

            echo -n Updating portage SquashFS image...\ 

            if mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs -check_data &> /dev/null ; then

              echo ok

              echo -n Removing backup file...\ 

              if rm -f ${IMAGEDIR}/portage.sqfs.bak &> /dev/null ; then

                echo ok

              else

                echo failed!

                echo You should manually remove ${IMAGEDIR}/portage.sqfs.bak

              fi

              echo -n Now umounting /usr/portage union...\ 

              if umount -d /usr/portage &> /dev/null ; then

                echo ok

              else

                echo failed!

                echo Please make sure to umount the UnionFS from /usr/portage.

              fi

              echo -n Now umounting ${RAMDISKDIR}...\ 

              if umount -d ${RAMDISKDIR} &> /dev/null ; then

                echo ok

              else

                echo failed!

                echo Please make sure to umount the ramdisk from ${RAMDISKDIR}.

              fi

              echo -n Now umounting /usr/portage SquashFS...\ 

              if umount -d /usr/portage &> /dev/null ; then

                echo ok

                echo -n Now remounting /usr/portage SquashFS...\ 

                if mount -t squashfs -o ro,loop ${IMAGEDIR}/portage.sqfs /usr/portage &> /dev/null ; then

                  echo ok

                  echo All done, you may now enjoy your Updated Portage Tree :p

                else

                  echo failed!

                  echo Please make sure to mount ${IMAGEDIR}/portage.sqfs back on /usr/portage,

                  echo otherwise you won\'t have a Portage tree.

                fi

                exit $?

              else

                echo failed!

                echo Please make sure to umount \&\& mount the SquashFS from /usr/portage,

                echo in order to take advantage from the updated SquashFS image.

              fi

              exit $?

            else

              echo failed!

              echo

              echo Something funny has happened:

              echo You *should* have write permissions on ${IMAGEDIR}

              echo You *should* have mksquashfs

              echo You didn\'t cancel the creation process, now did you?

              echo -n restoring backup...\ 

              if  mv ${IMAGEDIR}/portage.sqfs.bak ${IMAGEDIR}/portage.sqfs &> /dev/null ; then

                echo ok

                echo please run \'mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs\' manually.

                echo \(Don\'t forget to make appropriate backups of ${IMAGEDIR}/portage.sqfs first!\)

              else

                echo failed!

                echo This is bad. Please review your config and make sure nothing

                echo funny is going on with ${IMAGEDIR}, and no other users/processes are

                echo writing to your SquashFS \(${IMAGEDIR}/portage.sqfs\)

                exit $?

              fi

            fi

          else

            echo failed!

            echo Please make sure you have UnionFS installed and it\'s module is loaded.

            echo emerge unionfs \&\& insmod /lib/modules/`uname -r`/fs/unionfs.ko

          fi

        else

          echo failed!

          echo Please make sure you have permission to mount ${RAMDISKDIR}.

        fi

      else

        echo failed!

        echo

        echo Before running this script you should:

        echo \ \ emerge squashfs-tools

        echo \ \ mkdir ${IMAGEDIR}

        echo \ \ mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs -check_data

        echo \ \ mv /usr/portage /usr/portage_

        echo \ \ mkdir /usr/portage

        echo \ \ echo \"${IMAGEDIR}/portage.sqfs /usr/portage squashfs ro,loop 0 0\" \>\> /etc/fstab

        echo \ \ mount /usr/portage

        echo \ \ Read further instructions in \'`pwd`/`echo ${0} | sed "s/.*\///"`\'

      fi

    else

      echo failed!

      echo Please make sure you have write permissions on ${RAMDISKDIR}.

    fi

  else

    echo failed!

    echo Please make sure you have write permissions on ${RAMDISKDIR}.

  fi

else

  echo failed!

  echo Please make sure you have write permissions on ${IMAGEDIR}.

fi

exit 0
```

----------

## BigMichi1

 *Quote:*   

> #REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules. 

 

unionfs kernel module (if it's not already in the used kernel) can also be installed through 

```
emerge unionfs
```

also there is something wrong in the Preparatory steps in the header of the script:

```
mksquashfs /usr/portage /images/usr.portage.sqfs
```

 and 

```
echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
```

the usr.portage.sqfs should be portage.sqfs else the script fails when it tries to create the temporary backup

----------

## Da Fox

[quote="BigMichi1"] *Quote:*   

> also there is something wrong in the Preparatory steps in the header of the script:
> 
> ```
> mksquashfs /usr/portage /images/usr.portage.sqfs
> ```
> ...

 

Thanks, I've made the changes in the posted script. It was called that in the original script, I really haven't change any of the introductory comments  :Smile: 

----------

## synss

Last edited Sat 2006-05-17 v. 0.1.2

Split and updated here

Very nice!!!

A simple init script will do, and I assume /dev/shm being tmpfs (that's the default) and the image is laready prepared, I might make it more general if I have time something like

```
if [ ! -e $PORTAGE_SQFS ]; then create it; fi
```

anyway

```
echo "unionfs" >> /etc/modules.autoload.d/kernel-2.6

echo "squashfs" >> /etc/modules.autoload.d/kernel-2.6
```

/etc/conf.d/portage.sqfs

```
PORTAGE_SQFS="/var/tmp/portage.sqfs"

PORTAGE_RO="/media/portage_ro"

PORTAGE_RW="/dev/shm/portage"

PORTAGE_TMP="/var/tmp/portage.tmp"

PORTDIR="/usr/portage"
```

/etc/init.d/portage.sqfs

```
#!/sbin/runscript

# Copyright 1999-2006 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: $

depend() {

   need localmount

}

start() {

   ebegin "Mounting squashfs'ed portage tree"

   [ -d $PORTAGE_RO ] || mkdir -p $PORTAGE_RO

   mount -t squashfs -o loop,ro $PORTAGE_SQFS $PORTAGE_RO

   [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW

   mount -t unionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR

   eend 0

}

stop() {

   ebegin "Updating portage tree"

   if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then

      einfo "Syncing the tree"

      mksquashfs $PORTDIR $PORTAGE_TMP -check_data

   else

      einfo "Nothing to do"

   fi

   eend 0

   ebegin "Unmounting the tree"

   umount $PORTDIR

   umount $PORTAGE_RO

   rm -rf $PORTAGE_RW

   [ -e $PORTAGE_TMP ] && mv -f $PORTAGE_TMP $PORTAGE_SQFS

   eend 0

}
```

----------

## lazy_bum

Hi.

Can someone "unify" this stuff about portage with unionfs? New topic?

Oh, and squashfs-tools 3.0 broke my squashed portage...

```
dmesg | tail

*snip*

SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)

SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)
```

Ideas? New kernel module with 3.0 support? (or change to unionfs? ;-)

::edit::

Don't know why "watching" this topic isn't sending any mail.

----------

## Da Fox

 *lazy_bum wrote:*   

> Hi.
> 
> Can someone "unify" this stuff about portage with unionfs? New topic?
> 
> Oh, and squashfs-tools 3.0 broke my squashed portage...
> ...

 

kernels <2.6.16 don't support SquashFS 3.0, this was added in 2.6.16. So you should update to a more recent kernel.

----------

## bluefox81

I'm using suspend-sources 2.6.14, unionfs 1.1.4.2-r2, rsync 2.6.8 with the rc scripts... well all seems working, no problem on mount squashfs filesystem (v 2.0...) nor with unionfs.

I updated my portage-tree once for testing... just adding some ebuilds and all done.

Now while trying a "serious" update rsync stop doing anything (without exiting, it just seems freezed) more precisely deleting files

```

deleting sys-apps/portage/files/2.1_pre7/1170_r3084_bug_128362.patch

deleting sys-apps/portage/files/2.1_pre7/1160_r3083_bug_126801.patch

deleting sys-apps/portage/files/2.1_pre7/1150_r3082_bug_117713.patch

deleting sys-apps/portage/files/2.1_pre7/1140_r3077_noclean.patch

deleting sys-apps/portage/files/2.1_pre7/1130_r3087_bug_129098.patch

deleting sys-apps/portage/files/2.1_pre7/1120_r3094_bug_129193.patch

deleting sys-apps/portage/files/2.1_pre7/1110_r3096_emerge_args_validation.patch

deleting sys-apps/portage/files/2.1_pre7/1100_r3063_bug_128506_distdir_error.patch

deleting sys-apps/portage/files/2.1_pre7/1090_r3055_bug_128284_get_open_fds.patch

deleting sys-apps/portage/files/2.1_pre7/1080_ensure_dirs.patch

deleting sys-apps/portage/files/2.1_pre7/1070_r3036_bug_127930_unpack_timestamp.patch

deleting sys-apps/portage/files/2.1_pre7/1060_r3034_bug_127897_realpath.patch

deleting sys-apps/portage/files/2.1_pre7/1050_forum_3210399_empty_digest.patch

deleting sys-apps/portage/files/2.1_pre7/1040_r3015_user_fetch.patch

deleting sys-apps/portage/files/2.1_pre7/1030_r3009_bug_127563_ccache_stat.patch

deleting sys-apps/portage/files/2.1_pre7/1020_r3006_bug_127573_cachedir.patch

deleting sys-apps/portage/files/2.1_pre7/1010_r3004_lazy_virtuals.patch

deleting sys-apps/portage/files/2.1_pre7/1000_r2994_workdir_mode.patch

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(242) [receiver=2.6.8]

```

it catch the ^C but no other consequences...

waiting without sending a signal rsync print timeout error at io.c [receiver=2.6.8]

kill -9 or other signals doesn't work... rsync cannot be stopped, and obviously unionfs connot be unmounted

I think it's a unionfs problem... but dmesg doesn't output anything, otherwhise the way rsync is locked bring me to think it's a kernel/module level problem...

any ideas...

pleaze... i really like and need a squashfs rw portage-tree....

thanks in advance at all guy!

----------

## xming

Ok as promised I have put a guide of portage on jffs2 on my home page

----------

## bluefox81

tnx... can u give an idea of how long takes mount this particular fs? advantages and disadvantages vs squash fs + unionfs (apart than proably with jffs2 all works properly  :Wink: )... because the only time i correctly sync my portage tree (just some ebuild added, there wasn't need to delete anything) all works very fast (ie. dependencies deep scans, searches, etc) , also updating the squashfs file taken few seconds... (i tought REALLY GREAT!!!! until those problems appears...  :Sad: )... so if u give me... and us... a performance idea would be really nice

tanks in advance!

----------

## xming

jffs2 vs squashfs +unionfs

pro

no kernel patches

simple, just one package to emerge

less change for breakage

no need to merge back the changes

contra

long mount time, 15 sec to 1 min depending on the hardware

use more disk space than squashfs (+/- 3x)

xming

----------

## bluefox81

tnx... maybe the "long" time for mount the fs could be secondary... if i'm right we can mount /usr/portage only the first time we use emerge... so a simple script can do that without wait during the boot process...

----------

## xming

yes, you can use the code on my site and put it in a file and execute it before the 1st time you do a emerge

----------

## Da Fox

Could you please explain why you would want to use jffs2 on your harddisk? From what I gather on the wikipedia entry it is designed for use on Flash devices.

There is also no mention of it being significantly faster (or slower for that matter) then any other filesystem, so why are you experiencing such speed up? If it is because you think that there is less disk accesses that has to be wrong, as the whole squashfs image only takes 33MB, which is easily chached in memory, and the tmpfs is also completely in memory, so there shouldn't be a lot of disk activity right? The one thing that does take some time is the (re)creation of the squashfs image.

----------

## xming

First of all it only takes 90 MB instead of 500+ MB so there is more chance that thing gets cached and it's still 80 % saving on disk space. And there is a significant speed up, because there is less (80%) to read from the disk. I am sure that squashfs is faster but it's read only, and I have my doubts if squashfs + unionfs is faster that jffs2.

Anyway I am not forcing you to use jffs2.

----------

## Da Fox

ofcourse you're not forcing anyone, it's all about choice  :Cool: 

I was just wondering why you were using jffs2, since it's designed for flashmemory. I have now also read some of your other reasons on your homepage, and they are good reasons. It's true that squashfs is readonly, but then again you don't need it to be read/write that often, only when you 'emerge --sync'. But most of the speed up is indeed because of caching and that is true for both filesystems. However I think that with the time it takes to mount jffs2 every time it might turn out that squashfs+unionfs is actually faster, rebuilding the image only takes about a minute. So most of the time as always lost in downloading the new files and updating the eix cache.

One last thing, squashfs 3.0 might not be in vanilla sources, but it definitely is in gentoo-sources, so you have that anyway.  :Smile: 

----------

## xming

mount time is a non issue for me, reiser4 has also long mount time (less than jffs2 but longer than ext2, and even ext3 has longer mount time than ext2. I only have to mount once after reboot so on servers this is really not a problem and on my laptop I use software suspend so it's neither a isse here.

However I do emerge --syn approx. once a day, with squashfs I have to rebuild my fs each day, jffs2 just works. I did not choose jffs2 for any flash orientated features, I choose jffs for simplicity, less space (than ext/3, reiser) and speed.

I would have used e2compr if it was updated and in the vanilla. ARe there any other compress fs in the vanilla kernel?

xming

----------

## Gergan Penkov

In fact as the tree is already cached in the memory and only the changes are on disk (as unionfs overlay - in the way I use it), rebuilding the squashfs image is really fast <1min, on the other hand unionfs does not need patching - simple kernel-module re-compile.

----------

## jjlawren

Assume I'm using squashfs mounted by NBD over a network. What's the best way for updating the filesystem without causing problems on the other systems? Do they need to unmount it first, or can the squash file simply be replaced?

----------

## enderandrew

Color me stupid, but if you've got a small HDD and old hardware, but wouldn't the better solution be simply to use something like prlock ( https://forums.gentoo.org/viewtopic-t-55031-highlight-prlock.html ) to cut down portage?  The guy who wrote the script said he cut down his portage tree to 95% of its size.

Pros:

Free up HDD space

Syncs are FASTER since you are excluding all kinds of things

Don't have to take forever to squash anything

Don't have to have the HDD space free for unsquashing

Works on all kernels

Cons:

If you want to emerge something new that has been removed from portage, you have to use prlock again to add the ebuild back in, or download the ebuild manually.

Given that we're talking older hardware, there is a good chance that you don't emerge completely new software all that often, if for no other reason that lack of HDD space.  You probably keep what you've got, and perhaps upgrade from time to time.

----------

## Da Fox

You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.

Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope  :Wink: ) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.  

In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it  :Very Happy: 

----------

## Gergan Penkov

And not to forget the fragmentation, and this on old hardware - the portage fragments extremely the hard disk with squashfs and probably with in memory mounted unionfs overlay there will be virtually no chances for fragmentation  :Smile:  which is IMHO the best thing out of this and the speed of course  :Smile: 

----------

## enderandrew

 *Da Fox wrote:*   

> You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.
> 
> Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope ) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.  
> 
> In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it 

 

In the prlock thread, the example was going down to 18 megs without any compression.

It is faster, and it uses less bandwidth which makes it more friendly on the various Gentoo mirrors.

As far as work, prlock has an easier setup, and is also a one-time thing that ends up saving you time (no compress/expand/recompress, as well as faster syncs and searches).  Having a smaller portage naturally is a win-win-win.

If you *REALLY* wanted, you could combine both methods and shrink portage down very small.

----------

## yoshi314

this method is way better https://forums.gentoo.org/viewtopic-t-465367-highlight-squashfs.html

i use it on daily basis now.

----------

## sfragis

 *Phlogiston wrote:*   

> I hope  my guide about different filesystems for portage does fit here quite well.
> 
> 

 

Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:

block size of 65536 bytes (default) => 38 MB

block size of 16384 bytes => 44 MB

block size of 4096 bytes (minimum accepted by mksquashfs) => 55 MB

Hence the best value for the block size is 64KB which seems in contraddiction with the result of Phlogiston's guide   :Shocked:  (smaller block size leads to less filesystem usage). Has anyone tried to tune this or other parameters in order to achieve different results?

----------

## Phlogiston

 *sfragis wrote:*   

>  *Phlogiston wrote:*   I hope  my guide about different filesystems for portage does fit here quite well.
> 
>  
> 
> Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:
> ...

 

Hmm probably this is different when using squashfs, but actually I can't explain.

But my question: Is there any howto that sums this stuff here up and gives clear instructions about using portage on a squashfs? I mean what are the results of that thread? Squasfs + unionfs ->rw and then just portage on it? 

Thanks!

----------

## adsmith

Option 1: (easy, read-only)

download http://www.math.duke.edu/~adsmith/portage.sqsh periodically (it's updated weekly), and mount it to /usr/portage.

Option 2: (slightly more complicated, read-write)

use unionfs over squashfs, see here: https://forums.gentoo.org/viewtopic-p-3338393.html

----------

## Phlogiston

 *adsmith wrote:*   

> Option 1: (easy, read-only)
> 
> download http://www.math.duke.edu/~adsmith/portage.sqsh periodically (it's updated weekly), and mount it to /usr/portage.
> 
> Option 2: (slightly more complicated, read-write)
> ...

 

Thanks! I'd like to sync it myself but I don't like the init scripts because I never shutdown actually.

----------

## Phlogiston

This is quite interesting:

```

Creating little endian 3.0 filesystem on /tmp/images/portage.sqfs, block size 65536.

Little endian filesystem, data block size 65536, compressed data, compressed metadata, compressed fragments

Filesystem size 38251.43 Kbytes (37.35 Mbytes)

        25.09% of uncompressed filesystem size (152465.07 Kbytes)

Inode table size 1652513 bytes (1613.78 Kbytes)

        35.97% of uncompressed inode table size (4594717 bytes)

Directory table size 1407370 bytes (1374.38 Kbytes)

        40.76% of uncompressed directory table size (3453208 bytes)

Number of duplicate files found 12688

Number of inodes 146450

Number of files 123037

Number of fragments 2275

Number of symbolic links  1

Number of device nodes 0

Number of fifo nodes 0

Number of socket nodes 0

Number of directories 23412

Number of uids 2

        root (0)

        portage (250)

Number of gids 0

```

Especially the duplicate files   :Cool: 

I'm thinking more and more that the way of portage syncing and storing is quite deprecated. Using portage and syncing just the compressed images cuts down the traffic incredibly.

/edit: Wow this rocks:

```

time /usr/local/sbin/psync.2

[...]

All done, you may now enjoy your Updated Portage Tree :p <-this actually not  8) 

real    2m19.984s

user    0m27.094s

sys     0m18.061s

```

Thanks a lot... Although I think that the first emerge -Duvat world afterwards is not really faster...

----------

## sfragis

I wrote a small script to wrap the portage tree sync and automagically mount/umount the unionfs overlay and keep the squashfs image up to date. Code uses parts of scripts published in this and other threads of the Gentoo forum. There's a small header in it with a brief explanation of what I did to get things working. When I'll have more spare time I'll write a detailed html page. Feel free to trash it, use it and edit it   :Wink: 

----------

## Phlogiston

Hmm I have a new problem now: After activating squashfs and unionfs in my kernel config, suspend2 does not work any longer. It hangs at doing atomic copy. Any hints?

----------

## sfragis

Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted?

----------

## micko

 *Quote:*   

>  *Phlogiston wrote:*   
> 
> Option 2: (slightly more complicated, read-write)
> 
> use unionfs over squashfs, see here: https://forums.gentoo.org/viewtopic-p-3338393.html 
> ...

 

I wrote an extremely simple script that sync's portage and restarts the service, so the squashfs-image gets updated. If I forget to use the script and sync normally nothing will go wrong, unless my computer crashes, and even then everything is fixed by a new sync. I never shut down either and sync maybe about 4-5 times a week. Again, if I forget to use the script once but remember the next time, it will update the image then.

This unionfs-method is great also because it works well with layman.

That is for my fastest computer. For my server and laptop I've written a script that umounts portage dl's the portage.sqfs with scp from my desktop and mounts the new one. Everything works just great.

----------

## Phlogiston

 *sfragis wrote:*   

> Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted?

 

I compiled the stuff directly into the kernel. (I hate modules   :Cool:  ) But I will find out if it`s squashfs or unionfs, and then try with modules. PS: It didn't help to umount portage before. So even I don't use sqaushfs or unionfs it breaks hibernation. That is at least a bug in my opinion.

BTW using latest beyond3 sources.

/update: I just found out that its not squashfs nor unionfs related. It just has to do with my kernel, I don't know, using an older one now.

----------

## tost

Now I´m a bit confused

I really like the idea of using squashfs for Portage, but for me it seems that it doesn´t really work.

I activated all the stuff in the kernel and after this viewPortageX (that Superkaramba theme) shows me that some updates are available.

So I run emerge-sync-squash.sh. But at the end of it emerge -NuDpv world  says that no updates are available.

And now the things get strange, because  emerge -pv vcdimager wants to upgrade the package ?

Whats going wrong ?

----------

## adsmith

I'm not sure I entirely understand your problem.   If you jsut man that a world update is missing things, it may be totally unrelated to squashfs.  

1) Is "getbinpkg" activated in FEATURES in your make.conf?  That sometimes causes missed packages in world updates

2) are you sure the squashed portage tree is complete and mounted properly, and that PORTDIR is set properly?

3) Is your world file actually up-to-date?

----------

## tost

mmhh I thought the problem is a consequence of my compressed portage tree, but now I saw that it doesn´t work either when I use portage in "default-mode" (uncompressed)

 *Quote:*   

> 3) Is your world file actually up-to-date?

 

I´ll check it, thanks

tost

----------

## Phlogiston

I'm still using DaFox script to update my portage and it works well (though, any updates?)

But today I noticed an ugly slowdown: emerge -ua world takes minutes to calcuate the updates. I don't know if its the portage version I am using(     Installed:           2.1.1_rc1-r2) or the latest kernel update.

Anyone else noticed this?

----------

## Phlogiston

Hey squashfs portage user whats up? I'm still using the psync script here and it works perfectly. Is anyone else still using it or is there any better way to do it?  :Cool: 

----------

