# TIP: Compressing portage using squashfs: initscript method

## synss

Admin Edit:

squashmount related information has been split from this topic. It is now here:

Compressing filesystems with squashfs and squashmount

--pjp

Tue Nov 28 12:10:10 JST 2006, v. 0.1.5(4)

I have left gentoo and have not updated this script for a while, though it probably still works, mv has taken over and proposed new features and improvements, aufs is one good thing among others. And his script can be used on any directory (I would recommend /usr/src/linux and the docs) You should probably have a look at the thread and check mv's version, now. Last version is available here --- I was sure I had left a notice like this one here but only realize now I must have forgotten...

Abstract The portage tree takes typically 600MB on /usr/portage, some people have chosen to have it on a separate partition with a small block size todecrease its global size

decrease fragmentation

increase the speed during searches and updateHowever that also means that the free space on that partition is lost. Another, better, solution is to use a stack file containing the filesystem with small block size to avoid the shortcomings of the separate partition. Following the discussion in this post, the best solution found is to compress the portage tree using the compressed, read-only squashfs filesystem and mount a read-write partition on top of it using unionfs. Second, for convenience, the portage tree should be mountable on boot-up and updated on shutdown, hence I have written an initscript to automate this process. I will publish and update this program here.

Assumptions I will assume that you have read   the original idea as it details what you need to get started and that you are familiar with the bootup/shutdown sequence of your computer, 

read the Gentoo doc on initscripting. 

DISTDIR should be out of the tree, etc. Please just read the original post as everything is well documented there and come back once you have done that, unless you do understand what you are doing!

Introduction In brief, you will need to load the relevant kernel modules and emerge the userland utilities 

```
emerge -avt sys-fs/unionfs sys-fs/squashfs-tools
```

, unmask if needed (I am now successfully using sys-fs/unionfs-1.2 on sys-kernel/suspend2-sources-2.6.16-r8 and sys-fs/squashfs-tools-3.0)

```
modprobe loop squashfs unionfs
```

 add the modules to /etc/modules.autoload.d/kernel-2.6.

the script 

```
# /etc/conf.d/squash_portage

# SQFS_DIRNAME points to the directory that will contain the sqfs

# images, recommended value is /var/tmp

SQFS_DIRNAME="/var/tmp"

# Leave PORTAGE_RW empty for use with tmpfs, a ram-based filesystem,

# This is recommended unless you are short of RAM

PORTAGE_RW=""
```

and

```
#!/sbin/runscript

# Copyright 1999-2006 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2 

# $Header: $ 

#

# /etc/init.d/squash_portage allows efficient compression of 

# Gentoo portage arborescence

#

# It requires support for the loop device and squashfs enabled in the kernel,

# module autoloading is also *highly* recommended.

# sys-fs/squashfs and sys-fs/unionfs are necessary for read-write support.

#

# Author: Mathias Laurin <mathias_laurin @ users.sourceforge.net>

# 2006-11-28, v.0.1.5(4)

source /etc/make.conf

SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"

SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"

SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"

DEF_RW="/dev/shm/.portage-rw"

depend() {

   need localmount

}

start() {

   ebegin "Mounting read-only squashfs image"

   mount -rt squashfs -o loop,nodev,noexec $SQFS_CUR $PORTDIR

   retval=$?

   eend $retval

   [ $retval -ne 0 ] && return $retval

   ebegin "Mounting read-write with unionfs"

   if [ ! $PORTAGE_RW ]

   then

      einfo "  mounted in tmpfs (RAM)"

      PORTAGE_RW="${DEF_RW}"

   fi

   [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW

   chmod 0750 $PORTAGE_RW

   chown portage:portage $PORTAGE_RW

   mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR

   eend $?

}

stop() {

   ebegin "Updating portage tree"

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"

   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

   then

      einfo "  Syncing the tree"

      mv -f $SQFS_NEW $SQFS_OLD

      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null

      retval=$?

      ln -sf $SQFS_NEW $SQFS_CUR

   else

      einfo "  Nothing to do"

      retval=0

   fi

   eend $retval

   ebegin "Unmounting the tree"

   umount -t unionfs  $PORTDIR

   umount -t squashfs $PORTDIR

   rm -rf $PORTAGE_RW

   eend 0

}

```

That obviously is bash as any initscript. Make it executable and 

```
/etc/init.d/squash_portage start

rc-update add squash_portage default
```

The script assumes that you already have an sqfs image on your disk ready to be mounted and called $SQFS_DIRNAME/portage.sqfs it will save the new image if you sync'ed the tree the rest is pretty obvious and the rw partition is mounted on /dev/shm by default for speed and simplicity. But beware that it means you will lose your update in case of power failure or if you are not caring. I would recommend to restart the script upon hibernation if you use it and do not always reboot into your ram image.

----------

## yoshi314

th...this is great! i would never think of implementing it via initscript  :Very Happy: 

nice job. 

although there is one drawback with this overlay - when you upgrade your kernel you might get stuck with no portage tree at all, since not every kernel might have squashfs and unionfs modules integrated. there could be a solution in form of an mini-overlay that would contain only ebuilds for squashfs and unionfs (and required eclasses) for forgetful folks. (would it be difficult to make such a backup overlay from an existing portage tree via an bash/sh script?)

sorry for writing that here, but i thought is fits this topic more - it's a more elegant solution.

----------

## synss

Thank you!!!  :Very Happy: 

No problem for writing here either...

I do not think there should be any problem: 1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks"  :Smile:  can compile the necessary modules and modprobe them (no need to reboot into the new kernel).

2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.

I have had a problem with union that was acting weird during unmount, but that was a mistake from my side (a non-existing locale when compiling glibc, I think) So you might be unable to sync, but you can then compile and load the modules and go on. In any case, it is always possible to download an image of the tree from a nearby ftp server  :Wink:  yes, that is exactly for avoiding that that I implemented the incremental timestamped backups while debugging. (Then I leave it cause it is a good idea and you can easily clean the backups via cron if you want to.) So the only risk you are taking, AFAIK, is that you might have to mount the squashed image by hand after recompiling the modules.

Thank you again!

----------

## lazy_bum

Thanks man, this is great! Fast, small and working! (-:

I'm using sys-fs/squashfs-tools-2.2_p2, sys-fs/unionfs-1.1.4-r2 on a sys-kernel/gentoo-sources-2.6.14-r5

----------

## MxxCon

have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?

from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.

----------

## Gergan Penkov

 *MxxCon wrote:*   

> have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
> 
> from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.

 

with squashfs you don't need the free space to unpack it, and these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.

----------

## MxxCon

 *Gergan Penkov wrote:*   

> these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.

 hmm i must have misread something then. i assumed they would create a single compressed file from a dir tree.

----------

## Gergan Penkov

Well I though the same in the beginning, even with such implementation the things would have been just ok if there was some sort of file-container ala tar around the files, of course the compression ratio would not have been very good but there would have been virtually not that much fragmentation and loss of free space  :Smile: 

----------

## lazy_bum

Something is wrong with the new version:

```
cat /etc/conf.d/squash_portage

source /etc/make.conf

PORTAGE_BASENAME="/usr/portage"

PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs"

PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"

PORTAGE_RO=$PORTDIR

PORTAGE_RW="/dev/shm/portage"
```

And, when running the script:

```
/etc/init.d/squash_portage start

 * Mounting squashfs'ed portage tree ...

Usage: mount -V                 : print version

       mount -h                 : print this help

       mount                    : list mounted filesystems

*snip*

A device can be given by name, say /dev/hda1 or /dev/cdrom,

or by label, using  -L label  or by uuid, using  -U uuid .

Other options: [-nfFrsvw] [-o options] [-p passwdfd].

For many more details, say  man 8 mount .                                 [ ok ]
```

... and /usr/portage/ is empty...

I have squashed portage in /usr/portage.sqfs

And i wonder what "date" is doing in here:

```
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"
```

Is it possible that a symlink is created to nothing? When i stop the script day after sync'ing the tree, and the date is different?

Hope it's usefull.

----------

## synss

"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...

I am in a cybercafe on XP with a French kbd so debugging is not easy... But I do not see what else but the first mount can be a problem.

you can also try to mount the tree by hand, sync and copy-paste

```
source /etc/make.conf              

PORTAGE_BASENAME="/usr/portage" 

PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs" 

PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs" 

PORTAGE_RO=$PORTDIR 

PORTAGE_RW="/dev/shm/portage" 

if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then 

                [ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW 

                mksquashfs $PORTDIR $PORTAGE_NEW -check_data 

                [ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS 

                ln -sf $PORTAGE_NEW $PORTAGE_SQFS 

        fi 

        umount $PORTDIR 

        umount $PORTAGE_RO 

        rm -rf $PORTAGE_RW 
```

that should make sth clean for starting the initscript and use it and forget it.

----------

## lazy_bum

 *synss wrote:*   

> "date" creates incremental backups
> 
> What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.
> 
> I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...
> ...

 

Well, i don't know if it wasn't my computer problem. Had few day of "fun" with kernel panic and stuff...

But maybe it would be better to move previous portage to something like "portage-old.sqfs" and the new one would be "portage-current.sqfs". It would help for the "forgetful folks" to autodelete and keep only one portage (that you use) and one (as a backup). Otherwise few days and you have 10 squashed files... or i'm wrong? But the scprit deletes only the PORTAGE_SQFS and it's a symlink.

Oh, and I had a problem when didn't sync (IIRC ;-), the script during a reboot created "portage" (4096 bytes) and a symlink to it...

 *synss wrote:*   

> I do not think there should be any problem: 1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" :) can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
> 
> 2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.

 

What about kernel change? You have to rebuild the unionfs module, but how to do it without a portage tree? When you boot the new kernel, you don't have a unionfs module. Or i'm missing something in here?...

Again IIRC when i was having kernel panic, kernel change and then "brand new kernel" the unionfs module at boot was

```
Loading module unionfs           [!!]

No module called `unionfs`       [!!]
```

Then the startup script printed "no module called unionfs" and i had no portage.

PS. Sorry for all those "IIRC" but this was a long weekend with "fight with hardware". /-:

----------

## gAzo0o

I had the same problem. Fixed it by setting the PORTDIR (/usr/portage by default) varible in /etc/make.conf

----------

## lazy_bum

Looks like it was my "kernel panic" problem.

The only problem left is 5-6 portage-$(date +%F).sqfs after 5-6 days. (-;

::edit::

And i will try again version 0.14 of the script.

----------

## Mousee

Not working at all here...

```

HyperonPS ~ # /etc/init.d/squash_portage start

 * Mounting Squashfs'ed Portage Tree... ...

mount: wrong fs type, bad option, bad superblock on /dev/loop0,

       missing codepage or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

                                                                          [ ok ]

```

Going back to the Wiki one, it works 10x as good as this one.

Thanks tho  :Smile: 

----------

## synss

 *Mousee wrote:*   

> Not working at all here...
> 
> ```
> 
> HyperonPS ~ # /etc/init.d/squash_portage start
> ...

 

If you do not understand what you are doing, better not do it, you are right. BTW, if you want to mount sth with squashfs and unionfs it helps having them installed.

----------

## adsmith

I'm glad to see you've found a way to make unionfs work with it reliably.  I'll give it a try shortly.

(My current solution is still to have my main server make the squahsed image for me on my weekly emerge --sync)

By the way, it is good to combine the new metadata system with squahsfs, so you don't just duplicate the data yet again.

To do this, add "-metadata-transfer" to FEATURES and add the line "portdbapi.auxdbmodule = cache.metadata_overlay.database" to /etc/portage/modules.

This avoids the /var/cache/edb metadata cache from being created, as that data already lives in /usr/portage/metadata in the squashed image.

----------

## dfy

I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?

----------

## gerardo

I've created a BASH script to cleanup the portage squash files older than one month:

```
#!/bin/bash

SQFILENAME=/var/tmp/portage-$(date -d "1 month ago" +"%Y-%m-%-d").sqfs

echo "Removing obsolete portage squash files:"

echo " all files before $SQFILENAME will be deleted."

for x in `ls /var/tmp/portage-20*.sqfs` ; do 

    if [[  $SQFILENAME > $x ]]  ; then

        rm -f $x

        echo "deleting $x..."

    fi

done

```

----------

## jpmayer87

This modification to the init script removes the squashfs file that was in use before the unionfs was modified.  It also handles the case where the new file is the same name as the old one.  

```

#!/sbin/runscript

# Copyright 1999-2006 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: $

depend() {

        need localmount

        #after vartmp

   }

start() {

        ebegin "Mounting squashfs'ed portage tree"

           [ -d $PORTAGE_RO ] || mkdir -p $PORTAGE_RO

      einfo "mounting squashfs RO..."

           mount -t squashfs -o loop,ro $PORTAGE_SQFS $PORTAGE_RO

      einfo "Done"

           [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW

           mount -t unionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR         

           eend 0

   }

stop() {                                                       

        ebegin "Updating portage tree"

   if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then

   einfo "Syncing the tree"

   PORTAGE_OLD=`/bin/readlink $PORTAGE_SQFS`

   einfo $PORTAGE_OLD

   [ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW

   mksquashfs $PORTDIR $PORTAGE_NEW -check_data 

   einfo "Checking for files to remove..."

   

   if [ $PORTAGE_OLD != $PORTAGE_NEW ]; then

   einfo "Different Days!";

   einfo "Removing old SquashFS..."; 

   echo "will remove $PORTAGE_OLD";

   rm -f $PORTAGE_OLD

   else

   einfo "Same Day!"

   fi

   [ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS

   ln -sf $PORTAGE_NEW $PORTAGE_SQFS

   else

   einfo "Nothing to do"

   fi

   eend 0

   ebegin "Unmounting the tree"

   umount $PORTDIR

   umount $PORTAGE_RO

   rm -rf $PORTAGE_RW

   eend 0

   }

```

questions, comments welcome.  This was my first real attempt at modifying a bash script

JP

----------

## B.marc

Just wanted to say, that this works for me very good. Now I'm wondering why Gentoo does not provide portage also as squashfs-files. Downloading 40MB to 50 MB and mounting it is much faster than a standard sync!

 *gerardo wrote:*   

> I've created a BASH script to cleanup the portage squash files older than one month:
> 
> ```
> #!/bin/bash
> 
> ...

 

Have a look at "man find". find is really good for tasks like this. And much more elegant:

```

find /var/tmp -maxdepth 1 -name *.sqfs -mtime 30 -exec rm -f {} \;

```

Cheers

Marc

----------

## synss

 *dfy wrote:*   

> I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?

 

Only the first time is time consuming, thereafter, everything is cached and faster, that is another advantage of having the tree compressed, the tree is in RAM. See the other thread for discussion.

----------

## mv

I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:

 *synss wrote:*   

> 
> 
> ```
> PORTAGE_RW="/dev/shm/portage"
> ```
> ...

 

Using a fixed name in a world-writable directory is a serious security issue. In this case, it might be an acceptable risk, because the initscript is hopefully started before any local attack might have a chance to get started, but even then: Don't better even think about calling the script later on with a restart option.

There are two other disadvantages of keeping PORTAGE_RW in a RAM disk: First, your previous changes to the portage tree are lost when your system crashes and the script cannot shutdown properly. The second disadvantage is that - surprise - it costs valuable RAM; the point is that you get practically no speed improvement from this ramdisk here (at least in normal cases), because the newly read data from a portage tree will remain in your disk ram anyway. So, it seems better not to use a ramdisk at all for portage_rw. In contrast, during shutdown, if you do not want to keep a backup of your .sqfs file or are short of disk space, it might be a good idea to build the new .sqfs file in a ramdisk.

It was already mentioned in this thread that keeping backups of older .sqfs files without limiting their number automatically is not necessarily the best idea. Morever, it seems better to use .tar.bz2 as a backup anyway for two reasons: First, these files need less space than .sqfs and, more importantly, if you are forced to boot from a kernel without squashfs support (e.g. from some rescue disk), you can still unpack the .tar.bz2 file if necessary. On the other hand, the disadvantage of using .tar.bz2 files as backup is of course the double packing time.

If you build a new kernel, you can simply compile in squashfs and loop support (or build the modules). However, to build in the unionfs support, you need access to the portage tree (unless you made a copy of the ebuild in advance). It was already mentioned that in this case you still can mount the portage tree at least readonly. However, the current script does not mount the tree in this case to the "usual" place readonly.

In the initscript of http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html there is a different attempt of a corresponding squashfs_portage with the above fixes/enhancements. Concerning the backups, you may choose in the configuration between keeping .sqfs or .tar.bz2, and you may pass options to mv to define the method and number of backups.

Edit: Fix some typos.Last edited by mv on Fri Oct 20, 2006 10:07 am; edited 1 time in total

----------

## synss

 *mv wrote:*   

> I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:
> 
> etc.
> 
> 

 

Interesting comments. I will see what you have done. I also have a small bash script, which takes care of making the first image, I wanted to incorporate it in my initscript. Having only one backup, following what has been posted here by sbdy else, is probably a good idea. I also realized recently that my overlay directory is getting bigger. It does not make much sense to have a compressed portage and a large overlay, that is another issue, I (or you) may take into account.

----------

## synss

Interesting, yes, but you could mention me in the header for example...

if you want to make the script more complex, you definitely should have something like

```
#!/bin/sh

archive="${1}"

sqfs_new="${2}"

tmpdir="./portage-tmp"

mkdir $tmpdir

tar -xvjf "${archive}" -C $tmpdir

mksquashfs ./$tmpdir/portage/ $sqfs_new -check_data

rm -ri ./$tmpdir

ln -sf $sqfs_new portage.sqfs

/etc/init.d/squash_portage restart
```

in your functions, adapted slightly to fit in. I have used that when I did not have a direct access to the Internet, had to downlad the tree, then unpack/squash the tree. It is very possible that the output of the 

```
tar -x
```

 shall be piped into 

```
mksquashfs
```

 to save some time, but I wrote that one in 2 min and let it-left it. I may update my script when I have time. Take it if you want.

Honestly, I find things like 

```
my_sqfsd () {

   local MY_TMP

   MY_TMP="$(mktemp "${TMPDIR}/squash_portage.XXXXXX")"

   chmod 644 "${MY_TMP}"

   mksquashfs "${PORTDIR}" "${MY_TMP}" \

      -check_data -noappend >/dev/null \

      && my_mv_sqfs_old && mv -f "${MY_TMP}" "${PORTAGE_SQFS}"

}
```

 difficult to read... but maybe it is just me... 

Anyway thank you for pointing at the security issues and for updating the script. But you may mention me...

----------

## mv

 *synss wrote:*   

> you could mention me in the header for example...

 

You are right, of course. I had done so in the beginning, but at some point, I decided to do a rewrite from scratch and copied the header just from somewhere else. This bug is repaired now.

 *Quote:*   

> if you want to make the script more complex, you definitely should have something like [...]

 

I don't get the point of this function: As I guess from the last command, you mean that you already have mounted some portage-tree, but it is just not up-to-date, and you want to update it from a .tar.bz2 file? Then why don't you just execute

```
cd /usr/portage && rm -rf *

tar xjvf /path_to_file_with_tree.tar.bz2

/etc/init.d/squash_portage restart
```

(actually, the last command is not really needed, as it will be handled automatically on shutdown, but it might be useful if you are short of diskspace).

 *Quote:*   

> Honestly, I find things like 
> 
> ```
> my_sqfsd () {
> 
> ...

 

They are difficult to read, but this is the way required by shell-programming- there is not much one can do to make it better readable: The quotation signs are all necessary to make sure that also file names with spaces work correctly. The "&&" syntax can hardly be rewritten with "if" if the exitcode should be correct. OK, one might split the last line into two and/or change its indentation. Not sure whether this is better readable...

Since you mentioned overlays in an earlier posting: If you keep your overlays in /usr/portage/local (which is e.g. the default of layman), then you need no special treatment for them... [only the rm-command in my above suggestion should then be modified, of course, although it is not really a mess if you executed it by accident, as long as you didn't call "/etc/init.d/squash_portage restart"].

----------

## nato.Otan

This sound an excelent alternative to run portage lean and swift. The method used here reminds me about puppyos. PuppyOS is extremely small distro that runs off RAM, and when you click on the programs they start instantly in the range of nano-seconds. Pretty cool.

Here I give the link to the descriptions of the bowels of puppyOS, and perhaps it could be implemented on this script to make it even Faster Than Light!!! And rename portage to Tachyon!!!

How Puppy Works

----------

## synss

Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2 if that would work that would obsolete these scripts (for the best I must say) however, I cannot compile compFUSEd. And I do not know how to deal with that kind of problems. If anybody is successful with that, I would be very interested.

Being able to mount a freshly downloaded tarball and update it would be great IMHO. The author uses it with his kernel sources and says he can compile into the archive and install his kernel. There are many things I can think of with these FUSE-related fs.

----------

## mv

 *synss wrote:*   

> Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2

 

As I understood, they write in their own archive format (based on lzo). I have doubts that any writable filesystem can be fast and simultaneously compress many small files effectively. However, I would like that somebody proves that I am wrong with this conjecture.   :Wink: 

 *Quote:*   

> I cannot compile compFUSEd. [...]There are many things I can think of with these FUSE-related fs

 

Yes, it is a pity that for most FUSE filesystems there are no .ebuilds in the portage tree.

Actually, I would also prefer to use an FUSE-based unionfs http://freshmeat.net/projects/unionfs-fuse/, because it needs no kernel module. However, I am too lazy to write an .ebuild.

----------

## synss

 *mv wrote:*   

> Actually, I would also prefer to use an FUSE-based unionfs http://freshmeat.net/projects/unionfs-fuse/, because it needs no kernel module. However, I am too lazy to write an .ebuild.

 

Agreed, however, there is an ebuild for funionfs in the overlay sunrise, I will see whether it works here and try to use it.

As of compFUSEd, I believe the author says he does support tar.bz2 without restrictions. There are other compressed fs who only work on lzo/gz but this one claims it can deal with .tar.bz2, so it would be worth trying. I contacted the author yesterday asking for support, I also suggested him to commit an ebuild (he is a gentoo user) to bugzy, that would earn him feedback and support, and if it turns out to work well enough, well, that would be really nice.

As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think. Only size on the HD matters. And for compressing other things, well then you can chose speed or compression.

BTW, I do not really understand why you have implemented tarred backups in the script: you can just download a new image of the tree from the ftp servers around here if ever something breaks badly, you do not need to do these time-consuming backups, do you? That's why experimenting new fs is fun with the tree, if you break it, just download a new one! Risk-free.

My point above you did not understand (probably was not clear) was that, in your check_config() function, if [[ ! -e "{PORTAGE_SQFS} ]]; blah, instead of echoing a message, you should actually squash the tree to a new image and then mount it. That would allow complete newbies to just download the script and run it and compression would be done automatically upon the first use.

Thank you for mentioning me and I think you've changed the identation and now it is easier to read  :Smile: 

[EDIT] well, funionfs compiled flawlessly (x86) and did not complain when I asked it to mount my tree, thus 

```
emerge layman

layman -a sunrise

/etc/init.d/squash_portage stop

modprobe -r unionfs

echo "~sys-fs/funionfs-0.4.2" >> /etc/portage/package.keywords

emerge funionfs
```

```
   #mount -t funionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR    

   funionfs -o dirs=$PORTAGE_RW: $PORTAGE_RO=RO -o allow_other NONE $PORTDIR
```

 that is in the start() thing.

```
/etc/init.d/squash_portage start
```

Does not complain, I will try syncing later though, for I just did and syncing several times a day is bad practice. But it does look good!  :Laughing:  If syncing works I will update my HOWTO, chmod'ing as you do is also a good idea, obviously, I will probably add a few of your things in mine, but I'll leave the tarred backups out...

[SECOND EDIT]sys-fs/fuse-2.6.0_rc1

stable, from emerge (not the one in the kernel) although form their website, they say funionfs should work with older versions of fuse, + my kernel is stable gentoo-source-2.6.17-r8

[THIRD EDIT]https://bugs.gentoo.org/show_bug.cgi?id=143026

an ebuild for UnionFsFuse don't ask me what the difference is or which is best between funionfs and UnionFsFuse (but give me the answer)

----------

## mv

 *synss wrote:*   

> As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think.

 

The point where I expect an enormous speed loss for .tar.bz2 files (if this would really be supported) is when the data is written to the tree, i.e. for emerge --sync. Just to have some numbers (for emerge --sync with a fast server and a fast connection):Standard (without squash_portage): 3-6 minutes, depending on the filesystem fragmentation.

With squash_portage: <1 minute (that's why I like it so much: It saves not only space but also time, even if one includes the compression time at the shutdown).

With pure .tar.bz2: ??? (I expect 30 minutes or more unless you have very much ram/swap, because every time the (uncompressed!) data cannot be held in ram anymore, practically the whole tree has to be compressed, since any change in the data in the middle of a .tar.bz2 file implies changes until the end of the file by the structure of .tar.bz2 files).

 *Quote:*   

> you can just download a new image of the tree from the ftp servers around here

 

Not every computer has easy access to the internet. Moreover, the portage tree is rather large (compared to the few data for a typical emerge --sync): It can really cost money if you have to pay for the bandwith.

 *Quote:*   

> you do not need to do these time-consuming backups, do you?

 

A backup can be also handy if the previous emerge --sync dropped some ebuild/patch which you wanted. Yes, there is a cvs of the tree, but a backup is more convenient in such cases.

 *Quote:*   

> instead of echoing a message, you should actually squash the tree to a new image and then mount it.

 

An automatic action for such a fundamental change is very dangerous: Once the new tree is mounted, a complete newbie will not be able to remove the old tree anymore. The script could of course also remove the old tree, but then if something goes wrong with the mounting (wrong kernel, insufficient disk space, ...) the user is left without a portage tree. I think the printed instruction is clear enough, and in this way the user can understand what is going on and why, and he has no bad surprise just by trying the script.

----------

## synss

Update!

I tried to include the feedback posted in this thread (including some that were posted a long time ago...) plus a few cosmetic changes and a bit more informative messages.

Changelog:

2006-11-19

Security enhancements thanks to mv

Only two sqfs images will now reside on the system: portage-old.sqfs and portage-current.sqfs

changed name of sqfs-related variables

more informational messages (it is made clear that the read-only image is mounted first and that it exists even if unionfs fails, that has always been the way it works but it seems not everybody got it)

correct usage of eend (i.e. !! when mount fails instead of OK)

tried to keep it short and simple without too many bashisms

The updated scripts are in the first post. Comments and bugs welcome.

----------

## mv

Update also for the other script.

It allows now to choose alternatively funionfs instead of unionfs, see funionfs.ebuild.

funionfs is based on fuse and thus works with all new kernels.

Since, in contrast, unionfs has not yet been ported for the stable kernel 2.8.19, funionfs is currently the only way to use this approach with the newest kernels (and it has the advantage that probably newer kernel releases won't break it again). The disadvantage is that funionfs is way slower than unionfs.

----------

## NaiL

Maybe the next thing is to build the squash_portage.ebuild that install all the config files and check for modules and with use flag to select from unionfs or funionfs

----------

## NaiL

Why don't do the same with:

/var/cache/edb

/var/db/pkg

Maybe it's a noob question   :Embarassed:   :Embarassed: 

----------

## synss

 *NaiL wrote:*   

> Why don't do the same with:
> 
> /var/cache/edb
> 
> /var/db/pkg
> ...

 

No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...

And mery christmas!

----------

## NaiL

 *synss wrote:*   

> 
> 
> No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...
> 
> And mery christmas!

 

Why with /usr/src/linux? because there are lots of files?

mery christmas to you! too :p

----------

## mv

 *NaiL wrote:*   

> Why don't do the same with:
> 
> /var/cache/edb
> 
> /var/db/pkg

 

These two are not worth the trouble, because they do not take too much space. It would probably be more interesting to use it for /usr/share/doc or /usr/src/linux, because these two should compress very well. In principle, you can simply use the same script repeatedly only with different directory/filename settings. Only the previous variable names are then misleading...

Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html is now called squash_dir and has no misleadings variable names.

Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are: 1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs).

2. You can explicitly avoid recompression during shutdown (by creating a certain magic file). In particular, if you use this script to compress /usr/src/linux, it is usually not worth to recompress /usr/src/linux after a kernel recompilation: Instead it is more reasonable to leave the modified files in a separate directory and simply use this directory on the next start (thus never do any recompression during shutdown). However, despite these changes, I am not sure whether it is a good idea to compress /usr/src/linux: If you compiled a new kernel but e.g. unionfs does not run on it yet, you might get troubles to access your original kernel read-writable so that you are able to compile unionfs...

Anyway, /usr/share/doc should cause less troubles (but it may also cause some, because e.g. the ebuild for unionfs might want to install something there).

----------

## NaiL

For me the first important thing is to avoid disk fragmentation in the measure of possible, secondly speed up the portage metadata updates.

Thats why i think to do that with all portage related directories.

The fact of gain disk space, for me is a welcomed secondary effect.

----------

## mv

 *NaiL wrote:*   

> secondly speed up the portage metadata updates

 

Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient.

Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs  (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up.

----------

## NaiL

 *mv wrote:*   

>  *NaiL wrote:*   secondly speed up the portage metadata updates 
> 
> Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient.

 

I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well?

 *mv wrote:*   

> Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs  (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up.

 

I'm currently only using unionfs with the synss script. I'll try your script when i have some time!   :Razz:  (It looks very well. "all about choice")

----------

## mv

 *NaiL wrote:*   

> I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well?

 

For most users it works. Some things like eclasses in overlays can cause troubles. If you need the latter, an alternative is to use the sqlite cache method.

 *Quote:*   

> "all about choice"

 

You will finally probably end up writing your own script - after all, these are all just very simple scripts with a very limited purpose. It's only the idea which is important...

----------

## kuku

 *mv wrote:*   

> 
> 
> Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html is now called squash_dir and has no misleadings variable names.
> 
> Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are: [list]1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs).

 

Hi - I've tried your script and i got some errors:

```
 ./squash_portage restart

 * Caching service dependencies ...                                       [ ok ]

 * Mounting /var/portage/portage.sqfs as /usr/portage ...

fusermount: mountpoint is not empty

fusermount: if you are sure this is safe, use the 'nonempty' mount option

 * Failed mounting /var/portage/change with funionfs [exit with 1]

mount: unknown filesystem type 'unionfs'

 * Failed mounting /var/portage/change with unionfs [exit with 32]

mount: wrong fs type, bad option, bad superblock on ,

       missing codepage or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

 * Failed even rbinding /usr/portage [exit with 32]   
```

this is my /etc/conf.d/squash_portage

```
DIRECTORY="$(. /etc/make.conf ; echo "${PORTDIR}")"

FILE_SQFS="/var/portage/portage.sqfs"

FILE_SQFS_OLD="/var/portage/portage.old.sqfs"

PREFER="funionfs"

DIR_CHANGE="/var/portage/change"

```

any hint ?

----------

## mv

Thanks for the feedback. It seems that I introduced an error briefly before release but after testing the special case of empty DIR_SQUASH (which I do not recommend, anyway). Setting DIR_SQUASH to some empty directory should fix your problem.

Today's version of the script should hopefully also fix the bug.

I must say that I had some bad experience with funionfs: It seems that sometimes(?) portage cannot "see" the files mounted by funionfs, e.g. it occassionally complains about missing eclasses or even a wrong profile link (although an ls shows that the files are there): I guess some directory function is not always properly mapped by funionfs.

Does anybody have an idea what might be wrong?

So, in the moment, I can recommend only unionfs which you apparently haven't installed:

 *kuku wrote:*   

> 
> 
> ```
> mount: unknown filesystem type 'unionfs'
> ```
> ...

 

BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work...

----------

## kuku

 *mv wrote:*   

> 
> 
> BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work...

 

I only use funionfs of 2.6.19 so it might be helpfull  :Wink: 

----------

## mv

See unionfs.tar.gz on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html for ebuild/instructions on how to run the current unionfs snapshot with kernel 2.6.19.

----------

## Maxwell

Hi!

I'm trying to use an old pc as an host for some network services and i wanted to use its (64M) ram to speed things up. So when i searched the forums i found this.

As you all have tried several ways of doing this (sort of),  how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%?

Thanks for the help!

----------

## synss

 *Maxwell wrote:*   

> As you all have tried several ways of doing this (sort of),  how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%?

 

This is going seriously off-topic so you might as well start a new thread, but what you are looking for is initramfs. Data compression can be achieved in there with squashfs or cramfs, look for live CDs too, some of them (knoppix for example) do that.

----------

## Maxwell

Thanks for the tip!

 *Quote:*   

> This is going seriously off-topic

 

Yeah, i know, but i've already started a thread, but no replies were found. Sometimes i just prefer to ask the right guy  :Smile: 

 *Quote:*   

> but what you are looking for is initramfs.

 

Ok then!

BTW, how much compression can you get from the portage tree? how much ram is being used?

thanks for the help

----------

## jpmayer87

With squashfs I reduced the entire portage tree to about 44 Meg, that includes overlays as well.

JP

----------

## mv

Update: The new initscript version on http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html now also supports (and even prefers) aufs over unionfs. An ebuild for aufs is available from here.

----------

## jsosic

Squashfs not only saves space, but makes portage almost twice faster!!!

I've made two tests. First is update-eix, and second is emerge -upDv world. Computer is restarted, and then tests were made one after another. Then, Squasfs+unionfs was setup and computer restarted, with tests repeated. After seeing this, i just removed classic portage tree, I'm staying on squash!!!

```
update-eix   1.66s user 2.31s system  5% cpu *1:06.70 total*

update-eix   1.32s user 1.60s system 82% cpu *   3.52 total*
```

```
emerge -upDv world   9.35s user 1.08s system 31% cpu *32.987 total*

emerge -upDv world   9.20s user 1.52s system 57% cpu *18.761 total*
```

CPU is skyhigh, but you can always use nice! And disk almost doesn't thrash it's head around with this method, so it's much ionicer than the original portage. I don't know why this method doesn't become standard for portage tree?!?!

AWESOME

 :Twisted Evil:   :Twisted Evil:   :Twisted Evil: 

----------

## stdPikachu

 *mv wrote:*   

>  *NaiL wrote:*   Why don't do the same with:
> 
> /var/cache/edb
> 
> /var/db/pkg 
> ...

 

```
prospero ~ # du -chs /var/cache/edb/ /var/db/pkg/

111M    /var/cache/edb/

172M    /var/db/pkg/

282M    total
```

That's too much space in my book. The average file size is utterly tiny as well, no wonder portage thrashes the disc so much.

Are there any plans aftoot to have inline compression built into portage? IMHO using the squashfs/unionfs kludge is annoying since you don't get user transparency. Surely there's a call for a PortageFS to be enabled by default? Portage may have some cool features, but the overheads of it are *massive* and anything that brings those down would be a good thing in my book. Using the best part of 1GB space of uncompressed text just for package management seems ludicrous to me.

Failing that, a utility that automatically creates and mounts squashfs/similar images after every emerge --sync might be useful.

----------

## mv

 *stdPikachu wrote:*   

>  *mv wrote:*    *NaiL wrote:*   Why don't do the same with:
> 
> /var/cache/edb
> 
> /var/db/pkg 
> ...

 

But it's not the true usage if you use reiserfs (the main bulk is then taken by the environment.bz2 files which won't compress much). Moreover, /var/cache/edb will be much smaller if you use the metadata_overlay database (there is certainly a thread or a wiki how to do that) - mine is just 5M.

Nobody prevents you from using two further symlinks to squash_dir to do the same with the mentioned directories.

However, you should of course be aware that /var/db/pkg changes much more often than /usr/portage and that portage might fail to work if it is not possible to mount this directory writable (e.g. you might get problems to emerge aufs if aufs is not running). Of course, you can still unsquashfs the directory in such an emergency situation...

----------

## mv

 *mv wrote:*   

> But it's not the true usage if you use reiserfs

 

I tested now on my system with reiserfs: Uncompressed, /var/db used 88MB, compressed it uses 50MB.

You must estimate yourself whether this is worth the possible problems (if you cannot emerge aufs due to missing writing permissions). Of course, if you use a braindead filesystem like ext3 for /var/db you should really think about changing it...

----------

## eatnumber1

I've been using mv's scripts for quite some time. Thx to you!

Anyway, I'm on a laptop and was working on tmpfs mounting stuff in /var to reduce disk spinups, and I got to directories like /var/run which should be able to be tmpfs mounted, but can't because programs expect directories to be in there. Well, I figured, I should just use squashfs and tmpfs mount the rw branch! After much work, I eventually got /var/log, /var/run, and /var/db/pkg mounted in this way (I also tmpfs mounted /var/lock).

The challenge to this was that I had to get the initscript to start before bootmisc... Apparently portage has something to prevent this. Fortunatley, it also has a way to override it. Just put the following in /etc/runlevels/boot/.critical

```
checkroot

modules

checkfs

localmount

clock

/* Your squashfs scripts should be here */

bootmisc
```

I also had to change the initscript's depend function to

```
depend () {

        need localmount

        before bootmisc

}
```

So now I have two squash_ initscripts. One named squash_dir and one named squash_early, and all the actual scripts that I start are symlinks of one of the two.

On a different topic, I don't remember where the images were originally stored, or where the branches were originally mounted, but I do remember changing it to something I thought much better so I thought I'd share it. I have a /var/squashfs in which is the following directories: images, mnt, and tmp. Then in images is another directory called old. So in images is the current squashfs images, in images/old is backups of the most recent old image, mnt and tmp contain directories for each mount I have (linux for my kernel sources, etc...) which are my ro and rw branches.

Last thing i've noticed is that when stopping and starting the initscript, it is not correctly releasing the loop device and when I try to manually do it, it gives me an error.

P.S. Can someone tell me the latest point at which /var/db/package should be started?

P.P.S. One suggestion: To change the initscript to use a single config file similar to how gentoo does /etc/init.d/net.lo and /etc/conf.d/net. That way it would make it easier to manage multiple squashfs locations.

----------

## mv

 *eatnumber1 wrote:*   

> /var/log, /var/run

 

I am not sure whether it is a really good idea to rely on unionfs/aufs for such critical parts of the system as long as these modules are not part of the kernel - if something goes wrong with a new kernel, you won't be able to boot!

Maybe a better way for these directories is to copy them to a tempfs and then use mount --bind (and for shutdown first unmount and then copy [the changed parts] back). Moreover, you will not waste a loop device this way!

Concerning /var/log, I have even more doubts, because if your system crashes you would probably like to be able to read its last words...

 *Quote:*   

> and /var/db/pkg mounted in this way

 

This is only needed for emerge commands, so you can treat it in the same way as /usr/portage (in particular, no "early" mounting is needed),

 *Quote:*   

> On a different topic, I don't remember where the images were originally stored, or where the branches were originally mounted, but I do remember changing it to something I thought much better so I thought I'd share it. I have a /var/squashfs in which is the following directories: images, mnt, and tmp. Then in images is another directory called old. So in images is the current squashfs images, in images/old is backups of the most recent old image, mnt and tmp contain directories for each mount I have (linux for my kernel sources, etc...) which are my ro and rw branches.

 

This has the disadvantage that all your squashfs images must remain on the same partition and that this partition is not necessarily the same as that of the original directory. I usually use the following configuration in /etc/conf.d/squash*

 *Quote:*   

> # The directory you want to keep compressed:
> 
> DIRECTORY='/gentoo32/usr/share/games'
> 
> #Comment out if you want backups:
> ...

 

i.e. all data which I might really need to access is in the same (parent) directory as the directory I want to squash.

 *Quote:*   

> Last thing i've noticed is that when stopping and starting the initscript, it is not correctly releasing the loop device and when I try to manually do it, it gives me an error.

 

I had this problem 3 times after a long uptime and after often starting/stopping/restarting several squash* scripts in various orders, but I was never able to reproduce it later on. I suspect this is a bug in mount itself. (But maybe you have run into a different problem?)

The usage of loop devices is the most serious drawback of the whole approach anyway, because it limits the number of directories which you can manage in this way dramatically. For example, I have a 64bit and a 32bit installation, and so it is reasonable to use it currently for /gentoo{32,64}/{var/db,usr/{share/games,src/kernel}, my joint portage tree (without distfiles) and a joint directory for $distfiles/{svn-src,cvs-src} which means that 8 loop devices are in permanent usage on my system...

 *Quote:*   

> P.P.S. One suggestion: To change the initscript to use a single config file similar to how gentoo does /etc/init.d/net.lo and /etc/conf.d/net. That way it would make it easier to manage multiple squashfs locations.

 

But I would like to keep one init-script for each squashfs-mounted directory, so that you can easily force compression of a certain directory with 

```
/etc/init.d/squash... restart
```

 (e.g. for backup purposes which is particularly useful for the kernel). As I understand, you do not mind having separate /etc/init.d/squash* files, but you want only one /etc/init.d/squash_dir file. I am not convinced that this is a clearer configuration, because then in this file each variable would need an additional index to determine the name. Do you know which mechanism is used in /etc/init.d/net.lo to read /etc/conf.d/net and to determine "lo"? When I glanced through it, it seems to make use of undocumented features (e.g. ${svclib}) so I am afraid such an initscript won't work after the next baselayout update.

----------

## eatnumber1

 *Quote:*   

> I am not sure whether it is a really good idea to rely on unionfs/aufs for such critical parts of the system as long as these modules are not part of the kernel - if something goes wrong with a new kernel, you won't be able to boot!

 

I use kamikaze sources which has unionfs in the kernel... so even if I mess something up, i'll still be able to mount it. Also, the directories in /var/run are not so critical as to make the system completely unbootable, it just makes it so many programs cannot start. That is still very fixable. Lastly, I could always just revert to my old kernel, fix whatever I did wrong and boot into the new one again =).

 *Quote:*   

> Moreover, you will not waste a loop device this way!

 

When I noticed the fact that it was unable to release the loop devices after unmounting, I increased the number of loop devices to 64... so I don't think wasting loop devices should be worried about.

 *Quote:*   

> Concerning /var/log, I have even more doubts, because if your system crashes you would probably like to be able to read its last words...

 

This is true, and is the one big drawback to having /var/log mounted in this way, I think the benefit of not having the disk spin up *almost* randomly to write the logs outweighs the potential loss of data. Also, if you crashed due to a reproduceable error, you could always just disable the init script and reproduce the error, then you'd be able to read the logs.

 *Quote:*   

> This has the disadvantage that all your squashfs images must remain on the same partition and that this partition is not necessarily the same as that of the original directory.

 

I see the problem of having all of them on one partition, but I prefer to have them in one easily manageable, central location. I don't see fact that the partition is not necessarily the same as that of the original directory as a disadvantage... am I missing something significant?

 *Quote:*   

> Last thing i've noticed is that when stopping and starting the initscript, it is not correctly releasing the loop device and when I try to manually do it, it gives me an error.

 

Upon (re)investigating this problem, it turns out that a loop device is only locked indefinitely if a new squashfs image is created. As an example, if I recently emerge --sync'ed, and I then re-squash the tree, I then notice that losetup -a has two /usr/portage devices, but mount only reports one as in use.

```
kiki init.d # losetup -a

*snip*

/dev/loop/5: [0803]:2409097 (/var/squashfs/images/portage.sqfs)

/dev/loop/6: [0803]:1440 (/var/squashfs/images/portage.sqfs)
```

```
kiki init.d # mount   

*snip*

/var/squashfs/images/portage.sqfs on /var/squashfs/mnt/portage type squashfs (ro,loop=/dev/loop6)

unionfs on /usr/portage type unionfs (rw,dirs=/var/squashfs/tmp/portage=rw:/var/squashfs/mnt/portage=ro)
```

So I then do losetup -d /dev/loop/5 to try to get rid of the unused one, and I get this:

```
kiki init.d # losetup -d /dev/loop/5

ioctl: LOOP_CLR_FD: Device or resource busy
```

Also, fuser /dev/loop/5 and lsof | grep /dev/loop/5 produce no output (nothing is using it).

Is this the problem you were having?

 *Quote:*   

> The usage of loop devices is the most serious drawback of the whole approach anyway, because it limits the number of directories which you can manage in this way dramatically.

 

Like I began to say earlier, I have max_loop=64 on my kernel line in grub which gives me 64 loop devices to play with... so I don't think i'll be running out any time soon, and if I do I can just increase it again.

 *Quote:*   

> But I would like to keep one init-script for each squashfs-mounted directory, so that you can easily force compression of a certain directory with 
> 
> ```
> /etc/init.d/squash... restart
> ```
> ...

 

I was suggesting one squash_dir script in /etc/init.d which the initscripts you use is symlinked off of that (which you do) and one config file for all squash_* in /etc/conf.d (like /etc/conf.d/squash_unionfs or something) much like how /etc/conf.d/net works.

 *Quote:*   

> Do you know which mechanism is used in /etc/init.d/net.lo to read /etc/conf.d/net and to determine "lo"? When I glanced through it, it seems to make use of undocumented features (e.g. ${svclib}) so I am afraid such an initscript won't work after the next baselayout update.

 

I just had a look at the net.lo script and it does seem to use those undocumented features

```
*snip*

local iface="${SVCNAME#*.}"

*snip*
```

So if you are afraid of using these which probably won't go away, but may, you could do some sed or awk magic in the file, parsing one line at a time and use a configuration like

```
linux directory /usr/src/linux

linux order unionfs
```

This way is easier to manage config files for multiple squashfs images, and most importantly provides the potential for global variables such as 

```
all order unionfs
```

P.S. One more feature i'd like to request: The ability to add stuff to the depend() function w/o having to change the initscript. The net.lo script provides this functionality (although it does depend on undocumented features). See the net.lo script for an example.

P.P.S. My apologies for the long post.

----------

## stdPikachu

 *mv wrote:*   

> But it's not the true usage if you use reiserfs (the main bulk is then taken by the environment.bz2 files which won't compress much). Moreover, /var/cache/edb will be much smaller if you use the metadata_overlay database (there is certainly a thread or a wiki how to do that) - mine is just 5M.

 

Hmm. I have had problems with Reiser before, and am very much an ext3/JFS man. I don't think that the package database should depend on using a particular filesystem for good performance.

Thanks for pointing portage's capability to use various DB backends though; I've just switched my main machine to use sqlite (one of my favourite little apps, I wish more progs supported it!) and now have a dependency cache of 23MB (uncompressed) in a single file which has reduced disc thrashing significantly, plus the ability to perform SQL dep queries  :Smile: 

Are there any plans to do similar things for the package database (/var/db/pkg) too?

----------

## mv

 *eatnumber1 wrote:*   

> I increased the number of loop devices to 64...

 

I was not aware that this is possible, because the kernel did not seem to have a configuration option for this.

 *Quote:*   

> if I recently emerge --sync'ed, and I then re-squash the tree, I then notice that losetup -a has two /usr/portage devices

 

How do you re-squash the tree? With /etc/init.d/squash_portage restart? I cannot not reproduce this.

Maybe for some reason some part of /usr/portage is in use so that the umount fails and so the loop device is not freed. However, the script will even try a "lazy" umount in such a case, so something really severe must prevent the umount (but perhaps "lazy" does not work well with loop devices - maybe this is the "bug" in umount which I meant).

 *Quote:*   

> I have max_loop=64 on my kernel line [...] and if I do I can just increase it again.

 

What is the maximal possible number? Certainly not more than 256, because each loop has its own minor device number...

 *Quote:*   

> I was suggesting one squash_dir script in /etc/init.d which the initscripts you use is symlinked off of that (which you do) and one config file for all squash_* in /etc/conf.d (like /etc/conf.d/squash_unionfs or something) much like how /etc/conf.d/net works.
> 
>  *Quote:*   Do you know which mechanism is used in /etc/init.d/net.lo to read /etc/conf.d/net and to determine "lo"? When I glanced through it, it seems to make use of undocumented features (e.g. ${svclib}) so I am afraid such an initscript won't work after the next baselayout update. 
> 
> I just had a look at the net.lo script and it does seem to use those undocumented features
> ...

 

Actually, it must use much more, because this won't load /etc/conf.d/net automatically...

 *Quote:*   

> do some sed or awk magic in the file, parsing one line at a time and use a configuration like[...]

 

No, that is not a good idea: The power of gentoo configuration is that it uses bash, and so you can easily write all sort of magic within your config (in particular, testing for all sort of conditionals using external programs, sourcing other files etc):

 *Quote:*   

> most importantly provides the potential for global variables such as 
> 
> ```
> all order unionfs
> ```
> ...

 

You want to set global variables? Just source some "squash_defaults" file from within your config file which sets these variables. You want only one file? Then link all /etc/conf.d/squash_* to the same file and use the mentioned undocumented magic by yourself. Here is an example (assuming that you have links from /etc/conf.d/squash_portage and /etc/conf.d/squash_db to this file):

```
db_vars () {

        DIRECTORY='/var/db'

        FILE_SQFS_OLD="${DIRECTORY}.sqfs.bak"

}

portage_vars () {

        DIRECTORY='/usr/portage'

}

"${SVCNAME#squash_}"_vars

DIR_CHANGE="${DIRECTORY}.changes"

DIR_SQUASH="${DIRECTORY}.readonly"

DIR_TMP='/tmp'
```

 *Quote:*   

> P.S. One more feature i'd like to request: The ability to add stuff to the depend() function w/o having to change the initscript.

 

Untested: 

```
#!/sbin/runscript

source /etc/init.d/squash_dir

depend () {

   # new dependencies

   before localmount

}
```

----------

## steveL

 *stdPikachu wrote:*   

> Hmm. I have had problems with Reiser before, and am very much an ext3/JFS man. I don't think that the package database should depend on using a particular filesystem for good performance.

 

I am a recent convert to ext3 after 6 years exclusively on reiser. I saw this: http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc but didn't quite believe it til I was working on update which I run via symlink. I was getting all kinds of random garbage in the script and weird bash errors. Finally I switched to ext3, and it's never happened again. I still use reiser for /usr/portage as it's all signed and recoverable via sync (and fetch for distfiles) but I wouldn't recommend it for anything I wanted to keep. ext2 is fine for tmp imo.

 *Quote:*   

> Thanks for pointing portage's capability to use various DB backends though; I've just switched my main machine to use sqlite (one of my favourite little apps, I wish more progs supported it!) and now have a dependency cache of 23MB (uncompressed) in a single file which has reduced disc thrashing significantly, plus the ability to perform SQL dep queries 
> 
> Are there any plans to do similar things for the package database (/var/db/pkg) too?

 

Hmm, sounds really interesting.. /me logs onto irc.freenode.org to find out more..  :Wink: 

----------

## mv

This is now really not the topic of the thread, but however let me reply:

 *steveL wrote:*   

> I saw this: http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc

 

This article suggests that the ext3 journal policy could save you from power failures for bad hardware, but this is simply wrong: The problem is that in case of power failure not only the sector you wanted to write gets wrong data but perhaps even the next sector(s) or, in case that the sector register looses power first, even a completely false sector gets updated. There is no file system which can decrease this risk. In fact, the risk is even higher with ext3, because it simply writes more data (because of the journal redundancy).

 *Quote:*   

> I was getting all kinds of random garbage in the script and weird bash errors. Finally I switched to ext3, and it's never happened again.

 

Such bad and unlikely errors can happen everywhere. If they happen while you used ext3, they would probably never happen again after you switched to reiserfs. There are many people who use reiserfs (or ext3 or whatever) for many years and never had any problem.

 *Quote:*   

> Are there any plans to do similar things for the package database (/var/db/pkg) too?

 

I hope not, because this is something you really want to have in plain text. However, internally portage uses another database in /var/cache/edb for those parts of /var/db/pkg which it needs often (that's why /var/cache/edb is 5M on my machine, although I use the metadata backend).

----------

## steveL

 *mv wrote:*   

> This article suggests that the ext3 journal policy could save you from power failures for bad hardware, but this is simply wrong: The problem is that in case of power failure not only the sector you wanted to write gets wrong data but perhaps even the next sector(s) or, in case that the sector register looses power first, even a completely false sector gets updated. There is no file system which can decrease this risk. In fact, the risk is even higher with ext3, because it simply writes more data (because of the journal redundancy).

 

Er that didn't make much sense to me, it must be a linguas issue, sorry. I don't care if it's writing a bit more if its emphasis is on keeping my data safe first, and performance second. Each to their own, although it seems pretty clear:

 *Quote:*   

> Why doesn't ext3 get hit by this?  Well, because ext3 does physical block journalling.  This means that we write the entire physical block to the journal and, only have the updates to the journal are commited, do we write the data to the final location on disk.  So if you yank out the power cord, and inode tables get trashed, they will get restored when the journal gets replayed.

 

It's not even about the power off issue to me; portage (on reiser) came up with garbage in its files; a --sync soon sorted that, but it's happened to others on IRC as well.

 *mv wrote:*   

>  *Quote:*   I was getting all kinds of random garbage in the script and weird bash errors. Finally I switched to ext3, and it's never happened again. Such bad and unlikely errors can happen everywhere. If they happen while you used ext3, they would probably never happen again after you switched to reiserfs. There are many people who use reiserfs (or ext3 or whatever) for many years and never had any problem.

 

Yes I know, like I said, I used reiser exclusively for 6 years. But there is no way those errors were acceptable to me, and it does make me wonder about glitches I used to blame on buggy software in Mandrake. And it doesn't happen with ext3 at all, so my experience means I cannot recommend reiser in good faith. YMMV as ever.

 *mv wrote:*   

>  *Quote:*   Are there any plans to do similar things for the package database (/var/db/pkg) too? 
> 
> I hope not, because this is something you really want to have in plain text. However, internally portage uses another database in /var/cache/edb for those parts of /var/db/pkg which it needs often (that's why /var/cache/edb is 5M on my machine, although I use the metadata backend).

 

Well I like plain-text too. But I really recommend using -metadata-transfer if you want to speed up sync. Kuroo can't deal with it last time I looked, which was why we wrote update in the first place. As for a db backend it's been talked about for ages, and pkgcore has support for various backends, although I am sure they could do with help on them. It's in the tree now as well (autounmask really helps :)

----------

## mv

 *steveL wrote:*   

> I don't care if it's writing a bit more if its emphasis is on keeping my data safe first, and performance second.

 

My point is that especially in case of power failures/hardware problems/... there is nothing which can guarantee that your data is safe. It is simply a matter of "luck" whether writing more redundancy gives you more security or more risk, because each writing operation (even into a journal) is a risk. Reiser4 is of course better in this respect, since by its "dancing trees" concept it has the full security without doubling the writing operations. But unfortunately, one cannot expect to see Reiser4 ever in the official kernel (and moreover, AFAIK Reiser4 has serious fragmentation problems). In fact, all filesystems in the linux kernel have so serious drawbacks that none of them deserves be used.   :Wink: 

 *Quote:*   

> It's not even about the power off issue to me; portage (on reiser) came up with garbage in its files

 

I understood this, but something must have been the cause of this, although it is practically impossible to find out the reason now: Either some hardware failure/power failure or some software failure must have been the cause. (Many years ago there was a bug in reiserfs which could cause such an effect, but this bug was fixed also many years ago, so practically I think one can exclude software failure meanwhile.)

 *Quote:*   

> But there is no way those errors were acceptable to me

 

Probably those people who swear on reiserfs/ext3 are all those who had such inacceptable (but actually usually due to hardware) errors with ext3/reiserfs. I am also not exception in this respect  :Sad:   Moreover, I had followed some of the discussions about the (non-)inclusion of the well-functioning e2comp code (compression for ext2) several years ago, and this were reasons enough for me to not use code of the ext2/3 maintainers anymore. This "it is new and works well, but I haven't written it, therefore it is bad by definition, and we must invent some reason to abolish it"-attitude is unbearable to me.

 *Quote:*   

> Well I like plain-text too. But I really recommend using -metadata-transfer if you want to speed up sync.

 

Why do you say "But"? I also suggested this (although I didn't have a link at hand...). One has to distinguish the two types of databases which portage needs: One is the database of all packages in the portage tree (/usr/portage/metadata and /var/cache/edb/dep). The other is the database of all installed packages and their files (/var/db/pkg). metadata-transfer (or the db backends) only affect the former. But I was talking about the latter which is more delicate, because any problem in the corresponding database would wreck your whole gentoo installation. Since all portage substitutes should update this database correctly when installing packages and, moreover, already many scripts rely on this database, I would not like to see dramatic changes in its format.

----------

## steveL

 *mv wrote:*   

> My point is that especially in case of power failures/hardware problems/... there is nothing which can guarantee that your data is safe.

 

That's what journalling is for; the point is that the syscall shouldn't return until the write to the journal at least is completed. If there's a power-failure, the write isn't marked as committed (and yes this is an awful lot like database transactions.) So yeah, maybe there's scope for redesign in that, but reiser isn't doing it atm. Reiser3 seems to have died, and 4 is still way too experimental to recommend.

 *Quote:*   

> It is simply a matter of "luck" whether writing more redundancy gives you more security or more risk, because each writing operation (even into a journal) is a risk.

 

See above. You appear to be saying that the risk is proportional to the amount of data written, which may be true, but it has no bearing on this discussion afaict.

 *Quote:*   

> Reiser4 is of course better in this respect, since by its "dancing trees" concept it has the full security without doubling the writing operations. But unfortunately, one cannot expect to see Reiser4 ever in the official kernel (and moreover, AFAIK Reiser4 has serious fragmentation problems). In fact, all filesystems in the linux kernel have so serious drawbacks that none of them deserves be used.  
> 
> 

 

Yes dear  :Wink: 

 *Quote:*   

>  *Quote:*   It's not even about the power off issue to me; portage (on reiser) came up with garbage in its files 
> 
> I understood this, but something must have been the cause of this, although it is practically impossible to find out the reason now: Either some hardware failure/power failure or some software failure must have been the cause. (Many years ago there was a bug in reiserfs which could cause such an effect, but this bug was fixed also many years ago, so practically I think one can exclude software failure meanwhile.)

 

Er on what do you base that assertion? Are you saying it's not a software failure? Odd then that it never happens with ext3, and that others report similar issues when they notice them. Even weirder is your assertion that since one bug was fixed in Reiser many years ago, there are no others.

 *Quote:*   

>  *Quote:*   But there is no way those errors were acceptable to me 
> 
> Probably those people who swear on reiserfs/ext3 are all those who had such inacceptable (but actually usually due to hardware) errors with ext3/reiserfs. I am also not exception in this respect   Moreover, I had followed some of the discussions about the (non-)inclusion of the well-functioning e2comp code (compression for ext2) several years ago, and this were reasons enough for me to not use code of the ext2/3 maintainers anymore. This "it is new and works well, but I haven't written it, therefore it is bad by definition, and we must invent some reason to abolish it"-attitude is unbearable to me.

 

Please get this straight: it is exactly the same hardware. And please don't accuse me of NIH - i never wrote any of the filesystems I have used in the past 25 years. So yeah, when I see a filesystem suddenly produces random garbage I tend not to want to use it. Do you blame me? And there was no reason for me to notice it before; it's clearly something that shows up with symlinks (and there may well be other bugs.)

And yes, I push the machine just as hard now as I did before, so overheating etc are not the cause. The only problem since switching I have had, was with the portage tree on reiser. I'm happy to keep that, as I described before.

 *Quote:*   

>  *Quote:*   Well I like plain-text too. But I really recommend using -metadata-transfer if you want to speed up sync. 
> 
> Why do you say "But"? I also suggested this (although I didn't have a link at hand...). One has to distinguish the two types of databases which portage needs: One is the database of all packages in the portage tree (/usr/portage/metadata and /var/cache/edb/dep). The other is the database of all installed packages and their files (/var/db/pkg). metadata-transfer (or the db backends) only affect the former. But I was talking about the latter which is more delicate, because any problem in the corresponding database would wreck your whole gentoo installation. Since all portage substitutes should update this database correctly when installing packages and, moreover, already many scripts rely on this database, I would not like to see dramatic changes in its format.

 

Sure, that's why I like plain-text  :Smile:  Yes, missed your earlier post about metadata-transfer, my bad.

----------

## mv

 *steveL wrote:*   

> That's what journalling is for

 

No, this is a common misunderstanding. No journalling can save you from problems during power failure. It can only reduce the risk a little bit so that chances are good that you still have a consistent filesystem if the failure was not too bad.

 *Quote:*   

> the point is that the syscall shouldn't return until the write to the journal at least is completed. If there's a power-failure, the write isn't marked as committed (and yes this is an awful lot like database transactions.) So yeah, maybe there's scope for redesign in that, but reiser isn't doing it atm.

 

reiser is doing this. The argument in the url you posted is that if power loss occurs during playing the journal and the disk destroys for this reason more data then it was supposed to write (namely a whole sector) reiser cannot restore this whole sector, because not the whole original sector but only the modified data is stored in the journal. But of course, nobody can guarantee that the dying disk destroys only one sector during dying, i.e. ext3 is not really much safer than reiser in this situation (and as mentioned, since reiser writes less data, there are other situations in which reiser is safer, so there is not much difference from the security viewpoint).

What you actually want (that no syscall should return until it can guarantee a successfull operation) is an atomic commit. The only filesystem which supports this conceptually is Reiser4. That's why I was not really joking when I said that each filesystem in the kernel has serious drawbacks: Reiser4 is the first one which is conceptionally good (but has other drawbacks).

 *Quote:*   

> Reiser3 seems to have died, and 4 is still way too experimental to recommend.

 

Unfortunately.

 *Quote:*   

> Er on what do you base that assertion? Are you saying it's not a software failure? Odd then that it never happens with ext3, and that others report similar issues when they notice them.

 

If you look for a while in various forums you will find such reports for each filesystem. And if you then follow the discussions, it usually turns out to be a hardware issue which happened once (or in some cases even regularly) for whatever reason.

 *Quote:*   

> Please get this straight: it is exactly the same hardware. And please don't accuse me of NIH

 

I am sure that this happened to you as you described. But it happened to you once in 6 years. Once within 6 years for whatever reason some block was written wrong (or to the wrong place) in such a way that your system lost consistency. It is not so unlikely that this happens to one out of several thousand persons after a long while and could happen with every filesystem. And it is not so surprising that it does not happen immediately again to the same person. I am rather sure that if you reformatted a new reiserfs instead of ext3, you would not have the problem either.

 *Quote:*   

> Even weirder is your assertion that since one bug was fixed in Reiser many years ago, there are no others.

 

I only heard about this one bug in reiserfs which could cause such a bad data loss (and actually many people who speak about bad experience with reiserfs were a victim of this bug). Of course, this does not mean that there cannot be another bug, but at least in the discussions which I had observed so far, it seems that nobody has experienced anything which was statistically so extraordinary or reproducible. So it seems likely that there is not another such serious bug (of course, there were some smaller bugs which had been unobserved for quite a while, but such "cosmetic" bugs are different issue.)

----------

## steveL

 *mv wrote:*   

> No journalling can save you from problems during power failure. It can only reduce the risk a little bit so that chances are good that you still have a consistent filesystem if the failure was not too bad.

 

Er sorry, I disagree. Journalling can and should provide this. My aside about the design was on whether this has to be done with the dual write you mentioned, or whether the logical approach can have merit.

 *Quote:*   

>  *Quote:*   the point is that the syscall shouldn't return until the write to the journal at least is completed. If there's a power-failure, the write isn't marked as committed (and yes this is an awful lot like database transactions.) So yeah, maybe there's scope for redesign in that, but reiser isn't doing it atm. 
> 
> reiser is doing this. The argument in the url you posted is that if power loss occurs during playing the journal and the disk destroys for this reason more data then it was supposed to write (namely a whole sector) reiser cannot restore this whole sector, because not the whole original sector but only the modified data is stored in the journal. But of course, nobody can guarantee that the dying disk destroys only one sector during dying, i.e. ext3 is not really much safer than reiser in this situation (and as mentioned, since reiser writes less data, there are other situations in which reiser is safer, so there is not much difference from the security viewpoint).

 

Hmm.. I need to think about that for a while, tbh. One thing that springs to mind is if the sector can never be restored by reiser (since it only exists in one place) and might be recoverable by ext3, I am already happier with ext3. But like I say, I am not concerned about the power loss situation, since I have never had a problem with them since I started using journalling.

 *Quote:*   

> What you actually want (that no syscall should return until it can guarantee a successfull operation) is an atomic commit. The only filesystem which supports this conceptually is Reiser4. That's why I was not really joking when I said that each filesystem in the kernel has serious drawbacks: Reiser4 is the first one which is conceptionally good (but has other drawbacks).
> 
> 

 

Yeah I know; I am glad you know the terminology as well, it makes the discussion easier  :Smile: 

 *Quote:*   

>  *Quote:*   Er on what do you base that assertion? Are you saying it's not a software failure? Odd then that it never happens with ext3, and that others report similar issues when they notice them. 
> 
> If you look for a while in various forums you will find such reports for each filesystem. And if you then follow the discussions, it usually turns out to be a hardware issue which happened once (or in some cases even regularly) for whatever reason.

 

Yes but the assertion you made was:

 *Quote:*   

>  this bug was fixed also many years ago, so practically I think one can exclude software failure meanwhile

 

And you have just acknowledged that development of reiser3 is effectively frozen; so how can you be so sure there are zero bugs in it?

 *Quote:*   

>  *Quote:*   Please get this straight: it is exactly the same hardware. And please don't accuse me of NIH 
> 
> I am sure that this happened to you as you described. But it happened to you once in 6 years. Once within 6 years for whatever reason some block was written wrong (or to the wrong place) in such a way that your system lost consistency. It is not so unlikely that this happens to one out of several thousand persons after a long while and could happen with every filesystem. And it is not so surprising that it does not happen immediately again to the same person. I am rather sure that if you reformatted a new reiserfs instead of ext3, you would not have the problem either.

 

No, the point is I only noticed it when I started developing update, ie the only time one of my code files has been a symlink (since I run it from /sbin/update.) And it didn't happen once, it happened again and again. I am more than willing to accept that my CPU might overheat once in a while, but as I said it never happens with ext3, no matter how long I have been working, and I have more faith in ext3 as a well-maintained fs, which has taken years to get the acceptance of ext2 users. Maybe that's because it is more reliable on flaky hardware? The point is, it is more reliable, and gives the power-off benefit we discussed.

And come on, guys, how many of us are running on machines we built ourselves; can you really be so certain the hardware isn't flaky? And what about BIOS problems we see all the time, do you really think OEMs are immune?

Having said that, it'd be great to debug reiser3 at some point, if I get time and others are motivated.

 *Quote:*   

>  *Quote:*   Even weirder is your assertion that since one bug was fixed in Reiser many years ago, there are no others. 
> 
> I only heard about this one bug in reiserfs which could cause such a bad data loss (and actually many people who speak about bad experience with reiserfs were a victim of this bug). Of course, this does not mean that there cannot be another bug, but at least in the discussions which I had observed so far, it seems that nobody has experienced anything which was statistically so extraordinary or reproducible. So it seems likely that there is not another such serious bug (of course, there were some smaller bugs which had been unobserved for quite a while, but such "cosmetic" bugs are different issue.)

 

Possibly, but it's equally likely no-one ever noticed, just like I never did. After all, people who run reiser tend to be those who have gone with it against advice, so they are more likely to be, say, running unstable software. I just used to be so glad I wasn't on Windoze, I didn't care if there were glitches  :Wink: 

Thanks for the discussion so far. I've always liked the forums more than the dev m-l, and this conversation illustrates why.

----------

## mv

 *steveL wrote:*   

>  *mv wrote:*   No journalling can save you from problems during power failure. 
> 
> Er sorry, I disagree. Journalling can and should provide this.

 

One of the typical effects of power failure is that RAM looses consistency first. If your machine writes some data to the journal while ram looses consistency (but still writes that flag that the journal was updated correctly) you will have rubbish once the journal was replayed. (A checksum over the journal might help in such a case but AFAIK no filesystem does this). But things can even be worse, since the sector number (i.e. to where the data should be written) might have lost consistency, so it might happen that the filesystem actually writes rubbish somewhere on the disk.

 *Quote:*   

> One thing that springs to mind is if the sector can never be restored by reiser (since it only exists in one place) and might be recoverable by ext3, I am already happier with ext3.

 

Yes, in this situation you have more luck with ext3. On the other hand, if the power loss happens during writing the journal (as in the above example) you have more luck with reiserfs, because ext3 will write the full block of rubbish data from the journal when replaying it while reiserfs will only write the wrong changes and not destroy the existing data in this block. It all depends on where the power loss happens and what it does...

 *Quote:*   

>  But like I say, I am not concerned about the power loss situation, since I have never had a problem with them since I started using journalling.

 

Are you sure that you never had any power loss (under which I summarize also things like switching off the running machine or similar situations)? If you had even only one, how can you be sure that your problems do not originally come from this one? (The bad things about filesystem inconsistencies is that they can be like a growing illness: They usually do not affect something visible immediately)

 *Quote:*   

> No, the point is I only noticed it when I started developing update, ie the only time one of my code files has been a symlink (since I run it from /sbin/update.)

 

My whole gentoo configuration is made of symlinks (on reiserfs to reiserfs), I never had any problems. And I would guess that the part of the code which treats symlinks is the same for all filesystems (i.e. I would expect that only the part which reads/writes the symlink itself is filesystem specific), so I doubt very much that the symlink itself was the cause of your problems. I would expect that it is only by accident that your problems occurred in this moment.

 *Quote:*   

> And it didn't happen once, it happened again and again.

 

Did It happen again on a freshly formatted reiserfs? Or "only" on an perhaps already corrupted filesystem? (In the latter case, I should ask you about the reiserfschk version you used - it might be a bug in it which did not find this corruption.)

 *Quote:*   

> After all, people who run reiser tend to be those who have gone with it against advice, so they are more likely to be, say, running unstable software.

 

reiserfs was the default filesystem for SuSE for many years (which at least in Germany was the most popular distribution in that time), so many people were even advised to run it.

----------

## mv

Putting /var/db into a squash+aufs mount revealed a serious bug of either python or aufs:

The python code 

```
#!/usr/bin/python

import sys, os

os.rename("/var/db/pkg/app-emacs", "/var/db/pkg/app-emacs.bak")
```

 leads to the strange error message 

```
OSError: [Errno 18] Invalid cross-device link
```

 if /var/db is mounted using suashfs+aufs.

This means that the automatic update of portage (renaming of installed packages) will fail with this error.

It is really a python problem: The corresponding mv-command works without any errors. Moreover, after that "manual" mv-command (and renaming back the directory), the python script executes without any error: The reason is obviously that /var/db/pkg/app-emacs then exists in the overlayed (writable) directory branch and not only in the sqfs-branch.

Does somebody have an idea for a workaround (so that one can keep /var/db aufs-mounted but use portage anyway)?

----------

## Diredicker

Is this compress function also possible with the portage replacement paludis?

----------

## mv

 *mv wrote:*   

> Putting /var/db into a squash+aufs mount revealed a serious bug of either python or aufs

 

Just for the records: This was fixed with portage-2.1.3_rc6. Thanks to Zac Medico for fixing the problem with the os.rename usage.

 *Diredicker wrote:*   

> Is this compress function also possible with the portage replacement paludis?

 

I am not sure what this question means. The squashfs+unionfs/aufs approach is not related at all with portage. It should work with any program/directory you want (perhaps with some unexpected restrictions as demonstrated by the above mentioned bug).

----------

## Corona688

 *steveL wrote:*   

> I am a recent convert to ext3 after 6 years exclusively on reiser. I saw this: http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc but didn't quite believe it til I was working on update which I run via symlink. I was getting all kinds of random garbage in the script and weird bash errors. Finally I switched to ext3, and it's never happened again.

  People don't always have a choice, though.  if you want to install Gentoo on a 2GB hard drive, you'll want to install with ReiserFS;  no other filesystem I know of can cram all of portage and a couple kernel source trees into that amount of space.

Closer to topic, I remember doing something like this to save space on a minimal(and I mean *minimal*) Gentoo system;  Pentium-133 with 1GB hard drive space...  except I didn't cram my portage tree, I crammed /usr/share/docs/, /usr/share/man, etc.  Here's a tip for that;  uncompressed files cram better than compressed ones, so decompress the manpages before making a cramfs out of them.  (Be sure to fix symlinks after.)  Better performance too, since your computer doesn't have to decompress any particular manpage twice.

----------

## steveL

 *Corona688 wrote:*   

> People don't always have a choice, though.  if you want to install Gentoo on a 2GB hard drive, you'll want to install with ReiserFS;  no other filesystem I know of can cram all of portage and a couple kernel source trees into that amount of space.

 

Er if I only had a 2GB hdd on the box, i wouldn't even put portage on it, i'd mount it over NFS and most likely use a binhost since i am guessing it would be an old, slow, machine with not much RAM.

Besides, I still use reiser for my portage tree, since all the files get checked against sums and are easily restored with a --sync. I just wouldn't trust my root and /home to it any more. But as ever, it's down to what you choose; it's your box  :Wink: 

----------

## mv

An Update of the initscripts (including squashfs + aufs/unionfs/funionfs mounting/saving) is available here.

The main advantage of the new version is that you can use a temporary directory for changes if e.g. you need a squashfs-mounted directory temporarily writable but do never want to save the changes permanently. A sample application of this case is e.g. when you install texlive with the texlive ebuild, making use of the cdinstall useflag, and you want to mount /usr/share/texmf-dist and /usr/sahre/texmf-doc from a squashfs: In some cases you might want e.g. to compile (temporarily) some doc or example from these directories without copying the whole directory.

Note that some environment variables have changed in the new release, most notably: DIR_TMP was removed.

Instead a general mechanism was introduced how you can specify masks for temporary files/directories.

In particular, it is now easy to create the temporary squashfs-file on the same partition as the original squashfs-file so that moving is very fast (and it is now no longer reasonable to use a ramdisk unless you have extraordinary much ram).

Another minor point is that the bug about the mysterious loss of loop devices discussed earlier in this thread is now probably avoided by no longer attempting lazy unmounts if the normal unmount fails. The disadvantage of this solution is now that the script fails to stop if the directory is still in use. You can restore the previous behavior by setting a variable.

----------

## synss

 *mv wrote:*   

> An Update of the initscripts (including squashfs + aufs/unionfs/funionfs mounting/saving) is available here.

 

I am just back and I use your script now, and it is very nice!!  :Smile: 

----------

## synss

baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms. I had a look at your (mv's) script but it is a bit over my head to patch it so that it does not fail on dash... If any one is able and has time, that would be a nice thing to do.

----------

## mv

New POSIX compatible release 5.2

(For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).

 *synss wrote:*   

> baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms.

 

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

----------

## steveL

 *mv wrote:*   

> I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

 

Woot! Well done mv :D

----------

## synss

 *mv wrote:*   

> New POSIX compatible release 5.2
> 
> (For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).
> 
>  *synss wrote:*   baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms. 
> ...

 

That was fast, thanks!

EDIT: just to say I have just tried it and it works well with dash. (But my system does not really boot faster.)

----------

## mv

There is a new version of the initscript available at the initscripts. (The most important bugfix is that an unmount error is now not mistakenly ignored.)

Moreover, also a simple script can be found there which treats several of the corresponding initscripts simultaneously, if you want to recompress/start/stop/view the status of them manually.

Finally, there is also a patch for squashfs-tools-3.2_p2 which redirects the progress bar of mksquashfs to stderr. The advantage is that the init-script will then display the progress bar without printing additional confusing information (but you can also have the latter without the patch with the new version of the script).

If you use the script note that squashfs-tools-3.3 is meanwhile in the portage tree but will create a format which seems to be incompatible with the current gentoo kernel patchset, so you should not upgrade to squashfs-tools-3.3 in the moment...

----------

## mv

There is again a new version of the script available. Now it is possible to ignore certain files/dirs resp. ignore "touches" of these files/dirs (in the main directory). This is important if you squash a tex directory and want to avoid unnecessary re-squashing just because one of the system tools updated "ls-R" without actually making any changes to that file.

----------

## Darknight

Thanks for this nice script.

----------

## 188562

synss Thank you very much for the idea and its embodiment!

My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines

#mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR

mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

After that, everything worked! The truth is now at the time of each file portage.sqfs re-rebooting again this takes some time.

----------

## synss

 *init_6 wrote:*   

> synss Thank you very much for the idea and its embodiment!
> 
> My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines
> 
> #mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
> ...

 

Thank you! now I am not sure I understand your last sentence correctly: you do not need to reboot to update the image, you only need to restart the initscript, which can be done in a monthly cron job, too, BTW.

----------

## 188562

Every time I do a "reboot" or "halt" or "poweroff" script re-creates file portage.sqfs so if I update portage "emerge --sync"

Most likely this feature aufs but I'm not sure.

----------

## IvanMajhen

Add this 

```
stop() {

   ebegin "Updating portage tree"

   rm -rf /dev/shm/.portage-rw/.wh..wh.plink

   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"

   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

   then
```

----------

## 188562

So here is the corrected version of the script for squashfs & aufs. All except out in Section stop working as it should work. 

```
#!/sbin/runscript

source /etc/make.globals

SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"

SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"

SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"

DEF_RW="/dev/shm/.portage-rw"

depend() {

    need localmount

}

start() {

    ebegin "Mounting read-only squashfs image"

    mount -rt squashfs -o loop $SQFS_CUR $PORTDIR

    retval=$?

    eend $retval

    [ $retval -ne 0 ] && return $retval

    ebegin "Mounting read-write with aufs"

    if [ ! $PORTAGE_RW ]

   then einfo " mounted in tmpfs (RAM)"

   PORTAGE_RW="${DEF_RW}"

    fi

    [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW

    chmod 0750 $PORTAGE_RW

    chown portage:portage $PORTAGE_RW

    mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

    eend $?

}

stop() {

#   ebegin "Updating portage tree"

#   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"

#   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

#   then

#      einfo "  Syncing the tree"

#      mv -f $SQFS_NEW $SQFS_OLD

#      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null

#      retval=$?

#      ln -sf $SQFS_NEW $SQFS_CUR

#   else

#      einfo "  Nothing to do"

#      retval=0

#   fi

#   eend $retval

    ebegin "Unmounting the tree"

    umount -t aufs $PORTDIR

    umount -t squashfs $PORTDIR

    rm -rf $PORTAGE_RW   

    eend 0

}
```

Here is this condition (if) why it is always true

 *Quote:*   

> #   ebegin "Updating portage tree"
> 
> #   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
> 
> #   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

 

----------

## IvanMajhen

This is wrong. Sqfs wont be updated when synced. 

This is how my stop function looks:

```
stop() {

   ebegin "Updating portage tree"

   rm -rf /dev/shm/.portage-rw/.wh..wh.plink

   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"

   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

   then

      einfo "  Syncing the tree"

      mv -f $SQFS_NEW $SQFS_OLD

      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null

      retval=$?

      ln -sf $SQFS_NEW $SQFS_CUR

   else

      einfo "  Nothing to do"

      retval=0

   fi

   eend $retval

   ebegin "Unmounting the tree"

   umount -t aufs  $PORTDIR

   umount -t squashfs $PORTDIR

   rm -rf $PORTAGE_RW

   eend 0

}

```

----------

## NaiL

I have using the MV's script for some time, (currently the last version).

I found a problem stopping the init script.

 *starting the script wrote:*   

>  /etc/init.d/squash_portage start
> 
>  * Mounting /var/tmp/portage-current.sqfs as /usr/portage ...             [ ok ]

 

 *mount command after the start wrote:*   

> mount
> 
> /dev/sda3 on / type reiserfs (rw,noatime,notail)
> 
> /proc on /proc type proc (rw)
> ...

 

 *stopping the script wrote:*   

> # /etc/init.d/squash_portage stop 
> 
>  * Unmounting /usr/portage ...
> 
> umount: /var/tmp/portage-current.sqfs: not mounted
> ...

 

 *mount command after the stop wrote:*   

> mount
> 
> /dev/sda3 on / type reiserfs (rw,noatime,notail)
> 
> /proc on /proc type proc (rw)
> ...

 

 *the config file wrote:*   

> # cat /etc/conf.d/squash_portage 
> 
> # /etc/conf.d/squash_portage
> 
> DIRECTORY="/usr/portage"
> ...

 

Maybe is it a problem of configuration?

----------

## Darknight

I have the same problem. As far as I know it used to work perfectly until some days ago but I may have missed that error for some time. Unfortunately I can't pinpoint any update, nor change the script to make it work with the new stuff...

EDIT: It has to do with temporary filenames. Set DIR_SQUASH as a workaround.

----------

## mv

 *Darknight wrote:*   

> I have the same problem

 

Sorry, somehow I had missed NaiL's posting.

Anyway, I cannot reproduce the problem on my machine if I use exactly NaiL's config (only with a different DIRECTORY for testing purposes).

Do you use the most current version (7.4) of the script? (The version is in a comment near the beginning).

However, as remarked in the documentation, using the temporary filename feature for DIR_SQUASH is always hackish: It depends on the undocumented /etc/mtab format and thus it might depend on the mount command (i.e. on the util-linux version) whether this will work. During my test just now I had util-linux-2.14.1 installed: The critical part is whether in line 686-688 the correct filename can be read from /etc/mtab by the sed-expression.

----------

## Darknight

I have the same util-linux version but somehow sed fails (I had nailed it, but I have not bash skillz to modify the script).

Anyway re-reading through the documentation gave me a solution that is good enough for me.

----------

## NaiL

Hi,

I'm using the 7.4 version of the script and:

sys-apps/util-linux-2.14.1

sys-fs/squashfs-tools-3.3

sys-fs/aufs-20081208-r1

sys-kernel/gentoo-sources-2.6.27-r8

Do you need any other information?

----------

## mv

NaiL, apparently, the problem is your strange output of

 *mount wrote:*   

> /dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)

 

This shouldn't be so. It should be instead something like

 *mount wrote:*   

> /var/tmp/portage-current.sqfs on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro,loop=/dev/loop0)

 

I cannot understand why your output is so. What happens after you do manually

```
mkdir /some_directory

mount -t squashfs -o loop,ro /var/tmp/portage-current.sqfs /some_directory
```

If this works as expected, the output of mount should then contain

 *mount wrote:*   

> /var/tmp/portage-current.sqfs on /some_directory type squashfs (ro,loop=/dev/loop*)

 

and afterwards you should be able to umount by

```
umount /var/tmp/portage-current.sqfs
```

(of course, afterwards, you can remove /some_directory).

Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?

----------

## NaiL

 *Quote:*   

> I cannot understand why your output is so. What happens after you do manually
> 
> ```
> mkdir /some_directory
> 
> ...

 

The result was:

 *Quote:*   

> /dev/loop1 on /some_directory type squashfs (ro)

 

 *Quote:*   

> Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?

 

Affter the upgrade i get the same output.

----------

## mv

NaiL, I have no idea what is responsible on your system that the same mount-command (with same versions of mount and squashfs than here) works differently on your system. Do you have perhaps some of these terrible hal things activated or some special udev rules which could cause this behavior? Here are my useflags for util-linux: 

```
crypt loop-aes nls slang unicode -old-linux -selinux -uclibc
```

----------

## NaiL

Tnx! I finally found the problem.

I was some how related to "loop-aes" USE flag. 

I had this USE flag disabled. By some reason it has to be activated, may be it will be good to document that some where.

I also noticed that.   :Shocked: 

 *Quote:*   

>   * The loop-aes code has been split out of USE=crypt and into USE=loop-aes.
> 
>  * If you need support for it, make sure to update your USE accordingly.

 

----------

## mv

In the new release of the script the temporary file name feature was removed for DIR_SQUASH. Instead, now a sane default is chosen for DIR_SQUASH in /var/run. Indeed, since even with the temporary file name feature the thing could only work by using a non-temporary file (in this case /etc/mtab), it is probably more reasonable to force directly to use a non-temporary directory. In this way the implementation is independent of undocumented behavior of the mount command.

----------

## jowr

Where can I get a squashfs utils that makes a SquashFS 4 file system? Apparently 2.6.29 now only has v4, and there are no [known to me..] userland utils for making v4 file systems.

----------

## Dont Panic

I've just upgraded to a 2.6.29 kernel, and I am now getting errors like this when trying to mount the squashfs:

```
SQUASHFS error: Major/Minor mismatch, older Squashfs 3.1 filesystems are unsupported
```

I may not be understanding correctly, but the information I'm reading on the Internet indicates that backward compatibility has been broken for squashfs as part of the process for bringing squashfs into the mainline kernel.

Here's a post from the author of squashfs:

http://lkml.indiana.edu/hypermail/linux/kernel/0901.1/02601.html

I'm not finding much information about this on squashfs' homepage on sourceforge

And I also see the entire set of sys-fs/squashfs-tools ebuilds have been hard-masked (as of the time of this post).

This looks like a bumpy ride for my squashfs portage directory.

----------

## Darknight

When I tried aufs didn't compile against 2.6.29, therefore I'm back to 2.6.27 for now.

----------

## Dont Panic

I'm using a kernel from Sabayon.

We're using an aufs patch for 2.6.29 that can be found here:

http://svn.sabayonlinux.org/filedetails.php?repname=Sabayon+Linux+Overlay&path=%2Fsys-kernel%2Flinux-sabayon%2Ffiles%2F2.6.29%2Flinux-sabayon-2.6.29-aufs.patch.bz2

Unfortunately, based on what I'm seeing so far, you don't need to be in a hurry to get a 2.6.29 kernel going.

----------

## mv

 *Dont Panic wrote:*   

> I've just upgraded to a 2.6.29 kernel

  When you use the 2.6.29 kernel you must use >=sys-fs/squashfs-tools-4.0_pre20090324: Only this generates the format which can be used by that kernel. Unfortunately, its format cannot be used with earlier kernel. So upgrading an existing sqfs-file is a bit cumbersome: You have to boot with an old kernel to mount the old sqfs file, then upgrade squashfs-tools and then make sure to re-squash the directory before you boot with the 2.6.29 kernel. An alternative way (if you cannot switch back to an older kernel temporarily) is to install an earlier version of squashfs-tools and use "unsquashfs" to decompress the directory manually to some free space on your harddisk; then upgrade squashfs-tools and compress the directory manually.

 *Quote:*   

> And I also see the entire set of sys-fs/squashfs-tools ebuilds have been hard-masked (as of the time of this post).

  I cannot confirm this. As far as I know, squashfs-tools was never hard-masked, and I did not hear of plans to do this.

----------

## Dont Panic

 *mv wrote:*   

>  *Dont Panic wrote:*   And I also see the entire set of sys-fs/squashfs-tools ebuilds have been hard-masked (as of the time of this post).  I cannot confirm this. As far as I know, squashfs-tools was never hard-masked, and I did not hear of plans to do this.

 

Oops, I just realized what I was looking at.  Eix was showing all the sys-fs/squashfs-tools in Red, which I interpreted as being hard-masked.  But what eix was really telling me was that it couldn't find my /usr/portage/*, which never mounted because of the kernel upgrade.   :Embarassed: 

Good Times!   :Very Happy: 

----------

## mv

A rather current live ebuild for aufs (together with ebuilds for some other union file systems) can now be found on the same place as the init-script: unionfs.tar.gz

Please be aware that if you use other ebuilds you should remove /sbin/mount.aufs and /sbin/umount.aufs for security reasons (at least, they seem to be subject to insecure temporary file creation). New versions of the init-script pass the options -i to mount/umount to avoid executing these scripts even if they are installed.

----------

## R.Aven

 *Dont Panic wrote:*   

> I'm using a kernel from Sabayon.
> 
> We're using an aufs patch for 2.6.29 that can be found here:
> 
> http://svn.sabayonlinux.org/filedetails.php?repname=Sabayon+Linux+Overlay&path=%2Fsys-kernel%2Flinux-sabayon%2Ffiles%2F2.6.29%2Flinux-sabayon-2.6.29-aufs.patch.bz2
> ...

 

Does anybody have an ebuild which actually includes this patch or could tell me how to apply it to the work-dir of aufs?

I just tried several -p* options, but it won't apply the patch.

----------

## Dont Panic

Have you tried building aufs/unionfs/funionfs as a module using the ebuilds in unionfs.tar.gz mentioned directly above in mv's post?

----------

## R.Aven

 *Dont Panic wrote:*   

> Have you tried building aufs/unionfs/funionfs as a module using the ebuilds in unionfs.tar.gz mentioned directly above in mv's post?

 

Yes, of course. But the aufs-999999.ebuild uses the current aufs dev tree, which doesn't support the 2.6.29 kernel branch, afaik not even officially the 2.6.28 version. So it won't compile without the applied patch mentioned above as dentry_open(...) seems to be changed within 2.6.29.

```
laptop /var/portage/overlay/personal/sys-fs/aufs $ emerge -av aufs

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild     U ] sys-fs/aufs-99999999 [20081208-r1] USE="fuse -debug -hinotify -nfsexport -robr -utils% (-sec_perm%)" 0 kB [1=>2]

Total: 1 package (1 upgrade), Size of downloads: 0 kB

Portage tree and overlays:

 [0] /var/portage/repository

 [1] /var/portage/overlay/sunrise

 [2] /var/portage/overlay/personal

Would you like to merge these packages? [Yes/No] 

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) sys-fs/aufs-99999999 from gentoo-overlay

 * checking ebuild checksums ;-) ...                                     [ ok ]

 * checking auxfile checksums ;-) ...                                    [ ok ]

 * checking miscfile checksums ;-) ...                                   [ ok ]

 * Determining the location of the kernel source code

 * Found kernel source directory:

 *     /usr/src/linux

 * Found kernel object directory:

 *     /lib/modules/2.6.29/build

 * Found sources for kernel version:

 *     2.6.29

>>> Unpacking source...

 * Fetching CVS module aufs into /var/portage/distfiles/cvs-src ...

 * Running  cvs -q -f -z1 -d ":pserver:anonymous:@aufs.cvs.sourceforge.net:/cvsroot/aufs" login

Logging in to :pserver:anonymous@aufs.cvs.sourceforge.net:2401/cvsroot/aufs

 * Running  cvs -q -f -z1 -d ":pserver:anonymous@aufs.cvs.sourceforge.net:/cvsroot/aufs" update -dP aufs

 * Copying aufs from /var/portage/distfiles/cvs-src ...

 * CVS module aufs is now in /var/tmp/portage/sys-fs/aufs-99999999/work

>>> Source unpacked in /var/tmp/portage/sys-fs/aufs-99999999/work

>>> Preparing source in /var/tmp/portage/sys-fs/aufs-99999999/work/aufs ...

 * patching local.mk

 * disabling sysaufs

 * setting workaround_fuse

 * unsetting debug

>>> Source prepared.

>>> Configuring source in /var/tmp/portage/sys-fs/aufs-99999999/work/aufs ...

>>> Source configured.

>>> Compiling source in /var/tmp/portage/sys-fs/aufs-99999999/work/aufs ...

make -j3 KDIR=/usr/src/linux SUBLEVEL=29 -f local.mk 

fs/aufs25

make CONFIG_AUFS=m AUFS_EXTRA_CFLAGS="-I/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/include -DCONFIG_AUFS_BRANCH_MAX_127 -DCONFIG_AUFS_RR_SQUASHFS -DCONFIG_AUFS_BR_XFS -DCONFIG_AUFS_WORKAROUND_FUSE -DCONFIG_AUFS_GETATTR -DCONFIG_AUFS_LOCAL -DCONFIG_AUFS_MODULE -UCONFIG_AUFS -DLKTRHidePrePath=\\\"/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25\\\"" -C /usr/src/linux M=/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25 modules

make CONFIG_AUFS=m AUFS_EXTRA_CFLAGS="-I/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/include -DCONFIG_AUFS_BRANCH_MAX_127 -DCONFIG_AUFS_RR_SQUASHFS -DCONFIG_AUFS_BR_XFS -DCONFIG_AUFS_WORKAROUND_FUSE -DCONFIG_AUFS_GETATTR -DCONFIG_AUFS_LOCAL -DCONFIG_AUFS_MODULE -UCONFIG_AUFS -DLKTRHidePrePath=\\\"/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25\\\"" -C util

make[1]: Entering directory `/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/util'

cc -O2 -march=prescott -O2 -pipe -msse3 -fomit-frame-pointer -O2 -Wall -I/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/include  -DCONFIG_AUFS_BRANCH_MAX_127 -DCONFIG_AUFS_RR_SQUASHFS -DCONFIG_AUFS_BR_XFS -DCONFIG_AUFS_WORKAROUND_FUSE -DCONFIG_AUFS_GETATTR -DCONFIG_AUFS_LOCAL -DCONFIG_AUFS_MODULE -UCONFIG_AUFS -DLKTRHidePrePath=\"/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25\"   -Wl,-O1  c2tmac.c   -o c2tmac

cc -O2 -march=prescott -O2 -pipe -msse3 -fomit-frame-pointer -O2 -Wall -I/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/include  -DCONFIG_AUFS_BRANCH_MAX_127 -DCONFIG_AUFS_RR_SQUASHFS -DCONFIG_AUFS_BR_XFS -DCONFIG_AUFS_WORKAROUND_FUSE -DCONFIG_AUFS_GETATTR -DCONFIG_AUFS_LOCAL -DCONFIG_AUFS_MODULE -UCONFIG_AUFS -DLKTRHidePrePath=\"/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25\"   -Wl,-O1 -s  aulchown.c   -o aulchown

cc -O2 -march=prescott -O2 -pipe -msse3 -fomit-frame-pointer -O2 -Wall -I/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/include  -DCONFIG_AUFS_BRANCH_MAX_127 -DCONFIG_AUFS_RR_SQUASHFS -DCONFIG_AUFS_BR_XFS -DCONFIG_AUFS_WORKAROUND_FUSE -DCONFIG_AUFS_GETATTR -DCONFIG_AUFS_LOCAL -DCONFIG_AUFS_MODULE -UCONFIG_AUFS -DLKTRHidePrePath=\"/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25\"   -Wl,-O1  c2sh.c   -o c2sh

rm -f aufs.5

./c2tmac > aufs.5

awk '{ \

      gsub(/\140[^\047]*\047/, "\\[oq]&\\[cq]"); \

      gsub(/\\\[oq\]\140/, "\\[oq]"); \

      gsub(/\047\\\[cq\]/, "\\[cq]"); \

      gsub(/\047/, "\\[aq]"); \

      print; \

   }' aufs.in.5 >> aufs.5

rm -f etc_default_aufs

sed -e '/Id: /,$d' aufs.shlib > etc_default_aufs

chmod a-w aufs.5

echo '# aufs variables for shell scripts' >> etc_default_aufs

./c2sh >> etc_default_aufs

sed -e '0,/Id: /d' aufs.shlib >> etc_default_aufs

chmod a-w etc_default_aufs

test -x ./mount.aufs || chmod a+x ./mount.aufs

rm c2tmac c2sh

make[1]: Leaving directory `/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/util'

test -e aufs.5 || ln -s util/aufs.5 aufs.5

test -e aufind.sh || ln -s util/aufind.sh aufind.sh

test -e mount.aufs || ln -s util/mount.aufs mount.aufs

test -e auplink || ln -s util/auplink auplink

test -e aulchown || ln -s util/aulchown aulchown

test -e umount.aufs || ln -s util/umount.aufs umount.aufs

test -e etc_default_aufs || ln -s util/etc_default_aufs etc_default_aufs

test -e auchk || ln -s util/auchk auchk

make[1]: Entering directory `/usr/src/linux-2.6.29'

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/module.o

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/super.o

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/sbinfo.o

/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/super.c: In Funktion »au_show_wbr_create«:

/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/super.c:153: Warnung: Format ist kein Zeichenkettenliteral, und keine Formatargumente

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/branch.o

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/xino.o

  CC [M]  /var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/sysaufs.o

/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/xino.c: In Funktion »au_xino_create2«:

/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/xino.c:624: Fehler: Zu wenige Argumente für Funktion »dentry_open«

make[2]: *** [/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25/xino.o] Fehler 1

make[2]: *** Warte auf noch nicht beendete Prozesse...

make[1]: *** [_module_/var/tmp/portage/sys-fs/aufs-99999999/work/aufs/fs/aufs25] Fehler 2

make[1]: Leaving directory `/usr/src/linux-2.6.29'

make: *** [fs/aufs25/aufs.ko] Fehler 2

 * 

 * ERROR: sys-fs/aufs-99999999 failed.

 * Call stack:

 *               ebuild.sh, line   48:  Called src_compile

 *             environment, line 3535:  Called die

 * The specific snippet of code:

 *       emake KDIR=${KV_DIR} SUBLEVEL=${KV_PATCH} -f local.mk || die "emake failed"

 *  The die message:

 *   emake failed

 * 

 * If you need support, post the topmost build error, and the call stack if relevant.

 * A complete build log is located at '/var/tmp/portage/sys-fs/aufs-99999999/temp/build.log'.

 * The ebuild environment file is located at '/var/tmp/portage/sys-fs/aufs-99999999/temp/environment'.

 * This ebuild is from an overlay named 'gentoo-overlay': '/var/portage/overlay/personal/'

 * 

>>> Failed to emerge sys-fs/aufs-99999999, Log file:

>>>  '/var/tmp/portage/sys-fs/aufs-99999999/temp/build.log'

 * Messages for package sys-fs/aufs-99999999:

 * 

 * ERROR: sys-fs/aufs-99999999 failed.

 * Call stack:

 *               ebuild.sh, line   48:  Called src_compile

 *             environment, line 3535:  Called die

 * The specific snippet of code:

 *       emake KDIR=${KV_DIR} SUBLEVEL=${KV_PATCH} -f local.mk || die "emake failed"

 *  The die message:

 *   emake failed

 * 

 * If you need support, post the topmost build error, and the call stack if relevant.

 * A complete build log is located at '/var/tmp/portage/sys-fs/aufs-99999999/temp/build.log'.

 * The ebuild environment file is located at '/var/tmp/portage/sys-fs/aufs-99999999/temp/environment'.

 * This ebuild is from an overlay named 'gentoo-overlay': '/var/portage/overlay/personal/'

 * 

 * 

 * The following package has failed to build or install:

 * 

 *    ('ebuild', '/', 'sys-fs/aufs-99999999', 'merge')

 * 

```

----------

## Dont Panic

The Hitchhiker Sources also have aufs.

If you don't want to use Hitchhiker sources or Sabayon sources, you may want to still download one or the other, and just copy their /fs/aufs/ directory, and overwrite whatever you have for your /fs/aufs/ directory.

----------

## mv

 *R.Aven wrote:*   

> But the aufs-999999.ebuild uses the current aufs dev tree, which doesn't support the 2.6.29 kernel branch, afaik not even officially the 2.6.28 version.

 

I will not try 2.6.29 until a corresponding hardened-sources is out, but hardened-sources-2.6.28 works smoothly. However, shouldn't one better use aufs2-standalone for 2.6.29? So far, I wasn't able to compile it here: It seems that the build system has troubles with different KERNEL_DIR and KBUILD_OUTPUT directories. Did anybody else have success with it?

----------

## R.Aven

 *mv wrote:*   

>  *R.Aven wrote:*   But the aufs-999999.ebuild uses the current aufs dev tree, which doesn't support the 2.6.29 kernel branch, afaik not even officially the 2.6.28 version. 
> 
> I will not try 2.6.29 until a corresponding hardened-sources is out, but hardened-sources-2.6.28 works smoothly. However, shouldn't one better use aufs2-standalone for 2.6.29? So far, I wasn't able to compile it here: It seems that the build system has troubles with different KERNEL_DIR and KBUILD_OUTPUT directories. Did anybody else have success with it?

 

I just downloaded the patch and the corresponding fs and include files from aufs2-standalone-29, applied them manually and now it seems to compile correctly. Thanks for your help.

----------

## mv

There is now also an experimental ebuild for aufs2 on unionfs.tar.gz.

However, unless I completely misunderstood something, at least the aufs2-28 version (for 2.6.28) is completely broken: In fact, the module is not even compiled, because the symbols are not passed to the sub-makes by the buildsystem. The experimental ebuild patches the Makefile to fix this problem, but even then it dies soon with a compile error (tested against the patched hardened-sources-2.6.28-r7).

 *R.Aven wrote:*   

> just downloaded the patch and the corresponding fs and include files from aufs2-standalone-29, applied them manually and now it seems to compile correctly. Thanks for your help.

 

Does this mean you managed to compile aufs2-standalone-29 against the aufs2-patched kernel, or did you compile aufs1 against this kernel? If you did the former: how? Does the aufs2 ebuild work for you? (Is perhaps even the my_makefile_patch in the ebuild unnecessary for aufs2-29?)

----------

## R.Aven

 *mv wrote:*   

> There is now also an experimental ebuild for aufs2 on unionfs.tar.gz.
> 
> However, unless I completely misunderstood something, at least the aufs2-28 version (for 2.6.2 is completely broken: In fact, the module is not even compiled, because the symbols are not passed to the sub-makes by the buildsystem. The experimental ebuild patches the Makefile to fix this problem, but even then it dies soon with a compile error (tested against the patched hardened-sources-2.6.28-r7).
> 
>  *R.Aven wrote:*   just downloaded the patch and the corresponding fs and include files from aufs2-standalone-29, applied them manually and now it seems to compile correctly. Thanks for your help. 
> ...

 

I've done nothing with ebuild.The aufs version from the sunrise-overlay is currently installed, but won't compile on a 2.6.29 kernel. Therefore I used the patch mentioned above and applied them manually to my 2.6.29.1 kernel - which works here on my system. The utils installed with the aufs-2008* ebuild are also working with the aufs module I build using my patched kernel - at least I were able to mount my compressed portage tree with squashfs+aufs.

----------

## mv

 *R.Aven wrote:*   

> I've done nothing with ebuild.The aufs version from the sunrise-overlay is currently installed, but won't compile on a 2.6.29 kernel. Therefore I used the patch mentioned above and applied them manually to my 2.6.29.1 kernel - which works here on my system.

 

But the patch alone (at least that from aufs2-standalone) will not build the actual module for you. My question is how you managed to generate /lib/modules/2.6.29-*/misc/aufs.ko

 *Quote:*   

> The utils installed with the aufs-2008* ebuild are also working with the aufs module I build using my patched kernel

 

Did I understand correctly that you built the aufs.ko module still with the aufs-2008* ebuild (only after applying the aufs2-standalone patch to the kernel)? This is somewhat strange, because it means that you mix aufs version: You use the aufs2-kernel patch (meant for the aufs2 module) with the aufs1 module. So it is somewhat just accidental that this works.

----------

## R.Aven

The patch adds aufs functionality to the kernel itself. So I was able to build it - as I would build all kernel modules which are actually part of the official tree - with selecting the corresponding module via "make menuconfig", make and make modules_install. That's all I did. Now I've got the aufs.ko and can modprobe, and use it as before switching to 2.6.29.

----------

## mv

 *R.Aven wrote:*   

> The patch adds aufs functionality to the kernel itself.

 

The patch alone does not. I suppose then that besides applying the patch you also copied the other files of aufs2-standalone (except the Makefile) into your kernel tree?

Then there are of course no problems with the broken Makefile (because it is simply not copied). However, the new kernel didn't compile either with hardened-sources-2.6.28-r7 (the reason might be some of the gentoo or hardened patches, I didn't check yet).

But maybe you are right: Since - in contrast to aufs1 - the kernel patch is nontrivial anyway for aufs2, it does not make much sense to require that the user applies this patch manually and then only to install the module by an ebuild. Indeed, the new version of the aufs2 ebuild on the mentioned web page only installs all required files into the kernel source directory and asks the user to apply the patches. Also this solution is not optimal, but there is no really good solution for a module which cannot be compiled standalone but requires in addition nontrivial kernel patches.

Edit: I retried again with hardened-sources-2.6.28-r7, and now aufs2 works as it should. (Due to a mistake in using aufs with my kernel sources, I had probably patched by mistake sources which had already been patched for aufs1).

Conclusion: The new aufs2 ebuild (which is essentially the aufs2-standalone kernel patch with corresponding files in the kernel source directory) works without problems with hardened-sources-2.6.28-r7 and (I suppose) *-sources-2.6.29-*

----------

## shgadwa

I'm also trying to emerge aufs...

I'm using the Zen-Sources kernel-2.6.29-rc8-zen1 kernel. I can configure it for aufs support, and I've done that. But I am not able to emerge aufs (from sunrise overlay) or aufs2 (from ebuild that I downloaded)...

It says that its not going to be installed because of file collisions.

here is the build errors:

```
 * This package will overwrite one or more files that may belong to other

 * packages (see list below). You can use a command such as `portageq

 * owners / <filename>` to identify the installed package that owns a

 * file. If portageq reports that only one package owns a file then do

 * NOT file a bug report. A bug report is only useful if it identifies at

 * least two or more packages that are known to install the same file(s).

 * If a collision occurs and you can not explain where the file came from

 * then you should simply ignore the collision since there is not enough

 * information to determine if a real problem exists. Please do NOT file

 * a bug report at http://bugs.gentoo.org unless you report exactly which

 * two packages install the same file(s). Once again, please do NOT file

 * a bug report unless you have completely understood the above message.

 * 

 * Detected file collision(s):

 * 

 *    /usr/src/linux-2.6.29-rc8-zen1/include/linux/aufs_type.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/branch.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/f_op.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/module.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/opts.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dcsub.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vfsub.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/fstype.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/inode.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/aufs.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dcsub.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_ren.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/whout.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/loop.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dentry.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysaufs.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vfsub.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/file.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dir.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/hinotify.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_add.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/Kconfig

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/iinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysaufs.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/file.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysrq.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/Makefile

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysfs.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_del.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/super.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dir.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wbr_policy.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/spl.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/cpup.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/branch.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/debug.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/inode.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/cpup.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/ioctl.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/rwsem.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/loop.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dentry.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/finfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/super.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/xino.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/whout.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/plink.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/module.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/opts.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/magic.mk

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/debug.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sbinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wkq.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vdir.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wkq.c

 * 

 * Searching all installed packages for file collisions...

 * 

 * Press Ctrl-C to Stop

 * 

 * sys-kernel/zen-sources-2.6.29_rc8-r1

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/Kconfig

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/Makefile

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/aufs.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/branch.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/branch.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/cpup.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/cpup.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dcsub.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dcsub.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/debug.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/debug.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dentry.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dentry.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dir.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/dir.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/f_op.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/file.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/file.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/finfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/fstype.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/hinotify.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_add.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_del.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/i_op_ren.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/iinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/inode.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/inode.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/ioctl.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/loop.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/loop.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/magic.mk

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/module.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/module.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/opts.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/opts.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/plink.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/rwsem.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sbinfo.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/spl.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/super.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/super.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysaufs.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysaufs.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysfs.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/sysrq.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vdir.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vfsub.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/vfsub.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wbr_policy.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/whout.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/whout.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wkq.c

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/wkq.h

 *    /usr/src/linux-2.6.29-rc8-zen1/fs/aufs/xino.c

 *    /usr/src/linux-2.6.29-rc8-zen1/include/linux/aufs_type.h

 * 

 * Package 'sys-fs/aufs2-99999999' NOT merged due to file collisions. If

 * necessary, refer to your elog messages for the whole content of the

 * above message.

```

EDIT:

I already solved the problem... I had to add COLLISIONS_IGNORE="/usr/src/linux-2.6.29-zen1" to /etc/make.conf

Now it installed, no problems.  :Smile: 

----------

## mv

 *belikeyeshua wrote:*   

> I'm using the Zen-Sources kernel-2.6.29-rc8-zen1 kernel. I can configure it for aufs support, and I've done that. But I am not able to emerge aufs (from sunrise overlay) or aufs2 (from ebuild that I downloaded)...

 

If Zen-Sources already contain aufs, then why do you want to emerge the ebuild? The purpose of the ebuild is to give aufs support for kernels which do not contain it.

----------

## shgadwa

Oooh.... good point.

I was wondering about that. So, should I uninstall aufs2?

----------

## mv

 *belikeyeshua wrote:*   

> So, should I uninstall aufs2?

 

Probably it will do no harm, but I am not sure whether it is exactly the same as that version which is contained in zen-source.  (If you use zen-sources anyway, you should better also use the aufs contained there).

So it is probably safer if you uninstall aufs2 (sorry for my false suggestion in the other thread: I was not aware that zen-sources contain aufs).

However, be aware that uninstalling aufs2 will now remove the files you had overridden in zen-sources. To fix this you should re-emerge zen-sources afterwards.

----------

## crater2150

Hi,

I tried to set up the script for the portage tree, but when the script stops, it says: 

```
File /usr/portage/portage-sqfs.XXXXXX changed size while reading filesystem, attempting to re-read
```

, and repeats this, until it is killed manually.

My configuration:

```

DIRECTORY="/usr/portage"

DIR_CHANGE="/tmp/portage.XXXXXX"

DIR_SQUASH="/tmp/portage.sqfs"

TMP_SQFS="/tmp/portage-sqfs.XXXXXX"

FILE_SQFS="/var/portage/portage-current.sqfs"

FILE_SQFS_OLD="/var/portage/portage-old.sqfs"

```

----------

## mv

 *crater2150 wrote:*   

> 
> 
> ```
> File /usr/portage/portage-sqfs.XXXXXX changed size while reading filesystem, attempting to re-read
> ```
> ...

 

This filename is certainly not invented by the script, but nothing in the configuration you posted contains it. I suspect that you have a symbolic link pointing to that file (e.g. from one of the files in your configuration): Of course none of your files in the configuration should be inside the directory you want to squash (also not by symbolic links).

Another note: I just observed that if one uses the temporary name feature on DIR_CHANGE, the corresponding directory cannot be erased on stop. Probably some rewrite of the script is necessary to support the temporary DIR_CHANGE feature properly.

Edit: Important security note: Do not use fixed names in world-writable directories! E.g. your "/tmp/portage.sqfs" is a very bad idea! Such things can be subject to symlink-attacks!

----------

## crater2150

there weren't any symlinks to /usr/portage/portage-sqfs.XXXXXX, but removing the temporary name part from DIR_CHANGE solved the problem, thx

----------

## mv

There is now a new version of the script which should solve the problem with the temporary name feature in DIR_CHANGE. Morever, it brings also back the temporary name feature for DIR_SQUASH (in a non-hackish manner).

----------

## ocbMaurice

[url]Hi here,

I came across this problem/solution because I own an EEE 701 with 4GB SSD (and gentoo)  :Smile: 

I guess mv is the developer of the script at uni-wuerzburg?

Somehow I didn't really find this script before I wrote my own.

I think your script is more mature than mine, but is missing one or two features I implemented.

You'll find them in my response in the above linked thread.

It's mostly about the exclusion of ie. distfiles from /usr/portage.

The way I do it, you don't have to reconfig your distfiles dir ...

I didn't really yet test you script, but IMO it's lacking this feature  :Wink: 

Additionaly implemented some functions for the init script (ie. init, sync, info, todo: uninit)

Hope you like my additional ideas and will incorporate them into your script.

This should also go in the portage tree, once it's mature enough.

Sorry for crossposting, IMO the two threads should be merged!

Maurice

----------

## mv

The new version of the init-script supports a THRESHOLD and a new magic file IGNORE_THRESHOLD which can be used to override the default and thus force compression even if the THRESHOLD is not reached. The other suggestions (like init function or exclusion of directories by mount --bind) are not implemented due to security considerations: Such a delicate task as compressing and removing a whole directory tree should be done manually and manually be checked twice, since there are too many setups where this can cause problems (e.g. when root has not all permissions like in SELinux or a virtual box, in --make-shared/--make-slaved directories, automatic submounts, problems with symbolic or non-symbolic links in/out the compressed tree etc.). Without an init function, there is no proper way to implement the directory-exclusion-feature with mount --bind from within the script. As mentioned in the other thread, I would suggest a separate init-file like 

```
#!/sbin/runscript

depend () { need squash_....; }

start () { mount --bind ...; }

stop () { umount ...; }
```

 if you really need this. However, I cannot think of cases where this makes sense and where you cannot just use a symbolic link instead. For PORTDIR, it is certainly safer to put DISTDIR outside the PORTDIR tree. I would suggest the latter anyway for security reasons (independently whether some script is used) - this is meanwhile even suggested in make.conf.example. Even layman already has changed the default (although here it might make sense to include it in the compressed PORTDIR directory).

----------

## mv

The new version 9.0 of squash_dir now also supports sys-fs/unionfs-fuse which is new in the portage tree.

One advantage of this is that now you do not get any problems with new kernels if the aufs2 patch is not yet ported: The existing in-kernel fuse support suffices for unionfs-fuse (this was already the case for funionfs, but funionfs is not in the official portage tree).

Since I could not find the download address in this script immediately, here it is once more: initscripts

----------

## NaiL

It's nice to see that you are still working on it.  :Wink:  (almost 4 years)

Thanks for your work.

----------

## js08

Hi,

 I tried your script for my portage tree(squashfs + unionfs-fuse) and I'm a little bit surprised about the fact that rsync tries to delete a lot of files with the "_HIDDEN~"-ending. Each run of emerge --sync / eix-sync produces more of this file deletions.... 

 *Quote:*   

> 
> 
> receiving incremental file list
> 
> deleting .unionfs/sys-libs/timezone-data/timezone-data-2009t.ebuild_HIDDEN~
> ...

 

----------

## mv

This is a side effect of unionfs-fuse: This tool seems to mirror the directory .unionfs from the writable branch. This is not good: If somebody knows a way to avoid this, please let me know. I checked the unionfs-fuse manpage for a corresponding option but did not find one (I did not yet inspect the source or other documents).

So far, the only workaround which I know is to add --exclude=/.unionfs to the PORTAGE_RSYNC_OPTS.

----------

## mv

squash_dir now has a new name and has become a package of its own: You can find the current tarball on the same webpage as previously (but under a new name reflecting the current version). Please copy that tarball into your $DISTDIR so that the ebuild will later not download it again. In the tarball you will find in addition a corresponding ebuild and INSTALL instructions (not matter whether you want to use the ebuild or not).

Please also read the ChangeLog:

The most important change is that squash_dir will now create the squashfile and clear the original directory when the squashfile did not exist! This feature required previously in the thread is now implemented, but please be aware that it is a dangerous feature if you start with a wrong configuration (although several sanity checks are made, of course).

----------

## js08

ebuild works fine.

The only problem I have in my environment (AMD64 with 32bit-chroot using the same /usr/portage) is that unionfs-fuse doesn't work very reliable. in such a case /usr/portage is then no longer accessible and I have to restart

/etc/init.d/squash_portage. This happens one or two times per day without any error messages in syslog. It looks like nfs when the server disappears.

----------

## mv

 *js08 wrote:*   

> The only problem I have in my environment (AMD64 with 32bit-chroot using the same /usr/portage)

 

How do you do that? Using mount --bind? If this mount --bind happens before the (re)start of /etc/init.d/squash_*, the mounting in that script will of course have no effect in the chroot. I am calling 

```
mount --make-shared /
```

 (in another initscript) before calling mount --bind. This should cause the mount to propagate to the chroot although it may lead to some other problems to make / shared... On the other hand, I used the script only with aufs2: Maybe mounts by fuse are not propagated despite --make-shared, I did not try now. If this is the case, I don't know how your problem could be solved.

----------

## js08

Yes, I use mount --bind  for /usr/portage. It works fine during boot or shutdown but then after several hours something strange happens and /usr/portage is no longer accessible.

I know when it happens (due to a cronjob checking /usr/portage periodically) but at the moment I have absolutely no idea why it happens.

----------

## mv

 *js08 wrote:*   

> I know when it happens (due to a cronjob checking /usr/portage periodically)

 

Only checking? It does not mount/umount something or perhaps restart some initscripts (perhaps openrc thinks, squash_* should be restarted, for some reason)? Did you try mount --make-shared / as I suggested? (You would need to do this and the mount --bind before starting squash_*)

----------

## john.newman

hello,  I am trying to do this as per the directions and the code on the wiki (http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree).

It probably would work, except I get 

```
 * ERROR: aufs filesystem support is not available.

 * DO NOT USE EXIT IN INIT.D SCRIPTS

 * This IS a bug, please fix your broken init.d
```

well, emerge -s aufs has aufs2 masked.  no aufs...  any ideas? 

and that error message is a bit abrasive.   :Embarassed: 

I am so tired of watching portage copy when doing backups.  Can we just host it online somewhere and no more disk copy and sync?

----------

## mv

 *john.newman wrote:*   

> hello,  I am trying to do this as per the directions and the code on the wiki (http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree).

 

I suggest you try the scripts from the page of one of my previous posts; they do not use illegal things like exit on bad places (if it does not work with baselayout-1, please report here). On the same page you will also find a working aufs2 live ebuild. Do not use aufs with current kernels.

----------

## john.newman

thanks mv, i am making progress.  I have emerged your squash_dir package and am following the README here.  I actually did get portage to squash and mount, but I got an error regarding aufs. So I need to patch the kernel for aufs2 and load the module. Correct?

And the masked packages, like autoconf etc, those are OK right?   :Surprised: 

I will followup with results, and a tested install script for the impatient. (if results are 100%)

I am rolling this up through a few clean kvms to test and verify.  Kernel patching does not comfort me.  Ultimately it will live on a certain kvm, that will export the dir over NFS for every machine on the network.   as described http://en.gentoo-wiki.com/wiki/Sharing_Portage_over_NFS

(6 copies of portage is killing me.. 1 copy, squashed, and stored in one file for archiving will be a big improvement   :Cool:  .)

----------

## mv

 *john.newman wrote:*   

> I actually did get portage to squash and mount, but I got an error regarding aufs. So I need to patch the kernel for aufs2 and load the module. Correct?

 

Yes. Alternatively you can try to set ORDER="unionfs-fuse aufs" in /etc/conf.d/squash_foo in which case unionfs-fuse is attempted first. The advantage is that this does not need a kernel patch, only unionfs-fuse must be installed and the fuse module (which is in mainstream kernel) must be loaded. However, although the required space is roughly the same, it may be slower than aufs2. Moreover, there is the problem with the .funionfs directory mentioned some posts ago (you can also find the workaround there which is also described in the documentation: Search for PORTAGE_RSYNC_EXTRA_OPTS), and maybe some other problems with chroot described also in previous posts.

 *Quote:*   

> And the masked packages, like autoconf etc, those are OK right?   

 

It is just a question of time until these become stable. I am using them since months without problems. Only in the recent version some standard things can be done cleanly in a documented manner with autoconf.

----------

## john.newman

thanks mv, it is working well.  I have one 45 meg file, mounted over loopback and it is fast.  I say this should be built in as the default for /usr/portage once aufs etc gets stable.  Thanks for all the hours you spent on this, I can tell.  :Shocked: 

For reference and the impatient, here are the exact commands I used to install mv's script for /usr/portage.  I've ran through these 3 times so it should work fine for a 2.6.31 install as of 2010-01-30.  I am not saying paste these in and go, read through them and step one line at a time.  set -e and set -u are also your friends.

```
# download mv's squash-dir package, extract, and copy bz2 to distfiles to avoid 2x download

wget "http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/squash_dir-10.2.tar.bz2"

tar -xjvf squash_dir-10.2.tar.bz2 

cp squash_dir-10.2.tar.bz2 /usr/portage/distfiles/

# create a local portage overlay, you may have already done this

mkdir -p /usr/local/portage

echo 'PORTDIR_OVERLAY="/usr/local/portage"' >> /etc/make.conf

# copy the new ebuild to our local overlay

cp -a squash_dir-10.2/sys-fs /usr/local/portage/

# prepare the build

ebuild /usr/local/portage/sys-fs/squash_dir/squash_dir-10.2.ebuild manifest

# unmask some packages .. 

mkdir -p /etc/portage/package.keywords/

echo "=sys-fs/squash_dir-10.2" >> /etc/portage/package.keywords/squash_dir

echo "=sys-devel/autoconf-2.65" >> /etc/portage/package.keywords/autoconf

echo "=sys-fs/unionfs-fuse-0.23" >> /etc/portage/package.keywords/unionfs-fuse

echo "=sys-devel/autoconf-wrapper-8" >> /etc/portage/package.keywords/autoconf-wrapper-8

# build squash_dir 

emerge squash_dir

# relocate distfiles to /var/portage

mkdir -p /var/portage

mv /usr/portage/distfiles/ /var/portage/distfiles

echo "DISTDIR=\"/var/portage/distfiles\"" >> /etc/make.conf

# fix emerge --sync and the ro .unionfs file -- possibly not needed if using aufs2?

echo "PORTAGE_RSYNC_OPTIONS=\" --exclude=/.unionfs\"" >> /etc/make.conf

# I had to emerge git, you may already have

USE="-dso curl" emerge git

# git the aufs2 standalone patch for 2.6.31

git clone http://git.c3sl.ufpr.br/pub/scm/aufs/aufs2-standalone.git aufs2-standalone.git

cd aufs2-standalone.git/

git checkout origin/aufs2-31  #  31 = 2.6.31, XX = 2.6.XX

# make a copy of unpatched gentoo kernel for backup

cd /usr/src

cp -a linux-2.6.31-gentoo-r6 linux-2.6.31-gentoo-r6-aufs

cd linux-2.6.31-gentoo-r6-aufs

# apply needed aufs2 patches

patch -p1 < ~/aufs2-standalone.git/aufs2-base.patch

patch -p1 < ~/aufs2-standalone.git/aufs2-standalone.patch 

# and append "-aufs" to local version

sed -i 's/CONFIG_LOCALVERSION=\".*\"/CONFIG_LOCALVERSION=\"-aufs\"/' .config

# build and install kernel with minimal patches for aufs module

make

make modules_install

make install

# load new kernel (and pwn anyone who blindly pasted script)

reboot
```

after booted into -aufs kernel,

```
# build and install the new aufs2 module

cd ~/aufs2-standalone.git/

make

# is there a better way to get the ko file in the right spot?  I don't really know if this is the right techinque.. ?

mkdir -p /lib/modules/2.6.31-gentoo-r6-aufs/kernel/fs/aufs/

cp aufs.ko /lib/modules/2.6.31-gentoo-r6-aufs/kernel/fs/aufs/

# refresh modules

depmod -a

# load aufs module - hope this works.. if so it should be successful (you may already have loop loaded)

modprobe aufs

modprobe loop

# auto load those 2 next time

echo -e "loop\naufs" >> /etc/modules.autoload.d/kernel-2.6

# setup squash_portage script

cd /etc/init.d

ln -s squash_dir /etc/init.d/squash_portage

cd -

#basic config that was in the readme .. adjust as desired. 

echo -e "DIRECTORY=\"/usr/portage\"\\nDIR_CHANGE=\"\${DIRECTORY}.changes\"\\nDIR_SQUASH=\"\${DIRECTORY}.readonly\"\\nTHRESHOLD=40000" > /etc/conf.d/squash_portage

# start squashing portage, script auto creates new squash file system :-)

/etc/init.d/squash_portage start

# see if it worked right

mount

# try this

emerge --sync

# it works, add to default run level

rc-update add squash_portage default
```

i'm going to let this go for a week, no problems so far, and if still none next week I will experiment squshing /usr/src

to get this to export over nfs, i had to edit ~/aufs2-standalone.git/config.mk to have CONFIG_AUFS_EXPORT = y and CONFIG_AUFS_INO_T_64 = y (2nd one is only amd64).  And after rebuilding and reinstalling the aufs.ko, I had to set ...,fsid=2000) in /etc/exports for /usr/portage options .. where 2000 is a unique export ID

----------

## mv

The ebuild to squash_dir is now available on the mv overlay which can be installed with layman: You might have to do 

```
layman -f
```

 first to get the most current list of overlays, and then 

```
layman -a mv
```

 will install the corresponding overlay. It is recommended to put the line 

```
mv
```

 into /etc/eix-sync.conf (you have to generate this file if you have not done so, earlier) and to use 

```
eix-sync
```

 instead of eix --emerge. This way, you will always get the newest versions of the ebuild in case of updates (usually, updates will not be announced here). Of course, instead of the line mv you might also want to use * to update all layman ebuilds.

----------

## Dont Panic

Thanks mv.

I've found this to be a helpful tool on my partitions that are slightly low on space.

----------

## js08

Hi I have sometimes a problem with the squash_dir (portage) runscript. The stopping fails with an error ... rc-status shows that squash_portage is still running.. so it is no longer possible to start or restart the squashfs portage tree without manual intervention. It is also not possible to stop because it fails again and again...

in detail:

1) first try (stop)

```

/etc/init.d/squash_portage  stop

* Caching service dependencies...                                                                      [ ok ]

* Squashing /usr/portage to /usr/portage.sqfs...

* (this may take a while)                                                                              [ ok ]

* Unmounting /usr/portage...

umount: /usr/portage.readonly: device is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1))

* umount -d -- /usr/portage.readonly failed [exit with 1]

umount: /usr/portage.sqfs: not mounted

* umount -d -- /usr/portage.sqfs failed [exit with 1]                                                  [ !! ]

* ERROR: squash_portage failed to stop

```

mount shows that usr/portage.readonly is still mounted....

which is expected...

2) all following stop calls have this output....

```

/etc/init.d/squash_portage  stop

* Unmounting /usr/portage...

umount: /usr/portage: not mounted

* umount -- /usr/portage failed [exit with 1]                                                          [ !! ]

* ERROR: squash_portage failed to stop

```

3) start, restart produce this output

```

/etc/init.d/squash_portage  start

* WARNING: squash_portage has already been started

```

so as a workaround for using this script I have to mount /usr/portage manually and then call 

/etc/init.d/squash_portage  stop again....

ok, the real problem is that the device is sometimes busy. I don't know

why - because a manual umout /usr/portage.readonly works....

[/quote]

best regards

----------

## mv

 *js08 wrote:*   

> 2) all following stop calls have this output....
> 
> ```
> 
> /etc/init.d/squash_portage  stop
> ...

 

This is strange; I cannot produce this here: Recent versions of squash_dir (certainly since 10.3, probably also earlier ones) should even after such an error try to umount the other directories, too. Of course, this means anyway that the "stop" will fail, but everything should be umounted (if it can be umounted).

 *Quote:*   

> 3) start, restart produce this output
> 
> ```
> /etc/init.d/squash_portage  start
> 
> ...

 

This is clear and due to a slight misconception of openrc (and I suppose, the same for baselayout-1). The problem is: What should squash_dir do in such a case? Returning error status 0 and claiming that everything is ok is probably not appropriate (squash_dir received an error for umounting which can have all sorts of unexpected consequences). On the other hand, if the error status is nonzero, openrc will automatically assume that squash_dir was not stopped - there is no possibility from within the script to say "Something strange happened, but I stop anyway".

The solution which openrc wants in such a case (if you are sure that the stop was successful) is that you call  *Quote:*   

> /etc/init.d/squash_portage zap

 

However, once more: It is strange that squash_portage only wants to umount /usr/portage and after the failure does not try to umount the other directories - I cannot see from the code (in current versions) why this could happen, and I also cannot produce this behaviour.

----------

## js08

thanx for your fast reply!

I use the "bleeding edge"   :Wink:  from your overlay and have currently version 10.5 (2010/05/20) installed together with two slightly different vanilla 2.6.34 64bit kernels on two different hosts. And it's reproducable on both! And It happens nearly everytime when I stop it.

I get always this "umount: /usr/portage.readonly: device is busy." message and when I execute "umount /usr/portage.readonly" immediately after "squash_portage stop" it works and /usr/portage.readonly is no longer mounted. I have absolutely no idea why /usr/portage.readonlly is always busy in my environments....

PS:

And it's not the mount problem ( https://bugs.gentoo.org/show_bug.cgi?id=304443 ) because I no longer do a bind-mount for my 32bit-chroot. My workaround for the 32bit-chroot is to copy the whole 40MB portage.sqfs-file to the chroot directory in the chroot32-startscript. This is at the moment the only way to avoid the mentioned file system corruption...

----------

## js08

that the sys-fs/unionfs-fuse-0.25_alpha9999 doesn't run stable in my environment. 

It seems that high load (e.g. using eix-sync) kills the unionfs-fuse stuff and then /usr/portage is empty.... 

As a consequence of this I ended several times with an 4kb portage.sqfs file his evening....

and a busy /usr/portage.readonly.

----------

## mv

 *js08 wrote:*   

> II get always this "umount: /usr/portage.readonly: device is busy." message and when I execute "umount /usr/portage.readonly" immediately after "squash_portage stop" it works and /usr/portage.readonly is no longer mounted.

 

I suppose that there is not much the script can do about this. (Maybe inserting a sleep before line 592 helps?). However, the strange thing is that if you call squash_portage stop afterwards it should attempt again to umount /usr/portage.readonly: If this is succesful you see probably no message for this, but if you retry again it cannot be successful: Either at the second or the third call you should see a message that /usr/portage.readonly cannot be umounted (either for some reason in the second call or in the third call because it is already umounted).

 *Quote:*   

> that the sys-fs/unionfs-fuse-0.25_alpha9999 doesn't run stable in my environment.

 

Please report this upstream to unionfs-fuse. For manual testing and reporting: This is what "squash_portage start" should actually execute in your case:

```
mount squashfs -o loop,ro -- /usr/portage.sqfs /usr/portage.readonly

unionfs -o cow -o allow_other -o use_ino -o nonempty -o hide_meta_dir /usr/portage.changes=RW:/usr/portage.readonly=RO /usr/portage
```

----------

## aakef

Upstream is just reading here   :Wink:  and would like to see a detailed problem discription. [/quote]

----------

## js08

 *aakef wrote:*   

> Upstream is just reading here   and would like to see a detailed problem discription. 

 

ok the unionfs-fuse issue: (happens with mv's overlay version and the new 0.24 version, so I switched back to 0.23)

a simple eix-sync/emerge -sync which executes rsync ends with an empty /usr/portage. According to mtab it is still mounted as unionfs-fuse - so from my point of view the unionfs-fuse must be dead in some way because the underlying readonly-squashfs (/usr/portage.readonly) is still accessible. I'm not able to give more information because there are NO error messages in the logs and no kernel-oops. (only the rsync-error messages that it is not able to write/read some stuff to/from /usr/portage)  :Sad: 

 *Quote:*   

> 
> 
> I suppose that there is not much the script can do about this. (Maybe inserting a sleep before line 592 helps?). 
> 
> 

 

The workaround for this strange umount behaviour helps!!! First I tried 10 seconds, then I reduced the sleep to 1 second and it works. 

I tested and verified it this evening more than 10times on both hosts with a one second sleep in line 592. The error occurs now only when I comment out the "sleep 1" line.

----------

## aakef

Are you sure its 0.24 and not 0.25alpha? 0.24 is mostly a bugfix release to 0.23. But 0.25 simplifies path building (unionfs internal) and handles MAX_PATH_LEN better. But this change might have introduced bugs. 

However, I'm a Debian user and only found that thread here, as I had been curious what Gentoo is using unionfs-fuse for. So that means I am not familiar with what you are doing.

If you want to help me to debug it, I'm sure we will have have a solution immediately as soon as I know what is going on.

So could please desribe what you are doing and what is exactly the failure?

From mv's post I can see how you run unionfs, but then it starts to become unclear. There "busy" problem on umount? And/or you cannot access the union anymore?

If possible, please compile unionfs-fuse with "-DDEBUG" and then run it in the forground using the "-d" flag and capure the output. 

Presently there are not syslogs, as it would deadlock on writing syslogs, if /var is a union served by unionfs-fuse. So syslogs are disabled until the ring-buffer+syslog threads branch is ready (almost done and will be merged with 0.25 final).

Thanks in advance,

Bernd

----------

## mv

 *aakef wrote:*   

> There "busy" problem on umount?

 

I cannot reproduce it, but the code which should be executed on "squash_portage stop" is essentially 

```
flock -w 20 -- /etc/mtab.lock umount -i -- /usr/portage

flock -w 20 -- /etc/mtab.lock umount -d -i -- /usr/portage.readonly || flock -w 20 -- /etc/mtab.lock umount -d -i -- /usr/portage.sqfs
```

The second line should free the squash-filesystem (the second alternative in this line should actually not occur and is only there as a fallback for some versions of util-linux with loop-aes patch).

The problem seems to be that in the moment when the first umount returns, the corresponding directory /usr/portage.readonly might still be reported as busy: The sleep I suggested is between the two lines. Of course, I can hardcode the sleep in the script, but it appears to me like a hack: I would suppose that the first umount should not return until all filesystems in the union are freed (and thus no longer busy).

Edit: Inserted also flock and option -i (who knows: maybe this causes the problem? However, without it, I had problems with aufs2). The flock appears necessary since umount does not seem to be thread-safe, and with openrc it might happen that several instances of the script (for separate directories) run in parallel.

----------

## mv

If umount fails, squash_dir-10.6 will now sleep for 1s and retry (and repeat this procedure). Moreover, it attempts now to check whether the directory is already umounted and skip the umount if this is the case. I guess, in most cases this is the desired behavior although perhaps some strange error situations are not cought now.

----------

## js08

 *aakef wrote:*   

> Are you sure its 0.24 and not 0.25alpha? 0.24 is mostly a bugfix release to 0.23. But 0.25 simplifies path building (unionfs internal) and handles MAX_PATH_LEN better. But this change might have introduced bugs. 
> 
> However, I'm a Debian user and only found that thread here, as I had been curious what Gentoo is using unionfs-fuse for. So that means I am not familiar with what you are doing.
> 
> 

 

With both, first I got the empty /usr/portage directory with 0.25-xxx, then I switrched back to 0.24, same effect several times and now I'm using 0.23 again... I think gentoo's /usr/portage-directory is really good for unionfs-fuse stress-testing. I contains >100k small files so a rsync with the original /usr/portage-tree is from the network point of view normally really fast and the unionfs-fuse has a lot to do. And during such rsyncs /usr/portage becomes inaccessible and a "ls /usr/portage" shows an empty directory... 

 *aakef wrote:*   

> 
> 
> There "busy" problem on umount?
> 
> 

 

The busy umount problem has nothing to do with the 0.24/0.25alpha empty /usr/portage directory problem.

----------

## aakef

Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse? I assume the kernel is definitely the same? Are you able to create a test case for me? So I already know the unionfs-fuse command and now I know that ?it is not only the umount issue. But I still do not have the slightest idea, what is actually your problem. Saying that 0.23 works, but that 0.24/0.25 fail is not sufficient, I'm afraid.

1) How does it fail, what are the symptoms?

2) If it fails, can you check using 'ps ax | grep unionfs' if unionfs is still running?

3)  Depending on 2)

a) If it is running, does it show in 'top' a high cpu usage?

b) If it is not running anymore, you should get 

#define ENOTCONN        107     /* Transport endpoint is not connected */

bernd@bathl Ph.D>errno 108

#define ESHUTDOWN       108     /* Cannot send after transport endpoint shutdown */

on accessing the union path

4) What do I need to do to reproduce it? 

4.1) If it is not easy to reproduce, are you willing to run debug commands yourself? So recompile it with debug symbols (-g) and -DDebug (using cmake that should be easy). And then runing it either in valgrind or gdb and provide me the output.

Thanks,

Bernd

----------

## js08

 *mv wrote:*   

> If umount fails, squash_dir-10.6 will now sleep for 1s and retry (and repeat this procedure). Moreover, it attempts now to check whether the directory is already umounted and skip the umount if this is the case. I guess, in most cases this is the desired behavior although perhaps some strange error situations are not cought now.

 

works!

----------

## mv

 *aakef wrote:*   

> Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse?

 

fuse is not slotted in gentoo, so the answer will almost surely be "yes". Moreover, libfuse has not been upgraded for a while in gentoo: The latest stable (and simultaneously latest unstable) version of fuse in the portage tree is 2.8.1 which is therefore almost surely what js08 is using.

----------

## js08

 *aakef wrote:*   

> Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse? I assume the kernel is definitely the same? Are you able to create a test case for me? So I already know the unionfs-fuse command and now I know that ?it is not only the umount issue. But I still do not have the slightest idea, what is actually your problem. Saying that 0.23 works, but that 0.24/0.25 fail is not sufficient, I'm afraid.
> 
> 1) How does it fail, what are the symptoms?
> 
> 2) If it fails, can you check using 'ps ax | grep unionfs' if unionfs is still running?
> ...

 

on one host if have now re-installed the 0.25alpha:

0) /usr/portage mounted - ls shows the contents - fine.

1) emerge --sync

result:

```

>>> Starting rsync with rsync://81.91.242.10/gentoo-portage...

>>> Checking server timestamp ...

----------------------------------------------------------------

|               .d88888b.           888                        |

|              d88P" "Y88b          888                        |

|              888     888          888                        |

|              888     888 888  888 88888b.   .d88b.           |

|              888     888 888  888 888 "88b d8P  Y8b          |

|              888 Y8b 888 888  888 888  888 88888888          |

|              Y88b.Y8b88P Y88b 888 888 d88P Y8b.              |

|               "Y888888"   "Y88888 88888P"   "Y8888           |

|                     Y8b                                      |

|==============================================================|

|                :: QUBE MANAGED SERVICES LIMITED ::           |

|==============================================================|

|                     http://www.qubenet.net/                  |

----------------------------------------------------------------

|ipv4             : 81.91.242.10                               |

|ipv6             : [NOT AVAILABLE AT PRESENT - coming soon..] |

|servername       : mirror.qubenet.net.                        |

|bandwidth        : 1 Gbit/s                                   |

|server specs     : VMWare vSphere, 2GB RAM, Gentoo Linux OS   |

|server location  : QUBE LN1 :: London, United Kingdom         |

|contact          : Qube Support :: mirror-admin@qubenet.net   |

|phone            : +44-207-150-3810 (NOC)                     |

|--------------------------------------------------------------|

|Please do not abuse this mirror.                              |

|It is provided as a free service, thank you!                  |

|--------------------------------------------------------------|

receiving incremental file list

timestamp.chk

          32 100%   31.25kB/s    0:00:00 (xfer#1, to-check=0/1)

Number of files: 1

Number of files transferred: 1

Total file size: 32 bytes

Total transferred file size: 32 bytes

Literal data: 32 bytes

Matched data: 0 bytes

File list size: 27

File list generation time: 0.001 seconds

File list transfer time: 0.000 seconds

Total bytes sent: 113

Total bytes received: 1890

sent 113 bytes  received 1890 bytes  1335.33 bytes/sec

total size is 32  speedup is 0.02

----------------------------------------------------------------

|               .d88888b.           888                        |

|              d88P" "Y88b          888                        |

|              888     888          888                        |

|              888     888 888  888 88888b.   .d88b.           |

|              888     888 888  888 888 "88b d8P  Y8b          |

|              888 Y8b 888 888  888 888  888 88888888          |

|              Y88b.Y8b88P Y88b 888 888 d88P Y8b.              |

|               "Y888888"   "Y88888 88888P"   "Y8888           |

|                     Y8b                                      |

|==============================================================|

|                :: QUBE MANAGED SERVICES LIMITED ::           |

|==============================================================|

|                     http://www.qubenet.net/                  |

----------------------------------------------------------------

|ipv4             : 81.91.242.10                               |

|ipv6             : [NOT AVAILABLE AT PRESENT - coming soon..] |

|servername       : mirror.qubenet.net.                        |

|bandwidth        : 1 Gbit/s                                   |

|server specs     : VMWare vSphere, 2GB RAM, Gentoo Linux OS   |

|server location  : QUBE LN1 :: London, United Kingdom         |

|contact          : Qube Support :: mirror-admin@qubenet.net   |

|phone            : +44-207-150-3810 (NOC)                     |

|--------------------------------------------------------------|

|Please do not abuse this mirror.                              |

|It is provided as a free service, thank you!                  |

|--------------------------------------------------------------|

receiving incremental file list

rsync: failed to set times on "/usr/portage/app-accessibility": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/dasher": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/emacspeak": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/espeak": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/festival-freebsoft-utils": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/festival-it": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/gnome-speech": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/gnopernicus": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/mbrola": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/morseall": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/orca": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/perlbox-voice": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/pidgin-festival": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/sound-icons": No such file or directory (2)

rsync: failed to set times on "/usr/portage/app-accessibility/speakup-utils": No such file or directory (2)

...

```

2)

ps ax | grep unionfs

```

2539 pts/5    S+     0:00 grep --colour=auto unionfs

25419 ?        Ssl    0:01 /usr/sbin/unionfs -o cow -o allow_other -o use_ino -o nonempty /usr/portage.changes=RW:/usr/portage.readonly=RO /usr/portage

```

3a)

top shows ca. 40% processor-usage, also when try to rsync again

4) I think you don't need gentoo linux.  Get a snapshot of the portage tree from http://gentoo.inode.at/snapshots/, extract it, make a squashfile... and then execute a rsync with one of the gentoo mirrors. I dont know exactly what rsync options emerge --sync, eix-sync use but I think it's more or less this command line...

rsync -av --exclude=/.unionfs rsync://rsync.europe.gentoo.org/gentoo-portage /usr/portage

----------

## js08

 *mv wrote:*   

>  *aakef wrote:*   Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse? 
> 
> fuse is not slotted in gentoo, so the answer will almost surely be "yes". Moreover, libfuse has not been upgraded for a while in gentoo: The latest stable (and simultaneously latest unstable) version of fuse in the portage tree is 2.8.1 which is therefore almost surely what js08 is using.

 

fuse - 2.8.1, 

kernel (at the moment 2.6.34-gentoo, but before a vanilla 2.6.34)

----------

## aakef

Hmm, can't reproduce it  :Sad: 

/dev/loop0 on /tmpa/gentoo/portage.ro type squashfs (rw)

unionfs on /tmpa/gentoo/portage type fuse.unionfs (rw,nosuid,nodev,allow_other,default_permissions)

 /home/bernd/src/unionfs-fuse/unionfs/BUILD/src/unionfs -d -o cow -o allow_other -o use_ino -o nonempty -o hide_meta_dir /tmpa/gent/portage.rw=RW:/tmpa/gentoo/portage.ro=RO /tmpa/gentoo/portage

bathl:/tmpa/gentoo# /home/bernd/src/unionfs-fuse/unionfs/BUILD/src/unionfs --version

Debug mode, log will be written to ./unionfs_debug.log

unionfs-fuse version: 0.25

FUSE library version: 2.7.4

fusermount version: 2.7.4

using FUSE kernel interface version 7.8

bathl:~# rsync -a  rsync://rsync.europe.gentoo.org/gentoo-portage /tmpa/gentoo/portage/

 -========== B Y T E M A R K   H O S T I N G   M I R R O R  ==========-

bathl:~# rsync -a  rsync://rsync.europe.gentoo.org/gentoo-portage /tmpa/gentoo/portage/

Server name:             tux.rainside.sk

IP address:              212.89.225.155

System:                  Intel(R) Pentium(R) D CPU 2.80GHz, 4GB RAM

Bandwidth:               1 Gbit/s

Server location:         Bratislava, Slovakia

Contact:                 kominek@rainside.sk

Please note: common gentoo-netiquette says you should not sync

more than once a day.  Users who abuse the rsync.gentoo.org

rotation may be added to a temporary ban list.

I think I need to update libfuse and to check what then happens.

----------

## mv

I think it is related with the way rsync accesses files - I suppose it keeps some filehandle open which collides with cow or something similiar. Here is a script for which rsync reports a nonexistent dir, although it exists. I tried the same without the squashfs, and the error did no occur. However, with aufs2 there is also no such error, so it is somewhat the interplay between squashfs, unionfs-fuse, and rsync which matters: 

```
#! /bin/sh

T="`mktemp -d /tmp/XXXXXXXX`"

cd -- "$T"

mkdir readonly readonly/a changes union

echo 1 >readonly/a/1

cp -a readonly reference

mksquashfs readonly sqfs >/dev/null

rm -rf readonly/a

mount -t squashfs -o loop,ro -- "$T/sqfs" "$T/readonly"

# The touch is only to force a different time so that rsync wants to set times:

touch -d 10:30 reference/a

unionfs -o cow -o allow_other -o use_ino -o nonempty -o hide_meta_dir \

   "$T/changes=RW:$T/readonly=RO" "$T/union"

cd "$T/union"

set -x

/usr/bin/rsync -a -- "$T/reference/" .

set +x

cd /tmp

umount -i -- "$T/union"

sleep 1

umount -d -i -- "$T/readonly"

rm -rf -- "$T"
```

(I am sorry, but this week I am too busy and will not have time to discuss further).

Edit: Made the example slightly more "minimal". Note that if "a" is an empty directory, the problem does not occur either, so this is really the minimal test case.

----------

## js08

 *aakef wrote:*   

> Hmm, can't reproduce it 
> 
> I think I need to update libfuse and to check what then happens.

 

strange. 

- with unionfs-fuse versions >0.23 it is so reproducable that I'm not able update the portage tree - every rync fails. 

- I tried also the next older version in the portage tree libfuse 2.7.4 - same rsync issue

I still think that my environment is not very special. It's a simple 64bit pc linux environment which runs on a pc (amd) and on a laptop (intel)

so my next steps is to compile a debug version and let's see what happens

----------

## aakef

Martin, thanks a bunch for your reproducer script. While running it in debug mode, I noticed a few new bugs in 0.25.  I could only reproduce it with the  script after I added this line

ln -s readonly/a/1 readonly/a/link_1

I think I will add the script to our regression test script.

So now the issue: http://podgorny.cz/~bernd/hg/hgwebdir.cgi/0.25/rev/fc2f284a0fff

I have not the slightest idea how that could ever work in 0.23, but this is a kernel (or glibc) bug and not unionfs related. 

Please check out the recent 0.25alpha version and please report if it works or not.

Thanks,

Bernd

----------

## js08

Is it possible to upload/send you the output of ddd/gdb and the concerning console output? 

I have created at least one which ends in an empty /usr/portage directory.

EDIT: 

Since I didn't find an upload page here or on radek's website I uploaded the tar-file to uploaded.to

http://ul.to/7j52bl/unionfs-fuse.100525.out.tgz

----------

## aakef

Thanks for the logs, I think you used Martins previously compiled version, right? The bug will be most probably gone, if you rebuild using the current hg-version (0.25 branch). For example here:

utimens /app-dicts/stardict-freedict-eng-swe 1274740792.000000000

1274738977.000000000

   unique: 16404, error: -2 (No such file or directory), outsize: 16

unique: 16405, opcode: OPENDIR (27), nodeid: 4868, insize: 48

Definitely 'fixed' in 0.25-hg. As it is not a unionfs bug, but a kernel/glibc issue, I simply disabled the error for now. If the issue still exists, I would appreciate further debug logs. However, debugging must be enabled for the build, that way unionfs will provide lots of information what it does internally. Maybe it will be a good idea, if I simply also check for "-d" in unionfs (so far libfuse reads it) and enable debug information without the need to recompile...

Thanks,

Bernd

----------

## js08

 *aakef wrote:*   

> Thanks for the logs, I think you used Martins previously compiled version, right? 

 

exactly !

----------

## aakef

Hi all,

just would like to ask if you have got a chance to try out the recent 0.25-hg version already?

Thanks,

Bernd

----------

## mv

 *aakef wrote:*   

> just would like to ask if you have got a chance to try out the recent 0.25-hg version already?

 

Last time I re-emerged, I got the version from June 1, ChangeSet 462. It worked fine.

----------

## js08

works fine.

----------

## aakef

Great, thanks for testing it. Btw, the 'official' to hide the .unionfs  directory is "-o hide_meta_files". The reason are .fuse... files, which are created, if an open file gets deleted. A ubuntu user who also works with squashfs run into those files and so we need to blacklist those as well. "-o hide_meta_dir" still works, but is deprecated.

Cheers,

Bernd

----------

## mv

 *aakef wrote:*   

> the 'official' to hide the .unionfs  directory is "-o hide_meta_files".

 

This is known

----------

## msalerno

sys-fs/squash_dir-10.8

sys-fs/squashfs-tools-4.0

sys-fs/fuse-2.8.1

sys-fs/unionfs-fuse-0.24

I'm currently using these scripts to compress portage.  I had no issue with the install, but the majority of the time when I do an emerge --sync or an eix-sync, I get lots of "failed: Too many open files (24)"

"rsync: mkstemp "/usr/portage/xfce-base/xfce4-meta/.Manifest.gr8fut" failed: Too many open files (24)"

"lsof -u root / | wc -l" currently shows >1700 open files.  I have not messed with my limits.

----------

## aakef

Hmm, too many open files by rsync? That's a bit weird. When you get this, could you please check /proc/`pidof unionfs`/fd if there are open filedescriptors left over? If not, or if those reduce within a few seconds, then it is linux cache and the gentoo script will need to specify the option "-o max_files=number" (e.g number=16384").

But if those FDs stay, then it is bug in unionfs-fuse. If that is true, could you please tell me what kind of file those are (the links /proc/fs/fd will tell you). 

Yesterday I also pushed an new debug patches into the 0.25alpha branch, if you would enable debugging with that "-o /tmp/debug_file", we could check why those files are not closed.

Thanks,

Bernd

----------

## samonli

sys-fs/squash_dir-10.8

sys-fs/squashfs-tools-4.0

sys-fs/fuse-2.8.1

sys-fs/unionfs-fuse-0.25-9999

/etc/conf.d/squash_portage:

DIRECTORY="/usr/portage"

DIR_CHANGE="${DIRECTORY}.changes"

DIR_SQUASH="${DIRECTORY}.readonly"

THRESHOLD=40000

/etc/init.d/squash_portage start:

```

squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...

squash_portage       |mount: unknown filesystem type 'aufs'

squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

```

mount:

```

/dev/loop0 on /usr/portage.readonly type squashfs (ro)

unionfs on /usr/portage type fuse.unionfs (rw,nosuid,nodev,allow_other,default_permissions)

```

after set "ORDER=unionfs-fuse unionfs funionfs aufs"  in /etc/conf.d/squash_portage

/etc/init.d/squash_portage start:

```

squash_portage       |Failed to open //funionfs/: No such file or directory. Aborting!

squash_portage       |

squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...

squash_portage       |mount: unknown filesystem type 'aufs'

squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

```

Does this ok?

thanks

----------

## mv

 *samonli wrote:*   

> /etc/init.d/squash_portage start:
> 
> ```
> 
> squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...
> ...

 

This is ok: It is first attempted to mount with aufs (which fails, since apparently you do not have the kernel patched), so it falls back to the second choice in the default order which apparently succeeds (so quash_portage has probably succesfully started).

 *Quote:*   

> after set "ORDER=unionfs-fuse unionfs funionfs aufs"  in /etc/conf.d/squash_portage
> 
> /etc/init.d/squash_portage start:
> 
> ```
> ...

 

However, this I cannot reproduce. Are you sure that you did not start with ORDER="funionfs aufs ..."? In my test, after successfully mounting with unionfs-fuse, it exits as it should (and if unionfs-fuse fails,l it prints an error message before trying unionfs, then also failing and printing an error, and only then trying unionfs).

----------

## samonli

Sorry,my mistake,i miss the "" for ORDER in /etc/conf.d/squash_portage

Correct:

ORDER="unionfs-fuse unionfs funionfs aufs"

my:

ORDER=unionfs-fuse unionfs funionfs aufs

Thanks very much!

----------

## msalerno

Currently getting too many open files.

/proc/`pidof unionfs`/fd

ls -1 | wc -l

1024

----------

## msalerno

After I had that issue, I had to restart the sqash_portage service since it had been unmounted.  Now it's holding steady at 34 open files.

----------

## ernov

Is this guide up to date? I am not asking because I don't have that few hunderd mb on my hardisk, it's just Im planning to move my system to ssd. Is this howto a good way of treating ssd?

----------

## mv

 *ernov wrote:*   

> Is this howto a good way of treating ssd?

 

You mean concerning reducing of numbers of writes?  It depends: It will reduce the numbers of writes if you put TMPDIR and DIR_CHANGE to something else than ssd. To put TMPDIR to a ramdisk is reasonable safe, but putting DIR_CHANGE to ramdisk means that after a power loss all changes are lost.

----------

## mattst88

So I tried this last night, using the ebuilds from mv's overlay.

I'm getting errors during emerge --sync like

rsync: failed to set times on "/usr/portage/app-benchmarks/gtkperf": No such file or directory (2)

which is quite disappointing since I see that these were reported a few months ago. And indeed, I see 623 files open in /proc/`pidof unionfs`/fd.

What am I supposed to do? Use the unionfs-fuse-0.25-9999 ebuild? I'd rather not install mercurial just for a single package. Is there a snapshot ebuild somewhere?

----------

## mv

 *mattst88 wrote:*   

> Use the unionfs-fuse-0.25-9999 ebuild?

 

Yes; the bug is not fixed in earlier versions, and untils unionfs-fuse-0.25 is officially released, you will probably not find it in portage.

Another way is to use aufs2 (e.g. with the live-ebuild from the mv overlay).

----------

## mattst88

I hope newer versions fix a lot more than just that.

Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

This simply isn't worth it.

Also, I don't have any idea what's going on here, or what's causing it.

```
Thinkpad pkg # pwd

/var/db.changes/pkg

Thinkpad pkg # ls -lh

total 8.0K

drwxr-xr-x 3 root  root        4.0K Jan 11 16:35 app-emulation

crw-rwsr-- 1 32518 32808 1930, 1607 Feb 18  1970 dev-perlArchive-Zip-1.30

crw-rwsr-- 1 32518 32808 2063, 1567 Feb 18  1970 media-libsa52dec-0.7.4-r6

drwxr-xr-x 3 root  root        4.0K Jan 11 13:59 media-video

b-ws-ws--- 1 32518 32808 2190, 1583 Feb 18  1970 x11-libscairo-1.8.10
```

----------

## mv

 *mattst88 wrote:*   

> Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

 

Did you add least add the line

```
PORTAGE_RSYNC_EXTRA_OPTS="${PORTAGE_RSYNC_EXTRA_OPTS} --exclude=/.unionfs --exclude=.fuse_hidden*"
```

to /etc/make.conf? It is not optimal, but better than nothing, and without it you are screwed.

 *Quote:*   

> Also, I don't have any idea what's going on here, or what's causing it.

 

Indeed, something is very broken here - of course, this causes your problems with emerge --prune. I never had such problems (on the other hand, I normally use aufs2). Probably, this broken setting is now already stored in your archive, so I see no chance that we can reproduce how this happened. This is very serious, since /var/db/pkg contains valuable data - you have to reemerge the broken packages to fix it (and probably this re-emerging will cause file collisions and may leave orphaned files). I just can recommend to not use unionfs-fuse for a valuable directory like /var/db/pkg any more, if it does not work reliably on your machine...

----------

## mattst88

I gave up on this, for now at least.

Yeah, I had that line in /etc/make.conf.

You mention this bug being fixed between 0.24 and 0.25, but I don't see any commits after 0.24. http://hg.podgorny.cz/unionfs-fuse/shortlog

So, I don't understand.

----------

## mv

 *mattst88 wrote:*   

> but I don't see any commits after 0.24.

 

It seems that unionfs-fuse is now developed in mercurial, only. There is a live ebuild in the mv overlay.

----------

## mattst88

So I gave this another try with a unionfs-fuse snapshot from mercurial.

I'm using it on my laptop where it seems to work fine, but I'm also using it on my portage NFS server.

There, I see lots of strange 'Manifest file not found: '/usr/portage/sys-fs/udev/Manifest'' and stale NFS file handle errors. Any ideas what's going on here?

Usually, unmounting and remounting /usr/portage/ on the clients temporarily resolves the problem.

----------

## p04ty

What about this message: 

```
/etc/init.d/squash_portage sync

squash_portage     | * Use of the opts variable is deprecated and will be

squash_portage     | * removed in the future.

squash_portage     | * Please use extra_commands or extra_started_commands.

squash_portage     | * Syncing portage tree ...
```

----------

## mv

 *p04ty wrote:*   

> What about this message ...

 

This should be fixed in squash_dir-11.0.

----------

## p04ty

What's that? How would it compare to in-kernel squashfs?

----------

## mv

 *p04ty wrote:*   

> What's that? How would it compare to in-kernel squashfs?

 

I do not understand your question.

squash_dir is essentially only an init-script which in the simplest configuration (when started) uses the in-kernel squashfs (together with aufs or unionfs-fuse or others) to mount a squashed directory rewritable and which (when stopped) recompresses the new directory. Of course, it expects the in-kernel squashfs (and aufs or unionfs-fuse or others) to work, although there are some fallbacks for emergency cases. (But I am not sure whether this was your question...)

----------

## F1r31c3r

 *p04ty wrote:*   

> What about this message: 
> 
> ```
> /etc/init.d/squash_portage sync
> 
> ...

 

Hey people this has become global across all init.d scripts for OpenRC. If you use baselayout 2 it is advisable as the warning suggests to change the "opts" to "extra_commands" but in doing this you make backward compatibility a problem. Here is the original comments on this.

http://git.overlays.gentoo.org/gitweb/?p=proj/openrc.git;a=commitdiff;h=df1f02ac848a010092df2d3d40b8828051522b4b

This means that its a good idea to update on one hand but it means this warning is not critical.

Here is how i upgraded the commands:

 *Quote:*   

> 
> 
> # cd /etc/init.d
> 
> # grep -r "opts" *
> ...

 

This will display all files using the opts as statement. WARNING the opts that dont hove = on eg "$fsck_opts -R" are not to be changed only them that show up at the start of the scripts saying opts=

 *Quote:*   

> 
> 
> grep -r "opts=" *
> 
> 

 

The above command shows the real opts that need changing.

Here is the quick dirty way to change them to extra_commands NOTE the = sign after the words dont leave it out or it will break your system.

 *Quote:*   

> 
> 
> # grep -lr -e "opts=" * | xargs sed -i "s/opts=/extra_commands=/g"
> 
> 

 

Now double check it change them correctly:

 *Quote:*   

> 
> 
> # grep -r "opts=" *
> 
> it should find nothing
> ...

 

Pick yourself a init.d script and restart it see if it works i chose iptables and sysklogd:

 *Quote:*   

> 
> 
> # /etc/init.d/iptables restart
> 
> # /etc/init.d/sysklogd restart
> ...

 

They should all restart without the above named warnings and or any other warning or errors.

You have successfully migrated the opts variable on your baselayout 2 OpenRC gentoo box.

 :Very Happy: 

----------

## mv

 *F1r31c3r wrote:*   

> but in doing this you make backward compatibility a problem.

 

Since baselayout-1 is now gone from the tree and meanwhile even the oldest openrc in the tree supports extra_commands, I would not worry about backward compatibility. However, with patching init-scripts from the tree, I would wait until they are officially bumped (which does not necessarily mean a revision bump of the corresponding package - you have to check manually).

----------

## kyoshiro

 *F1r31c3r wrote:*   

>  *p04ty wrote:*   What about this message: 
> 
>  *Quote:*   
> 
> # cd /etc/init.d
> ...

 

Please have a special look at files containing ${opts} references, e.g. /etc/init.d/reboot.sh aso.

You'll have to fix them afterwards:

# grep -lr -e "{opts}" * | xargs sed -i "s/{opts}/{extra_commands}/g"

----------

## SlashBeast

```
sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i
```

----------

## lpapa

 *SlashBeast wrote:*   

> 
> 
> ```
> sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i
> ```
> ...

 

Still doesn't work for some scripts, for example, sshd script uses variable "myopts" and then refers to it as ${myopts}.

I fix it adding -e 's:${myopts}:${myextra_options}:g' but I don't know if this may affect other packages which I didn't installed on my system.

----------

## Stolz

In order to reduce the writes of my future SSD disk I've successfully installed squash_dir via layman and configured it to squash both /urs/portage and /usr/src, saving all the files in my rotational disk. Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.

----------

## Dont Panic

 *Stolz wrote:*   

> Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.

 

I was going to suggest /var/db/pkg

However, that directory doesn't meet your criteria for "not of vital importance".  Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered.    :Smile: 

----------

## Stolz

 *Dont Panic wrote:*   

> I was going to suggest /var/db/pkg
> 
> However, that directory doesn't meet your criteria for "not of vital importance".  Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered.   

 

Thanks Dont Panic, I'm aware of it, the squash_dir README mentions it, but since it's dangerous to play with it and my raid disks already have some S.M.A.R.T errors  I prefer to leave it alone  :Smile: 

----------

## mv

 *Stolz wrote:*   

> However, that directory doesn't meet your criteria for "not of vital importance".

 

Actually, I have good experience with squashing it with keeping a backup (setting FILE_SQFS_OLD). This backup even saved my installation once when a kernel panic caused damage in /var/db.

/usr/share/doc or even /usr/share is a good candidate for squashing from the viewpoint of compression ratio, but there is an enormous drawback: These directories are huge and change with every package upgrade. So you should put a huge THRESHOLD and then need sufficient disk space to have the compressed directory twice on disk together with the THRESHOLD (since, of course, it will be first compresssed and the old squash removed only afterwards). Also, many things in /usr/share are somewhat vital.

Other good candidates: /usr/share/games /usr/share/texmf-dist /usr/lib/libreoffice (the first two are mentioned in the README).

Note that if you compress a lot of directories you must increase the number of loopback-devices:

Put e.g. 

```
options loop max_loop=16
```

 into /etc/modprobe.d/loop.conf

----------

## gringo

Stolz pointed me to this thread and i have it now running in one of my systems, thanks a lot !

Only one thing though : if you enable the kernel-patch USE in aufs from mv overlay it will also apply the grsec one and if the kernel isnt grsec enabled it will fail to build here. 

Maybe a grsec USE is needed or sth. ?

oh, and thanks also for the zram-init stuff, ive been using it for some time already  :Smile: 

cheers

----------

## mv

 *gringo wrote:*   

>  if the kernel isnt grsec enabled it will fail to build here.

 

The patch should be fixed now to build in all cases.

----------

## gringo

 *mv wrote:*   

> The patch should be fixed now to build in all cases.

 

nice, thanks a lot !

cheers

----------

## costel78

Is it possible to port the init script to systemd ?

I looked over actual openrc script, but it is too far complex to simple migrating the commands   :Very Happy: 

----------

## mv

 *costel78 wrote:*   

> Is it possible to port the init script to systemd?

 

I don't know. AFAIK systemd does not have proper dependencies and probably also not a standard route during shutdown which is crucial for the script.

Actually, I consider the whole systemd a very bad idea (running permanently daemons which take resources only to save 1-2 seconds during booting is not sane), so I hope that I will never be forced to use it, and I am not at all interested in porting something to a system with a bad concept.

 *Quote:*   

> I looked over actual openrc script, but it is too far complex to simple migrating the commands  

 

Essentially, it is the "start" and "stop" which would have to be called at the proper place. The other provided command are only for wrapper scripts to check the current state (e.g. whether it would be saved). Probably one could put a part in a "library" which is sourced.

----------

## costel78

Thank you for your answer!

Yes, I already tried to simulate the logics behind and, so far, I found no way to start tar and bzip compress of the portage tree BEFORE /usr/portage was umounted.

What was the simulation:

 - a separate partition for /usr/portage (without distfiles, of course) mounted automatically at boot via initscript (noauto in fstab)

 - during shutdown compress it (tar -cjpf ....) and umount it 

As you said, I found no way to prevent systemd to try to umount the partition and kill tar process.

It seems that all thing it's to risky.

----------

## mv

 *costel78 wrote:*   

> As you said, I found no way to prevent systemd to try to umount the partition and kill tar process.

 

At least the mount and umount commands from sys-apps/util-linux provide a way to call helper scripts for mount and umount of certain filesystem types. I have never dealt with them and do not know how compatible this setting is. But it might be a possible to invent a "new" filesystem type and to do the actual action in these helper scripts. However, this needs to be well-tested: e.g. aufs also uses such helper scripts, so "recursive" callings of such scripts must be possible.

The advantage of this approach would be that it works with any init-system, the disadvantage is that it probably works only with util-linux.

Currently, I have no time to implement such a thing.

----------

## slycordinator

I'm having a problem with mv's squash_dir initscript.

My portage squashfs file is at /squashed_dirs/portage.sqfs and the portage.readonly and portage.changes are placed inside that same directory. And I named the portage initscript squash_portage

My portage.changes directory has a whole bunch of subdirs in it currently, since I did an "emerge --sync" earlier, but if I do "/etc/init.d/squash_portage restart" I simply get:

```
/etc/init.d/squash_portage restart

 * Unmounting /usr/portage ...                                                                   [ ok ]

 * Mounting /squashed_dirs/portage.sqfs as /usr/portage ...                                      [ ok ]
```

And the portage.sqfs is not recompressed with the portage.changes directory containing the same stuff it did previously.

----------

## mv

 *slycordinator wrote:*   

> And the portage.sqfs is not recompressed

 

This is strange. I cannot reproduce it here. Are you sure that you do not have a .no-save file and not set any threshold?

If yes, please help me debugging: Which version of squash_dir are you using? To simplify debugging, please try with the latest from the mv overlay (from today).Is the output of 

```
/etc/init.d/squash_portage print_dir_change
```

 indeed your portage.changes which contains data? In particular, does 

```
ls $(/etc/init.d/squash_portage print_dir_change)
```

 show that content?Does some of the following two commands output "1" to stderr? 

```
/etc/init.d/squash_portage need_squash

/etc/init.d/squash_portage will_squash
```

----------

## slycordinator

Note: This happens sporadically. Like when I shutdown the computer last night, the portage squashfs file got recompressed and the ones I've got for /var/db and /usr/share/texmf-dist didn't, but at least now (using version 11.6) doing a "/etc/init.d/squash_db restart" made it recompress on the first try. And the need_squash and will_squash variables showed that the tex directory will be recompressed. Guess it's fixed but not sure, given how sporadic it would happen.

edit: PEBKAC. Think it was the threshold setting.

----------

## js08

Hi, 

I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])

sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1

emerge --sync produces the following error:

```

app-backup/backup-manager/files/

app-backup/backupninja/

app-backup/backuppc/

app-backup/backuppc/files/

app-backup/backuppc/files/3.2.0/

app-backup/bacula/

app-backup/bacula/ChangeLog

      40.52K 100%   59.51kB/s    0:00:00 (xfer#24, to-check=1056/4230)

app-backup/bacula/Manifest

       8.54K 100%   12.22kB/s    0:00:00 (xfer#25, to-check=1055/4230)

app-backup/bacula/bacula-5.2.5.ebuild

      10.09K 100%   14.42kB/s    0:00:00 (xfer#26, to-check=1048/4230)

app-backup/bacula/bacula-5.2.6.ebuild

      10.09K 100%   14.09kB/s    0:00:00 (xfer#27, to-check=1047/4230)

rsync: rename "/usr/portage/app-backup/bacula/.bacula-5.2.6.ebuild.rJDBEZ" -> "app-backup/bacula/bacula-5.2.6.ebuild": Software caused connection abort (103)

app-backup/bacula/files/

app-backup/bacula/files/5.0.2/

app-backup/bacula/files/5.0.3/

app-backup/bacula/files/5.2.3/

app-backup/boxbackup/

app-backup/boxbackup/files/

app-backup/ccollect/

app-backup/cdbackup/

app-backup/cdbkup/

app-backup/cpdup/

app-backup/cpdup/files/

app-backup/dar/

app-backup/dar/files/

rsync: failed to set times on "/usr/portage/app-crypt": Software caused connection abort (103)

rsync: opendir "/usr/portage/app-crypt" failed: Transport endpoint is not connected (107)

rsync: recv_generator: failed to stat "/usr/portage/app-crypt/metadata.xml": Transport endpoint is not connected (107)

rsync: recv_generator: mkdir "/usr/portage/app-crypt/WiRouterKeyRec" failed: Transport endpoint is not connected (107)

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/usr/portage/app-crypt/acr38u" failed: Transport endpoint is not connected (107)

...

```

----------

## mv

 *js08 wrote:*   

> Hi, 
> 
> I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
> 
> sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1
> ...

 

And it worked before (i.e. with other kernels)? Then it looks to me like a race condition problem in unionfs-fuse or an incompatibility of fuse with the newest kernels.

If the start of the error message is not reproducible, I would guess on the former and suggest that you report the problem upstream at unionfs-fuse.

----------

## js08

 *mv wrote:*   

>  *js08 wrote:*   Hi, 
> 
> I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
> 
> sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1
> ...

 

yes, everything worked fine until 3-21-2012.

and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine.

----------

## mv

 *js08 wrote:*   

> and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine.

 

As I mentioned, please report upstream at unionfs-fuse - this needs help by the corresponding specialist (it probably is some race issue).

----------

## Massimo B.

Hi, since Portage gets too slow on my old ppc Notebook I found this Tip...

Where is the recent version of these init scripts maintained and who does it currently? I just found http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree, there are already newer versions than the original author wrote in the first post of this thread.

Usually I avoid using non-official approaches which tend to be un-maintained after a while, but since removing the SquashFS approach for the default Portage is just easy and the benefits of this tweak are just too nice.

Wouldn't it be nice to make the Howto a bit more official and centrally maintained by creating an ebuild with the init script in some of the overlays, so I don't need to copy/paste the init-script code from the browser into the editor?

----------

## mv

The project (squash_dir, consisting not only of the init-scripts but also of a shell-wrapper with zsh-completion  and an extensive README) is currently maintained on github.

The corresponding ebuild is in the mv overlay (available by layman).

----------

## Massimo B.

Sorry that I overlooked the middle of this thread. I'm also using sys-fs/squash_dir now from your mv overlay. Thank you very much for maintaining these scripts, nice piece of work, especially the very good README.

The only issue for now is that aufs3 sources don't build on PPC. On amd64 its working fine.

Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance (say slow 1core cpu and slow drive)? This is more important for me than space for some slow old machines and its the main reason that made me come to this thread, after reading Portage_tips.

----------

## mv

 *Massimo B. wrote:*   

> The only issue for now is that aufs3 sources don't build on PPC.

 

You might try the git sources (aufs-99999999.3, also available from the mv overlay). Let us hope that the kernel developers come to their mind and finally include aufs or at least overlayfs, although the latter has several drawbacks compared to aufs.

 *Quote:*   

> Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance

 

For compression, lzo is undoubtfully the fastest (and least effective) and xz the slowest (and most effective). However, except for the time on shutdown only decompression time plays a role. Concerning decompression, lzo is also the fastest, but surprisingly xz is not much slower. Since xz needs to access less data (because it compresses much better), it can be that on some systems (especially those with not much memory or huge cache) xz is even the winner for decompression, speedwise. Your really have to try.

On the other hand, if you have a really slow system and a huge directory, it can be that compression time of xz is really inacceptable - on some systems, I end up with gtip for this reason...

----------

## Massimo B.

Because I can't afford long startup or stop time, I restart the squashing on daily base:

```
$ cat /etc/cron.daily/1squashfs_flush

#!/usr/bin/env bash

for rc in /etc/runlevels/boot/squash_*; do

    if [[ 0 != "$($rc will_squash 2>&1 >/dev/null)" ]]; then

        $rc restart

    fi

done
```

Synchronizing a 1 week old Portage made this difference:

```
du -sh /usr/portage*

311M   /usr/portage

52M   /usr/portage.changes

280M   /usr/portage.readonly

67M   /usr/portage.sqfs
```

...which already exceeded my threshold of 40M.

Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only?

I have 3 squash_dir setups now: /usr/portage, /var/db/ and /var/lib/layman.

Because I need to use webrsync on some hosts, I've seen that after a new webrsync of the same snapshot it seems that all has been re-written:

```
$ du -sh /usr/portage*

711M   /usr/portage

711M   /usr/portage.changes

280M   /usr/portage.readonly

67M   /usr/portage.sqfs
```

Let's see if app-portage/emerge-delta-webrsync is different here.

----------

## mv

 *Massimo B. wrote:*   

> Because I can't afford long startup or stop time, I restart the squashing on daily base:

 

Are you aware that there is a squash_dir wrapper for this? *Quote:*   

> squash_dir restart

  should do the same as your script (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.

 *Quote:*   

> Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only?

 

Exactly.

 *Quote:*   

> I've seen that after a new webrsync of the same snapshot it seems that all has been re-written.

 

This is because of the COW mechanism of aufs: If something changes in a file, even if it is only the filestamp or some attributes, the file must be copied into the .changes directory. I guess webrsync changes all filestamps, at least temporarily. With other union-type filesystem the necessary COW might be even more intensive, e.g. (IIRC this was the case in some versions of unionfs-fuse) if only the directory filestamp changes then the whole directory might need to be written to *.changes - it really depends which mechanisms are used to merge the directories.

----------

## Massimo B.

 *mv wrote:*   

> Are you aware that there is a squash_dir wrapper for this? *Quote:*   squash_dir restart should do the same as your s cript (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.
> 
> 

 No I wasn't, thanks. squash_dir restart really simplifies that task or cronjob.

----------

## Massimo B.

With some manual work to get aufs3 building on ppc as described in the bug [1], your squash_dir is working fine.  You could add ~ppc keywords to squash_dir and runtitle.

[1] Bug 433497 - sys-fs/aufs3-3_p20120813: unrecognized command line option.

----------

## Massimo B.

When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts. This is ok.

After I got the aufs module built and loaded, squash_dir restart or start do not work.

```
$ squash_dir restart

/etc/init.d/squash_db is not running

/etc/init.d/squash_layman is not running

/etc/init.d/squash_local_portage is not running

/etc/init.d/squash_portage is not running

$ squash_dir start

Starting db...

squash_db           | * Mounting /var/db.sqfs as /var/db ...

squash_db           |mount: /var/db.sqfs is already mounted

squash_db           | * Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]

squash_db           |rmdir: failed to remove `/var/db.readonly': Device or resource busy                   [ !! ]_db           |

squash_db           | * ERROR: squash_db failed to start

Skipping /etc/init.d/squash_layman due to previous error

Skipping /etc/init.d/squash_local_portage due to previous error

Skipping /etc/init.d/squash_portage due to previous error

```

I first needed to umount all read.only stuff to get it starting. Maybe you can add some error safety here.

----------

## mv

 *Massimo B. wrote:*   

> When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts.

 

The problem is that it is not clear what to do:

One possibility is to add an "umount" option which tries to umount even if the start failed. However, this can have horrible undesired effects if the start failed for a different reason or if directories with the temporary name feature are involved.

The other possibility is to start anyway successfully in such a situation, because the directory is mounted but just not writable.

The disadvantage of the latter is that you will not realize that something is broken unless you study the logs (or actually attempt to write).

The real problem is that it is not possible to report to openrc a "partial success".

----------

## Massimo B.

I would like to improve performance, throughput and disk usage on a WebDAV server, which I mount via net-fs/davfs2. Now I think about if it is a good idea to store the content with squash_dir there.

Disadvantage would be of course that I can't view the files from the web-interface anymore. But I use the WebDAV only as central synchronisation for different sites using net-misc/unison.

Now the problem would be that maybe any change on the re-compressed squashfs would lead to a complete upload of the whole file again instead of sending delta only. Maybe this reason makes squash_dir the wrong solution for that, what do you think?

----------

## mv

 *Massimo B. wrote:*   

> Maybe this reason makes squash_dir the wrong solution for that, what do you think?

 

Yes, I guess this is the wrong solution. Maybe you can use a VCS like git instead?

----------

## Gordex

tried it today but didnt get it to work properly :/

Compression-var seems to be ignored..I got gzip compression all the way

an error message indicated that it tried to umount the readonly before the aufs when shutdown

openrc doesnt seem to launch it altho added to default runlevel..as there was no output in the rc.log and no proper mounts after reboot

I guess the script is a bit outdated 

anyway nice idea man!

----------

## mv

 *Gordex wrote:*   

> tried it today but didnt get it to work properly :/

 

Hard to say what you tried, but none of the effects you described were ever reported or did ever occur on one of my systems.

 *Quote:*   

> Compression-var seems to be ignored..I got gzip compression all the way

 

Perhaps you have not compiled squashfs-tools with support for any other compression algorithm. If you do not set COMPRESSION, the default is even xz.

 *Quote:*   

> an error message indicated that it tried to umount the readonly before the aufs when shutdown

 

The code does it certainly in the correct way. Perhaps you had mounted aufs manually in addion or the umounting failed for some reason. Without more information, it is not possible to say what went wrong.

 *Quote:*   

> openrc doesnt seem to launch it altho added to default runlevel..

 

This is the strangest thing, of course, because whether openrc starts it or not is not a question of the script. It indicates that there is something completely broken in your setup - perhaps the names of init.d/... and conf.d/ do not match, or you tried to add squash_dir instead of the symlink or whatever. Again, without more information, it is not possible to say what went wrong.

----------

## Gordex

I tried to doublecheck everything according to the README and got it working somehow now I believe   :Smile:  thanksalot.. the problem remaining is that COMPRESSION="xz" or lzo .. are still ignored .. I can manually squash it that way (mksquashfs -comp xz ..) but ur script takes gzip here anyway

not sure why

----------

## mv

Try to insert into line 495 of /etc/init.d/squash_dir:

```
eval "einfo mksquashfs options: ${sqfsdaddopt}"
```

When compressing, this should display you the -comp-Option actually used. If -comp does not appear here, somewhere COMPRESSION must have been explicitly defined to the empty string in your config, because the default "xz" is set in line 311.

----------

## Gordex

I did but nothin seems to have changed -_-

```
 # /etc/init.d/squash_portage start

 * It seems you start squash_portage for the first time with this configuration:

 * The squashed file /usr/portage.sqfs does not exist yet.

 * So I will try to initialize...

 * Squashing /usr/portage to /usr/portage.sqfs

Parallel mksquashfs: Using 4 processors

Creating 4.0 filesystem on /usr/portage.sqfs, block size 131072.

[===========================================================================================================

===========================================================================================================

================================================================================|] 133549/133549 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072

        compressed data, compressed metadata, compressed fragments, no xattrs

        duplicates are removed

Filesystem size 68404.20 Kbytes (66.80 Mbytes)

        26.03% of uncompressed filesystem size (262780.17 Kbytes)

Inode table size 1658360 bytes (1619.49 Kbytes)

        32.97% of uncompressed inode table size (5030387 bytes)

Directory table size 1511461 bytes (1476.04 Kbytes)

        36.17% of uncompressed directory table size (4178856 bytes)

Number of duplicate files found 8488

Number of inodes 157094

Number of files 133523

Number of fragments 2009

Number of symbolic links  0

Number of device nodes 0

Number of fifo nodes 0

Number of socket nodes 0

Number of directories 23571

Number of ids (unique uids + gids) 1

Number of uids 1

        portage (250)

Number of gids 1

        portage (250)

 * Removing all in /usr/portage

 * Now its content is only available as squashfs.

 * Name of the squashfs file: /usr/portage.sqfs

 * Mounting /usr/portage.sqfs as /usr/portage ...  
```

Admin edit: Added line breaks to unbroken line of "=" characters. --pjp

----------

## mv

 *Gordex wrote:*   

> I did but nothin seems to have changed

 

Now I understand: You were speaking about the "initial" mksquashfs, not about the one which is "normally" used when the directory is re-squashed.

You are right, here the option is ignored by mistake. Thanks for the report. I will fix it.

----------

## Gordex

ah awesome  :Smile:  I see.. thx

edit: you might add that aufs3 needs to be compiled with useflag fuse..I think thats what causes my earlier problems but not 100% sure

----------

## Massimo B.

Hi, again. Is there any rescue if aufs is not available?

Today I started my new kernel, but I forgot to build aufs3 kernel module. Now the emerge aufs3 works but fails to install, probably because /var/db is readonly.

I tried to install the module and the binaries from /var/tmp/portage manually but modprobe aufs3 still says "module not found" eventhough the .ko file is in the right directory (and found by bash_completion).

----------

## mv

 *Massimo B. wrote:*   

> Hi, again. Is there any rescue if aufs is not available?

 

Yes, but it is inconvenient: You can manually copy the readonly /var/db to some new temporary directory, then stop the squash_...-scripts, rename, /var/db to something else, and then rename the new temporary directory to /var/db. (If /var/db does not even a sane [readonly] content, you can also use unsquashfs to unpack /var/db.sqfs manually).

Of course, this works only if you have sufficient disk space for the unpackad copy of /var/db, but perhaps you can temporarily get the free space by moving something to a backup medium which you do not need for the restauration.

Then emerge aufs (hopefully it works now).

Finally rename both directories back and then reboot (or start squash_... if it works).

When everything is running you should remove the old data of /var/db/pkg/sys-fs/aufs-... and move the new data (from your last emerge) which is now in the renamed-back temporary directory to /var/db/pkg/sys-fs/aufs-...

After this the temporary directory can be deleted.

----------

## Massimo B.

Thanks for the help. That worked. Though I needed to force ORDER="aufs" in all the configurations, maybe because of the recent change. I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs.

If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches. Next kernel switch I need to remember to emerge aufs3 first against the new sources and build the aufs3-patched sources afterwards.

----------

## mv

 *Massimo B. wrote:*   

> I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs.

 

Yes, ORDER means the order of trying. So it should work even with the default ORDER but perhaps only print some error messages - but it should not fail if aufs succeeds.

 *Quote:*   

> If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches.

 

Unfortunately, the chances for this are practically zero, since despite a huge interest from the users, the kernel maintainers are not interested in including any type of union filesystem. Some kernel maintainers had announced that perhaps they might give overlayfs a chance. Although I am not so optimistic (since many months it is ignored, practically without any comments - the same has already happened to union mount for many years), maybe this is the first union type filesystem which has a chance to go into the kernel. For this reason, I have changed the default. I will of course change the default immediately back, if it seems more likely that aufs is included or overlayfs is going to be rejected.

----------

## Massimo B.

Thanks, the discussion about integrating aufs3 was the same with squashfs which finally was integrated into 2.6.29 after half of the Linux world was already using it. [german article]

Some other idea, when I was thinking about compressing binaries such as large firefox using UPX for faster startup: I could also just add things like /usr/lib64/firefox to squash_dir. That should just work with upgrades and re-squashing for now, since I even restart squash_dir once per day by cron for potential re-squashing.

EDIT: I successfully squashed firefox, from 43M to 20M, and startup feels a bit faster...

----------

## Massimo B.

Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem?

----------

## mv

 *Massimo B. wrote:*   

> Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem?

 

It is not save; you can loose any new information and perhaps even get a broken squash file. Even the remounting itself might fail if the filesystem is busy at that time.

----------

## Massimo B.

Before the last openrc update

```
    Tue Dec 18 21:18:32 2012 <<< sys-apps/openrc-0.9.8.4

    Tue Dec 18 21:18:40 2012 >>> sys-apps/openrc-0.11.8
```

..it was just possible to do squash_dir start or restart from inside a live-cd chroot. There (using Finnix on PPC) aufs and squashfs modules are provided. It was possible to mount the squash_dir parts rw and even recompress. So portage was completely functional from inside chroot.

Since the latest openrc update I get the following:

```
$ squash_dir zap

 * You are attempting to run an openrc service on a

 * system which openrc did not boot.

 * You may be inside a chroot or you may have used

 * another initialization system to boot this system.

 * In this situation, you will get unpredictable results!

 * If you really want to do this, issue the following command:

 * touch /run/openrc/softlevel

/etc/init.d/squash_db is not running...
```

Doing the touch /run... openrc tries to start bad dependencies:

```
$ squash_dir start

Starting db...

 * Caching service dependencies ...                                         [ ok ]

 * Mounting security filesystem ...                                         [ ok ]

 * Mounting debug filesystem ...                                            [ ok ]

 * Mounting cgroup filesystem ...                                           [ ok ]

 * /dev is already mounted

 * Starting udev ...                                                        [ ok ]

 * Populating /dev with existing devices through uevents ...                [ ok ]

 * Waiting for uevents to be processed ...                                  [ ok ]

 * Checking local filesystems  ...

/dev/sda5 is in use.

e2fsck: Cannot continue, aborting.

 * Operational error                                                        [ !! ]

 * Mounting local filesystems ...                                           [ ok ]

 * Mounting /var/db.sqfs as /var/db ...

mount: /var/db.sqfs is already mounted

 * Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]

rmdir: failed to remove �/var/db.readonly�: Device or resource busy         [ !! ]

 * ERROR: squash_db failed to start

Skipping /etc/init.d/squash_layman due to previous error

Skipping /etc/init.d/squash_local_portage due to previous error

Skipping /etc/init.d/squash_portage due to previous error
```

Is there any way to avoid the dependencies to get squash_dir in a working state within chroot as it was working before?

----------

## mv

 *Massimo B. wrote:*   

> squash_dir start or restart from inside a live-cd chroot.

 

In the new squash_dir-12.6 there is now a START command which executes the action of the "start" command but does not actually start the script. You can use the STOP command (which exists already quite a while) to execute the action of the "stop" command of such an officially non-started script. Of course, you must use these hacks very carefully: Double execution or stopping can have undesired results, especially if the temporary filename feature is used for the readonly directory.

----------

## Massimo B.

Thanks. I was able to start squash_dir even within chroot from live-cd now.

Next bug report, I tried to squash new directories:

```
$ cat /etc/conf.d/squash_icedtea6

DIRECTORY="/usr/lib/icedtea6"

DIR_CHANGE="${DIRECTORY}.changes"

DIR_SQUASH="${DIRECTORY}.readonly"

ORDER="aufs"

COMPRESSION=""
```

```
$ /etc/init.d/squash_icedtea6 start

squash_icedtea6     | * Caching service dependencies ...                                                                  [ ok ]

squash_icedtea6     | * It seems you start squash_icedtea6 for the first time with this configuration:

squash_icedtea6     | * The squashed file /usr/lib64/icedtea6.sqfs does not exist yet.

squash_icedtea6     | * So I will try to initialize...

squash_icedtea6     | * Squashing /usr/lib64/icedtea6 to /usr/lib64/icedtea6.sqfs

squash_icedtea6     |SYNTAX:mksquashfs source1 source2 ...  dest [options] [-e list of exclude...
```

EDIT: Parameter order has changed somehow, works now with this patch:

```
$ diff -u /etc/init.d/squash_dir /etc/init.d/squash_dir.new 

--- /etc/init.d/squash_dir   2013-01-08 10:54:05.946782911 +0100

+++ /etc/init.d/squash_dir.new   2013-01-08 10:53:21.788301570 +0100

@@ -247,8 +247,8 @@

       return 2

    }

    einfo "Squashing ${DIRECTORY} to ${FILE_SQFS}"

-   mksquashfs ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} \

-      "${DIRECTORY}" "${FILE_SQFS}" || {

+   mksquashfs "${DIRECTORY}" "${FILE_SQFS}" \

+          ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} || {

       Error ${?} "mksquashfs failed. Exiting."

       return

    }
```

----------

## mv

This is fixed in squash_dir-12.8

----------

## tabascoz

Hello, 

Did someone has written a systemd init script for squash_dir? Since the last update to gnome 3.8 and, thus, the need to switch to systemd , i had been unable to use this great toot.

----------

## mv

 *tabascoz wrote:*   

> Did someone has written a systemd init script for squash_dir?

 

AFAIK systemd is too poor to do this. I will not support this bad systemd concept, anyway.

----------

## mv

 *mv wrote:*   

> AFAIK systemd is too poor to do this.

 

It turned out that it can be done with systemd as well - only it is not possible to send the output to the terminal; that's a restriction due to the bad systemd concept. Thus, here is the announcement

squash_dir-13.0 does support systemd, experimentally.

It also contains a script openrc-wrapper which allows to use many native openrc services in other init-systems like systemd. This might be interesting also for other openrc init-scripts.

So this is the implementation: The script is native openrc, but can be fully used in systemd by the openrc-wrapper. The advantage of this solution for gentoo users is that the same configuration can be used for systemd as for openrc, i.e. squash_dir does not have to be configured twice on different places. The disadvantage is of course that native systemd users might not want to configure in /etc/conf.d and to create squash_* symlinks in /etc/init.d which is currently necessary even if openrc is not installed. In the long run, squash_dir might be replaced e.g. by a perl-script which manages all functionality in a single script which can be called externally and which manages its own database of "mounted" directories in e.g. /run, but I will not have time in the near future to implement such a universal solution. Anyway, this would be different project and should have a different name...

----------

## pjp

Split off Compressing filesystems with squashfs and squashmount

----------

## Khumarahn

What is the state of squashfs with squashdelta for portage tree?

Using the lines from teh wiki

```

# for daily squashfs snapshots

sync-type = squashdelta

sync-uri = mirror://gentoo/../snapshots/squashfs

```

results in

```

# emerge --sync

!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'

!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'

!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'

!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'

```

----------

## mv

 *Khumarahn wrote:*   

> What is the state of squashfs with squashdelta for portage tree?

 

It is abandoned, and AFAIK there are no plans to revive it.

----------

## Khumarahn

Thanks!!

What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync?

----------

## pjp

I've been using tmpfs and squashfs.

Create a tmpfs.

Create a squasfs of the portage tree.

When I sync, the squashfs is extracted to a tmpfs, the sync occurs to the in memory tmpfs and then a new squashfs is created. The tmpfs is then discarded (freeing the memory) and the new portage tree squashfs is mounted.

That process requires enough free memory to temporarily hold the tree during that sync / squashfs process.

Also, /usr/portage/distfiles and /usr/portage/packages are kept elsewhere. These directories were configurable using 

DISTDIR & PKGDIR in make.conf. man make.conf now references repos.conf. I've not found new documentation for their use in repos.conf (but since make.conf seems to work, I haven't made it a priority).

If I ever get around to making the script more generic, I had planned to post it.

----------

## Khumarahn

I am interested in squashfs primarily for a system with little memory, namely Teres 1 arm laptop. It has 2G ram and 16G MMC. With these little resources, optimizing portage from 1G to 50M is important. Then, the sync should ideally be also efficient... May not have enough ram free to keep whole portage tree.

----------

## pjp

tmpfs is optional. I apparently edited that out of my previous post. My squashfs images are ~59M.

----------

## mv

 *Khumarahn wrote:*   

> What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync?

 

You could use squashmount and call 

```
squashmount remount [repos mount point]
```

 in an /etc/portage/postsync.d hook.

----------

## Khumarahn

 *mv wrote:*   

> You could use squashmount and call 
> 
> ```
> squashmount remount [repos mount point]
> ```
> ...

 

Thanks! This worked beautifully. I get a strange error on squashmount remount:

```
mount: /usr/portage: wrong fs type, bad option, bad superblock on overlayfs, missing codepage or helper program, or other error.
```

But despite the error, it works correctly. Is there an easy way to see where the error comes from?

My config:

```
#!/usr/bin/perl (this is only for editors)

@order = qw(overlay overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);

$rm_changes = $rm_workdir = $rm_readonly = 0;

my $defaults = {

   COMPRESSION => 'xz',

   COMPOPT_XZ => ['-Xbcj', 'arm'],

};

@mounts = (

   standard_mount('portage', '/usr/portage', $defaults),

);

1;  # The last executed command in this file should be a true expression
```

```
#!/bin/sh

# This file remounts the squashmount mount-point "gentoo" after each syncing

# of the "gentoo" repository.

set -u

repository_name=$1

sync_uri=$2

repository_path=$3

# Run only for repository gentoo:

[ x"$repository_name" = x'gentoo' ] || exit 0

# Run only if mount-point portage is configured

[ x"$(squashmount --quiet print-tag portage 2>/dev/null)" = x'portage' ] || exit 0

squashmount remount portage
```

----------

## mv

 *Khumarahn wrote:*   

> I get a strange error on squashmount remount

 

squashmount has its own thread and a github bugtracker, so we should perhaps not clutter the discussion here.

My guess is that either your kernel is too old (older than 3.18 requires $obsolete_overlayfs being set, see squasmount --man) or some options are not compiled in your kernel (cf. MOUNT_OVERLAY in squashmount --man) and therefore mounting overlay(fs) fails and something else (probably unionfs-fuse) is used as a fallback. With 

```
squashmount list
```

 you can see what was effectively used for mounting. If you pass squashmount remount the option(s) -vvv you can see which commands/options are actually executed to understand which options overlayfs does not like. If you need further help or suspect a bug, I suggest we move discussion to either the above mentioned thread or (preferrably) to the github bug tracker.

----------

## Khumarahn

@mv, thank you! Running squashmount -vvv, I found that on the first attempt to mount overlayfs, the mount command does not accept the --workdir option. Probably because I am on a very old kernel with whatever custom patches by Allwinner and sunxi. By the next mount, without --workdir, it mounts and works.

I found that it is important to disable hardlinks for portage sync (`sync-allow-hardlinks = no`). Otherwise the sync is ugly, slow and takes lots of space.

I will ask more questions in the other thread or github, if I have any. So far, I am rather satisfied!

----------

## mv

 *Khumarahn wrote:*   

> --workdir option

 

This option (and the working directory) are used for the variant in newer kernels; squashmount calls this variant "overlay".

The variant in older kernels (without a working directory) is by squashmount called "overlayfs".

Since the default value of @order is qw(overlay!? overlayfs!? aufs! unionfs-fuse! unionfs! funionfs!), squashmount will first attempt 'overlay' and in case of failure 'overlayfs'. Thus, to suppress the error message, you might exchange the first two entries in your configuration file:

 */etc/squashmount.pl wrote:*   

> @order=qw(overlayfs!? overlay!? aufs! unionfs-fuse! unionfs! funionfs!);

 

This way, first the variant without a working directory is attempted. However, you will have to remove that line once you upgrade to >=linux-3.18.

----------

## Khumarahn

I already put overlayfs first... I get this that error message with

```
@order = qw(overlayfs? overlay)
```

----------

## mv

 *Khumarahn wrote:*   

> I already put overlayfs first..

 

You are right; I misremembered. The working directory was introduced in linux-3.15. The correct recommendation would probably have been instead to set

 */etc/squashmount.pl wrote:*   

> $obsolete_overlayfs = 1;

 

----------

## Khumarahn

```
$obsolete_overlayfs = 1;
```

That worked  :Smile: ))) Thanks!

----------

## Princess Nell

I went off the script posted here and aufs years ago and switched to https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs. And finally, I got around to this (local portage dir, app-portage/squashfs-portage) as I got sick of installing it manually on a new machine every time.

```

# Copyright 2021 Gentoo Authors

# Distributed under the terms of the GNU General Public License v2

EAPI=7

DESCRIPTION="Use squashfs and overlays for portage tree"

HOMEPAGE="https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs"

LICENSE="GPL-2"

SLOT="0"

KEYWORDS="~amd64 ~x86"

DEPEND="sys-fs/squashfs-tools"

RDEPEND="${DEPEND}"

BDEPEND=""

pkg_setup() {

   mkdir "${S}"

}

src_install() {

   newinitd "${FILESDIR}"/squashfs_portage-init.d squash_portage

   newconfd "${FILESDIR}"/squashfs_portage-conf.d squash_portage

}

```

Take the init script from HOMEPAGE and copy it to files/squashfs_portage-init.d. Remove the 4 lines starting with SQFS_CUR, OVERLAY_DIR, RO_DIR and PORT_DIR, and instead put them into files/squashfs_portage-conf.d. ebuild/repoman etc. to taste.

----------

## szatox

I've just had a reason to revisit my old, tiny VPS and take care of its ever growing portage tree.

Well, I guess I might just as well add bootstrap and reset buttons to the emerge sync script I wrote a few years ago and share it.

Things to watch out for during setup:

* adjust paths:

 - currently uses old /usr/portage instead of more modern /var/db/repos/gentoo, make sure TARGET matches your system settings

 - I put the actual data under /media because reasons. Feel free to send it anywhere else if it bothers you (or you want to use it on a machine with removable storage devices, just to avoid conflicts)

* name the script just like the target directory, or set environmental variables before running it. Multiple repositories can be supported with symlinks.

e.g. portage for /usr/portage, gentoo for /var/db/repos/gentoo, guru for /var/dv/repos/guru etc.

* Run with command bootstrap once (reset if the repo gets corrupted for whatever reason and you need to fully resync), mount during boot, update when you want to sync portage, and squash to compress the tree (will be needed for overlays if you have them).

Fall-through switch kicks off subsequent actions automatically and allows for some retries (or squashing overlays without running emerge --sync for everything again... Perhaps doing it in a loop instead of relying on script's name would be better now )

I handle this point with cron's @reboot and @daily tags.

```
# cat /opt/scripts/squash_it/portage

#! /bin/bash

#set -x

set_vars () {

        OVFS="${OVFS:-/media/overlay}"

        INSTANCE="${INSTANCE:-$(basename $0)}"

        TARGET="${TARGET:-/usr/${INSTANCE}}"

        SQDIR="${OVFS}/squash"

        LOWER="${OVFS}/lower/${INSTANCE}"

        UPPER="${OVFS}/upper/${INSTANCE}"

        WORK="${OVFS}/work/${INSTANCE}"

        TMP="${OVFS}/tmp"

        SQNAME="${INSTANCE}"

        SQFS="${SQDIR}/${SQNAME}"

        SQCMD="/usr/bin/mksquashfs"

# excluding files does not work as reliably as I'd like, but git it already compressed, so squashing it mostly wastes CPU time and clogs RAM and disk during update. We're better off leaving it in the overlayfs.

        SQPARAM="-quiet -no-progress  -comp xz -Xdict-size 100% -wildcards -e '.git/'"

# This was for testing, not needed anymore, but I'm too lazy to remove it right now

        MODE="eval"

}

delay (){

echo "### running command: ==> $@ <=="

sleep 5

eval "$@"

}

main (){

        set_vars

        case $1 in

        "sudo: reset" )

                "${MODE}" /bin/umount -l "${TARGET}"

                "${MODE}" /bin/umount -l "${LOWER}"

                "${MODE}" rm -rf "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQFS}"

                ;&

        bootstrap )

                "${MODE}" mkdir -p "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQDIR}"

                "${MODE}" mkdir "/tmp/empty.$$"

                "${MODE}" "${SQCMD}" "/tmp/empty.$$" "${SQFS}" ${SQPARAM}

                "${MODE}" rmdir "/tmp/empty.$$"

                echo "disk usage (squash + upper)  and uncompressed data size (lower) after bootstrap"

                du -hs "${SQFS}" "${LOWER}" "${UPPER}"

                main mount

                main update

                ;;

        update )

                "${MODE}" emerge --sync -q || exit 1

                echo "disk usage (squash + upper)  and uncompressed data size (lower) after sync"

                du -hs "${SQFS}" "${LOWER}" "${UPPER}"

                        ;&

        squash )

                "${MODE}" "${SQCMD}" "${TARGET}" "${TMP}/${SQNAME}" ${SQPARAM} || exit 2

                        ;&

        clean )

                "${MODE}" /bin/umount -l "${TARGET}" || exit 3

                "${MODE}" /bin/umount -l "${LOWER}" || exit 4

                "${MODE}" mv "${TMP}/${SQNAME}" "${SQFS}" || exit 5

                "${MODE}" find "${UPPER}/ -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} \+" || exit 6

                        ;&

        mount )

                "${MODE}" /bin/mount -o loop "${SQFS}" "${LOWER}" || exit 7

                "${MODE}" /bin/mount -t overlay overlay -o "lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK}" "${TARGET}" || exit 8

                echo "disk usage (squash + upper)  and uncompressed data size (lower) after remount"

                du -hs "${SQFS}" "${LOWER}" "${UPPER}"

                ;;

        *)

                echo "usage: $0 < sudo: reset | bootstrap | update | squash | clean | mount >"

        esac

}

main "$@"

```

The workdir of portage apparently contains 250MB of text files, uses 700-900MB of disk space (depending on FS) and compresses own to 35MB.

I know, I know, disk space is cheap, but my VPS only have 20GB of it.

----------

