# Storing portage-tree in one file: big speed gain

## bur

I recently found this tip in the gentoo-wiki and since it seems to be rarely used (or at least I know of nobody who does) but really gives big speed-ups when browsing the tree, I thought I'd point people to it:

Portage in Sparse File

Here's an example of the speed gain, portage is the new tree and portage.old the ... well, old one:

```

buren portage # time du -sh .

424M    .

real    0m16.825s

user    0m0.248s

sys     0m2.134s

buren portage # time find . > /dev/null

real    0m1.075s

user    0m0.260s

sys     0m0.802s

buren portage # cd ../portage.old/

buren portage.old # time du -sh .

401M    .

real    0m56.485s

user    0m0.309s

sys     0m1.925s

buren portage.old # time find . > /dev/null

real    0m0.840s

user    0m0.230s

sys     0m0.600s

buren portage.old #

```

Also any other operations like emerge -s or equery work much faster.

----------

## NeddySeagoon

Moved from Portage & Programming to Documentation, Tips & Tricks.

----------

## irondog

Okay, Reiser is the better choice for small files then. The trick is to think well about which filesystem to use on which files.

Really, loop mounting a file carrying a filesystem is nothing special. It should also be clear that portage uses a lot of files.

----------

## xanas3712

Well yes, but if it's so obvious why isn't everyone doing it?  I think it was worth posting..

BTW, what is meant by 

 *Quote:*   

> 
> 
> and if you want some real speed, you could do the same with /var/db/pkg and /var/cache/edb ( faster dependency checks etc ) , but doing that is bound to end in tears 
> 
> 

 

I was thinking about going ahead and doing this too, but I'm not sure why it would end in tears?

----------

## bur

irondog,

yes, it might be obvious, but I think it isn't widely used. And so I thought it was worth posting to point people to that idea.

xanas,

I wondered about that as well and don't see how it could be any different from /usr/portage. As long as one keeps a backup of the dirs trying shouldn't do any harm, so maybe I'll give it a try tomorrow.

----------

## synss

The nice thing with doing it with the tree is that if ever anything would go wrong, you can just download a new one. Recovery is nothing that easy with other dirs, however, if you keep backups, then it sure is fine.

And you can also compress the tree since you are at it

https://forums.gentoo.org/viewtopic-t-465367-highlight-.html

----------

## bur

Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. So the gain of space will only be temporarily, meaning I cannot use it for anything I want to keep for some longer time.

----------

## xanas3712

Yeah, I like this way, and it is very very fast by comparison to how it was.  I went ahead and backed up the other dirs and did them as well, not as much of a speed boost as doing it for the tree but still pretty nice.

----------

## yoshi314

 *Quote:*   

> Compressing the tree sounded interesting at first, but if I got it right updateing the tree still requires the full 500-600MB. 

 nope, you missed a thing  :Wink: 

the new squashfs image is created directly from mounted squashfs image and unionfs branch which contains the changes made by e.g  recent emerge --sync. so there is no need for unpacking the squashfs image. 

at least the squashfs-initscript method does that.

----------

## bur

yoshi314,

I see, I'll try that method when I find some time.

Regarding placing portage directories in a sparse file, I tried this with /var/db/pkg and /var/cache/edb as suggested by the wiki-article but actually experienced a decrease in speed. Both "du" and "find" took about 20-30% more time. It seems like the structure of these dirs differs from /usr/portage so they don't benefit from reiserfs but rather the opposite.

----------

## netmaster

Dude! Excellent tip man. Wow. Thanks!

----------

## c0r0

Ohh the tips works well and i like it  :Very Happy:  thanks for this

----------

## xming

Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details

----------

## yoshi314

 *xming wrote:*   

> Instead of sparse file or squashfs you can use jffs2, which only use +/- 120 MB and it's fully writable, see http://wojia.be for more details

 120mb is a lot, compared to squashfs image, which takes ~35mb. (at least with squashfs 3.0)

and if you just did emerge --sync, unionfs takes a couple of megs of RAM to track the changes, until you rebuild the squashfs image.

----------

## xming

120MB is stll a lot beter tahn 600MB + jffs2 is in the vanilla kernel and you never have to rebuild anything. HOw big is you unionfs overlay after few syncs?

----------

## bur

I tried making /usr/portage a jffs2 file but it didn't work.

Here's what I did:

```
# I use loop1 as loop0 is already in use

# I substituted blkmtd for block2mtd, I think that's what it's called in my kernel (ck-2.6.17)

# dmesg | tail or grepping for mtd didn't yield anything

03:53:16 [/usr/src/linux] # mknod /dev/mtdblock0 b 31 0

03:53:46 [/usr/src/linux] # losetup /dev/loop1 /root/portage.jffs2

03:53:58 [/usr/src/linux] # modprobe block2mtd device=/dev/loop1

03:54:11 [/usr/src/linux] # mount -t jffs2 /dev/mtdblock0 /usr/portage.new/

mount: wrong fs type, bad option, bad superblock on /dev/mtdblock0,

       missing codepage or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

```

Any idea what's wrong?

----------

## xming

Is jffs2 compiled in the kernel or as a module? Did you modprobe jffs2? Is your /root/portage.jffs2 formatted as jffs2?

Can you post the output of dmesg|tail?

----------

## bur

Strangely now another error popped up, when trying to load block2mtd I get this:

```
FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg)
```

And dmesg|tail says "block2mtd: Unknown parameter `device'".

I don't know why this happens now as yesterday it didn't and I entered the same comand as before:

```
13:38:53 [/home/bur] # modprobe jffs2

13:39:09 [/home/bur] # mknod /dev/mtdblock0 b 31 0

13:39:17 [/home/bur] # losetup /dev/loop1 /root/portage.jffs2

13:39:24 [/home/bur] # modprobe block2mtd device=/dev/loop1

FATAL: Error inserting block2mtd (/lib/modules/2.6.17-ck1-r2/kernel/drivers/mtd/devices/block2mtd.ko): Unknown symbol in module, or unknown parameter (see dmesg)
```

Is there any way to find out which parameters a module supports? Maybe it isn't called device anymore. Or maybe block2mtd isn't the right module at all, though I doubt that.

----------

## xming

replace

 *Quote:*   

> modprobe block2mtd device=/dev/loop1 

 

with

```
modprobe block2mtd block2mtd=/dev/loop1 
```

this is my script for 2.6.17

```

modprobe loop

mknod /dev/mtdblock0 b 31 0

losetup /dev/loop0 /tmp/portage.jffs2

modprobe block2mtd block2mtd=/dev/loop0

mount -t jffs2 /dev/mtdblock0 /usr/portage

```

For info about the modules try

```

modinfo block2mtd

filename:       /lib/modules/2.6.17-beyond3/kernel/drivers/mtd/devices/block2mtd.ko

license:        GPL

author:         Simon Evans <spse@secret.org.uk> and others

description:    Emulate an MTD using a block device

vermagic:       2.6.17-beyond3 preempt mod_unload PENTIUMIII gcc-3.4

depends:        mtdcore

srcversion:     EA21219D30D40143B1D006C

parm:           block2mtd:Device to use. "block2mtd=<dev>[,<erasesize>]"

```

----------

## bur

Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?

I think I found out I have to run losetup each boot and after that load block2mtd (because it needs the /dev/loop). How could this be done most easily? And when adding the jffs2 file to fstab, what parameters are needed? For example the sparse method used loop,noatime and some more.

Also where exactly is the portage tree stored? Is it like that:

/tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.

----------

## xming

 *bur wrote:*   

> Thanks, it works now and takes much less space then the sparse file and is much faster to access. What do I need to do to have it mounted automatically each boot?

 

I just put the script in /etc/conf.d/local.start

 *Quote:*   

> 
> 
> Also where exactly is the portage tree stored? Is it like that:
> 
> /tmp/portage.jffs2 -> /dev/loop0 -> /dev/mtdblock0 -> /usr/portage ? But if the actual data is in portage.jffs2, why is the file placed in /tmp? The data is not only needed temporarily, if I got it right.
> ...

 

data is in /tmp/portage.jff2, for me it's tmp, which it is data that can be deleted, screwed, .... If you prefer to place it elsewhere (I think /var/spool would be the perfect place) you can do that.

----------

## bur

Everything works fine now but I ran out of space on /dev/mtdblock0 making emerge --sync impossible. How can I resize the partition besides creating a new one from scratch? And is there a way to make it a little bigger than needed so that I won't run into that problem again?

----------

## xming

are you using a partition or a loopback file? If it's a loopback, you just need to recreate the file.

----------

## xentric

I'm running Gentoo on a Pentium3-850Mhz with 256Mb RAM...

I use ReiserFS on my partitions so I had to use ext2 as filesystem in the file.

Before putting Portage in the file, an "emerge --sync" would take about 20 minutes.

After putting Portage in the file it only takes 10 minutes   :Very Happy: 

Thanks!

----------

## zxy

Did anybody try to copy the squashfs file to ram and mount it there.

It's just 35 or 40 Megs.

What are the speeds/timings then?

----------

## Gentree

both of your speed comparisons maybe flawed since much of the gain will be due to defragging your reiserfs with the copy.

I no longer use that fs for just that reason.

Try copying portage.old to portage.not.so.old and benchmark all three.

There may well be some gains with your technique but I'd bet a fair bit of it is simply a well ordered fs against an old fragmented one.

@xentric

you cant really use emerge --sync as a comparison because the network and the server will be the major factors not your local fs. Also once you have done an rsync the next one will be quicker because there is nothing to sync!

Also using ext2 (an unjournaled fs) is bound to be faster than just about any other option, so let's compare the comparable. Put your old tree on ext2 and compare that.

All this may be useful but let's have a bit of science here if we're posting numbers.

personally I now user Reiser4 for portage (have done for over 2yrs actually.) That's much faster the reiserfs.

----------

