# [HOWTO] Best boot speedup yet!!

## devsk

The idea is to preload the useful files into kernel buffer cache before they are needed. Since I/O can happen parallel to the other CPU intensive tasks during bootup, this leads to faster bootup and faster startup time for your chosen applications. Other projects use static lists to preload the files. But I have automated the process of generating the "useful" prefetch database.

Have lsof installed before you install it and test it out. 

This is /etc/init.d/my-readahead.

```

#!/sbin/runscript

# Copyright 1999-2006 Gentoo Foundation

# Distributed under the terms of the GNU General Public License v2

# $Header: $

depend() {

        before logger

}

start() {

        ebegin "Starting readahead ROYALE"

        cat /etc/conf.d/* > /dev/null 2>&1 &

        cat `cat /etc/conf.d/sofile-list.load` > /dev/null 2>&1 &

        if [ -f /forcesampler ];then

                /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &

                rm -f /forcesampler

        fi

        eend 0

}

stop() {

        eend 0

}

```

The executable shell script /usr/sbin/sample-init-process:

```

#!/bin/bash

final_db="$1"

# total number of lists to keep in /etc/conf.d/

total_to_keep=4

# total number of samples to make in /forcesampler boot.

# It will take twice this amount of seconds to finish sampling.

total_samples_per_boot=200

# touch empty file if not there

[ ! -f "$final_db" ] && touch "$final_db"

let i=0

while [ "$i" -ne "$total_to_keep" ] ; do

        if [ ! -f "$final_db$i" ] ; then

                break;

        else

                j=$((i+1))

                if [ "$j" = "$total_to_keep" ] ;then

                        j=0

                fi

                if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then

                        i=$((j))

                        break;

                fi

        fi

        i=$((i+1))

done

if [ "$i" = "$total_to_keep" ] ;then

        i=0

fi

slot="$i"

cp "$final_db" "$final_db$i"

function collect_sample_old()

{

        lsof |awk '{print $9}'|grep \.so|sort|uniq > $1

        lsof |awk '{print $9}'|grep \.conf|sort|uniq >> $1

        lsof |awk '{print $9}'|grep /bin/|sort|uniq >> $1

        lsof |awk '{print $9}'|grep /sbin/|sort|uniq >> $1

        lsof |awk '{print $9}'|grep ttf|sort|uniq >> $1

        lsof |awk '{print $9}'|grep /libexec|sort|uniq >> $1

}

function collect_sample()

{

        /usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | grep -v "^$" | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]" > $1

}

collect_sample /tmp/init_sample1

i=0

while [ "$i" -ne "$total_samples_per_boot" ] ; do

        collect_sample /tmp/init_sample2

        cat /tmp/init_sample1 /tmp/init_sample2 | /usr/bin/uniquer > /tmp/init_sample3

        mv /tmp/init_sample3 /tmp/init_sample1

        sleep 0.5

        i=$((i+1))

done

cat /tmp/init_sample1 "$final_db$slot" | /usr/bin/uniquer > "$final_db"

rm -f /tmp/init_sample[123]

```

And this is /usr/bin/uniquer:

```

#!/bin/sh

/usr/bin/awk '{

if ($0 in stored_lines)

        x=1

else

        print

        stored_lines[$0]=1

}' $1

```

And then do this:

```

chmod +x /usr/sbin/sample-init-process /usr/bin/uniquer /etc/init.d/my-readahead

rc-update add my-readahead default

touch /forcesampler

```

What you have done is asked my-readahead to create a sampler process which samples what files (fonts, binaries, libraries, configs) are in use, every second. It collects 200 samples (will be busy for at least 10 minutes after boot) and keeps updating the database with a list of useful files.

Once you boot after this the first time, you will probably see a slow down because its collecting samples for use on the next and subsequent boots. Use your system as you normally would. Open firefox, gaim, OO or whatever. Once the sampler script has finished ('ps -aef|grep -v grep|grep sample-init-process' will tell you if its running), reboot. Now this and all subsequent boots will be much faster. <update> Something has changed with baselayout recently and requires you to hit enter when the sampler starts during the once-only boot for sample collection. </update>

If it takes too long for the sample-init-process to finish, reduce the total_samples_per_boot variable in the sampler script to 100 and try again by touching /forcesampler.

In future, whenever you feel like updating the prefetch database, just 'touch /forcesampler' and database will be updated on next reboot.

Uninstall is easy:

```
rc-update del my-readahead

\rm /etc/init.d/my-readahead

\rm /usr/sbin/sample-init-process

\rm /etc/conf.d/sofile-list.load*

\rm /usr/bin/uniquer

```

Last edited by devsk on Sat Aug 19, 2006 2:10 am; edited 9 times in total

----------

## johntash

I'll give it a try  =]

----------

## suicidal_orange_II

Me too - the idea sounds good and anything that speeds up booting (and program loading?) must be good   :Very Happy: 

Suicidal_Orange

----------

## devsk

I made it a howto and put more work into automating it after finding it shaved off close to 15 seconds off of my startup time. I strongly believe that if we have to preload something, it has to be something which will be used. And that's what this script tries to do.

I am down to 40 seconds for a full fledged KDE desktop with initrd and splash enabled at boot and tonnes of services like UPS, sensors, hddtemp etc. starting at default. Moreover, firefox and gaim start really fast when I get my desktop.

I am thinking this might be a better sampler, more concise and faster:

```

/usr/sbin/lsof -F n -f |grep "^n"|sed -e "s:^n::g" | sort | uniq | grep -v "^/dev"| grep -v "^/proc"| grep -v "^/tmp"| grep -v "^/var"| grep -v "/.mozilla"| grep "^/[a-z]"
```

What do you guys think?

----------

## cypherphreak

Mine appears to have not worked.

The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &

from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?

----------

## devsk

what does 

```
wc -l /etc/conf.d/sofile-list.load
```

 show? is there a list of files in there?

----------

## jit_v1

Thanks for the howto. I've just tried using it but unfortunately don't seem to have gained anything from my boot time  :Sad: 

The sofile-list.load seems to have been created correctly etc. and contains 180 files.

I noticed that the rc-script is being loaded in the default runlevel, could this be moved further up so that it starts earlier?

It's probably worth altering the rc-script so that it doesn't do the preload if /forcesampler exists - otherwise the sampler will pickup previously sampled files that may no longer be required.

I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.

jv

----------

## devsk

 *jit_v1 wrote:*   

> 
> 
> I also changed the depend part to 'before logger' - since 'before syslog-ng' wasn't working for me.
> 
> 

 ok, this is a bug. Did you rebuild the database after doing this? I think it didn't capture much for you. Was it still sampling after you had your system up? For me, the list has 500 filenames, all of those files are the ones which are used during the boot and afterwards. So, it helps me a lot.

----------

## jit_v1

Yeah, I rebuilt the database after, got an extra 20 or so files, but still didn't seem to make much difference to the startup time. The sampler process stayed running after bootup for around five minutes. I did notice that my boot time wasn't affected that much when the sampling was going on - you mentioned a delay of 10 minutes to boot - it took just a little longer than normal. Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?

jv

----------

## thomasvk

So if you put something in /dev/null it will be cached?   :Shocked: 

----------

## drwook

 *t0maz wrote:*   

> So if you put something in /dev/null it will be cached?  

 

Sure, whatever it reads gets buffered/cached.  so cat'ing it to /dev/null is as good a way as any.  Could probably just as well grep it for the word 'wibble' and redirect the output to /dev/null or something, but that'd just look odd in a script  :Wink: 

----------

## devsk

 *jit_v1 wrote:*   

> Perhaps I need to turn up the sample rate - do you think that changing the sleep amount to less than a second would be beneficial?
> 
> jv

 yeah, your system is too fast I think, so sampling at probably 0.5 or even 0.2 seconds might give you better results. Note that reducing the sampling interface will be more intrusive while collecting the first time. Also, you might have to up the number of samples because now it will finish in half the time with 0.5 sleep instead of 1 sec sleep. You don't wanna finish too early either.

Another reason for different size database is that you are probably running a different set of services than me.

One way of finding if its working or not is: disable the service my-readahead. Reboot and login into the box. Inside X, do a top and see how much memory is used. Note it down. Enable the service, reboot and login into the box and again do a top to see the memory used. This number should be substantially higher, but close to what you will eventually use if you used your system normally. You can compare o/p of 

```
cat /proc/meminfo
```

 for these two scenarios and it will clearly indicate that you have loaded extra stuff (all useful because you were using that stuff when we sampled it).

----------

## stahlsau

gj, thanx for the script, i like the idea  :Wink: 

At first, i thought you'd be kidding us with cat'ing the files to /dev/null. But after a lil thinking ...  :Wink: 

----------

## balk

 *cypherphreak wrote:*   

> Mine appears to have not worked.
> 
> The init script deletes forcesample as directed then runs /usr/sbin/sample-init-process /etc/conf.d/sofile-list.load &
> 
> from here it closes almost instantly. My question is do i have to manually create /etc/conf.d/sortfile-list.load since that would be the $1 value in the script which is where final_db is stored correct?

 

Do you have lsof installed? I didn't and the script obviously did not work.  :Smile:  I needed another cycle before it worked; I had  zero-byte /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0 files which I had to delete first.

I will now reboot and find out if there is any increase in speed.

[edit]

There seems to be ~some~ increase. I did not measure but it felt faster  :Smile:  Thanks for the nice script!

----------

## dway

Hi, I'm new on Gentoo (and on linux in general) and i have a fully fonctionnal system now (learning from the documentations, wiki and forum).

Just a little question : does this script break something if I try to use it or is it safe for the system (by the way how do you clear it form the system ?)

----------

## devsk

dway, if you can't figure out what its doing, you should learn to figure that first. The script is harmless but I would recommend you read up and figure what it is doing before going installing it. You need 'lsof' installed. There are only couple of files it creates. Its easy to get rid of the whole thing by just doing:

```

rc-update del my-readahead

\rm /etc/init.d/my-readahead

\rm /usr/sbin/sample-init-process

\rm /etc/conf.d/sofile-list.load*
```

----------

## dway

thx for infos, I'll try to figure it out before trying, regarding your advice (sorry for my english).

----------

## Ic3M4n

Hi all!

I have a problem starting the init script

I have an output like this:

```
Could not get dependency info for "my-readahead"!

Please run:

# /sbin/depscan.sh

to try and fix this
```

if I start the script in the default runlevel it starts a little before xdm. I try to insert it in the boot level and it starts before syslog-ng but I don't like the output error I post. please, some ideas for fix this?

----------

## devsk

 *Ic3M4n wrote:*   

> Hi all!
> 
> I have a problem starting the init script
> 
> I have an output like this:
> ...

 what baselayout, sysvinit, udev versions are you using?

----------

## cypherphreak

 *balk wrote:*   

> Do you have lsof installed?

 

Simple things like that are always overlooked by me, thanks for pointing that out, works fine now.

----------

## thomasvk

So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?

----------

## prymitive

 *t0maz wrote:*   

> So what if some script or something does some stuff and with that dumps a lot of stuff (like command ouput, think of beagle, for example) into /dev/null... does that get cached and do you lose your nice speedups? Or do I sound stupid now?

 

It's not about dumping some output stuff to /dev/null but reading files from disk so they are put in buffers so when some app want to read them it gets it from buffer with is a lot faster then reading from disk. You think that /dev/null is some buffor or am I stupid?

----------

## thomasvk

 *prymitive wrote:*   

> You think that /dev/null is some buffor or am I stupid?

 

You're stupid!

No, I'm sleeping...  :Embarassed: 

----------

## prymitive

 *t0maz wrote:*   

>  *prymitive wrote:*   You think that /dev/null is some buffor or am I stupid? 
> 
> You're stupid!
> 
> No, I'm sleeping... 

 

Don't get me wrong, I wasn't sure if I understood Your question, it's been long time since my english lessons  :Wink: .

----------

## Dr.Dran

@devsk

Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc

Thanx  :Very Happy: 

P.S. I your patch work better you may submit it to Uberlod (the baselayout dev)  :Very Happy: 

Cheers

Franco

----------

## stahlsau

 *Quote:*   

> Hi, excuse me but I would like to know if you have tryed this patch with the feature RC_PARALLEL_STARTUP enabled in the /etc/conf.d/rc
> 
> 

 

It works, and i see no problem with those two enabled. Imho there's no way they could interfere.

----------

## zxy

I just installed your script. Good work.

I have a suggestion that would speed thing up even more.

As I saw the files in  /etc/conf.d/sofile-list.load are alphabeticaly sorted. As I understand thigs, it would be better to just live files unsorted. I mean, they should stay in the same order as they get loaded.

If a file is near the end  of the list it gets loaded twice!

first time when it is realy needed (but not cached yet) and the second time when it wants to be cached, but caching it came too late.

----------

## devsk

sorting is required to take out the duplicates. Moreover, its very difficult to tell the order of loading of files from 'lsof' output.

----------

## zxy

I agree that all entries in the list must be unique.

I think that sorting (uniqueing) algoritm must be different.

The easiest to implement is to check for every entry that should to be added if it is already in the table and add it only if it is not.

I did't program much in bash so I can't help much with this.

I think that even if sorting algoritm is slow it runs only once. But every next time you get better improvement at startup.

Maybe I'll dig into man pages to implement this throwing out non unique entries (not realy sorting).

---EDIT

My digging and surfing results:

```
 sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' db.lst
```

this takes only non unique lines from file db.lst without sorting them.

If data from every second was cleaned with this and on the end the final file too, the result would be much faster boot. I guess

----------

## zxy

So here it is. I rewrote parts of the script (I hope the author alows it (no GPL statements there))   :Wink: 

It makes entries in the loading list unique, but it doesent sort them, so the first needed are preloaded (cached) first, so they can be used imediately from cache.

I changed the file /usr/sbin/sample-init-process

```

#!/bin/bash 

final_db="$1" 

# total number of lists to keep in /etc/conf.d/ 

total_to_keep=4 

# total number of samples to make in /forcesampler boot. 

# It will take twice this amount of seconds to finish sampling. 

total_samples_per_boot=150 

# touch empty file if not there 

[ ! -f "$final_db" ] && touch "$final_db" 

let i=0 

while [ "$i" -ne "$total_to_keep" ] ; do 

        if [ ! -f "$final_db$i" ] ; then 

                break; 

        else 

                j=$((i+1)) 

                if [ "$j" = "$total_to_keep" ] ;then 

                        j=0 

                fi 

                if [ -f "$final_db$j" -a "$final_db$i" -nt "$final_db$j" ];then 

                        i=$((j)) 

                        break; 

                fi 

        fi 

        i=$((i+1)) 

done 

if [ "$i" = "$total_to_keep" ] ;then 

        i=0 

fi 

slot="$i" 

cp "$final_db" "$final_db$i" 

function collect_sample() 

{ 

        lsof |awk '{print $9}'|grep \.so| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  > $1 

        lsof |awk '{print $9}'|grep \.conf| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  >> $1 

        lsof |awk '{print $9}'|grep /bin/|  sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  >> $1 

        lsof |awk '{print $9}'|grep /sbin/|  sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  >> $1 

        lsof |awk '{print $9}'|grep ttf|  sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  >> $1 

        lsof |awk '{print $9}'|grep /libexec| sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  >> $1 

} 

collect_sample /tmp/init_sample1 

i=0 

while [ "$i" -ne "$total_samples_per_boot" ] ; do 

        collect_sample /tmp/init_sample2 

        cat /tmp/init_sample1 /tmp/init_sample2 |  sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  > /tmp/init_sample3 

        mv /tmp/init_sample3 /tmp/init_sample1 

        sleep 1

        i=$((i+1)) 

done 

cat /tmp/init_sample1 "$final_db$slot" |  sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'  > "$final_db" 

rm -f /tmp/init_sample[123]

```

I generaly changed the | sort | unique  to the line from previous post. 

The data collecting took much more processor power (sed) and it took a little longer, so I lowered total_samples_per_boot to 150 (I think you can leave it at 200).

RESULT: startup (boot) is much better.

----------

## devsk

 *zxy wrote:*   

> So here it is. I rewrote parts of the script (I hope the author alows it (no GPL statements there))  

 feel free to edit it as you please. One more thing, you might want to consider is the sampling technique in post 4. It does lsof only once per sample and may be faster overall when combined with your new 'uniquer'.

----------

## zxy

Here are some boot time measurements without caching and with caching (my uniquing).

I also cached gdm background and icons. Because of many items that must be cached I didn't find booting much faster. But when the machine is booted   :Very Happy:  the poetry begins. Everything almost "just pops up".

On athlon I use Fluxbox and on Pentium M KDE is installed.

```

             Athlon64+SATA2(Raid0)      PentiumM+IDE

             no-cache    cache          no-cache    cache

------------------------------------------------------------------

boot         34          35             55          75

Open Office  6.7         4.12           14          4.23

Firefox      3.91        2.4            6.99        3.77

```

I'm going to try the new possibility with only 1 lsof.

----------

## AssociateX

I got this from here, how about using :

```
 lsof -F | sed "s/^n//g" | grep -v "^c" | egrep "^/bin|^/lib|^/sbin|^/usr" | sort | uniq 
```

Will it matter that any of the lsof lines in this howto so far have been producing dir's? Will this actually work:

```
cat /etc > /dev/null
```

Also, how long is:

```
ps -aef|grep sample-init-process
```

supposed to show:

 *Quote:*   

> root      9724  8481  0 09:49 pts/1    00:00:00 grep --colour=auto sample-init-process

 

I let this run all night and it's still there.

Also, I checked /etc/conf.d/sofile-list.load* and get this, and this is after sample-init-process running all night:

 *Quote:*   

> athlon ~ # wc -l /etc/conf.d/sofile-list.load*
> 
>   30 /etc/conf.d/sofile-list.load
> 
>    0 /etc/conf.d/sofile-list.load0
> ...

 

Of course that doesn't seem right.

Thanks a ton. I had went out and added a gig of ram to the 512mb that I ready had and really want to play with stuff like this.

----------

## lazarusrat

 *zxy wrote:*   

> 
> 
> My digging and surfing results:
> 
> ```
> ...

 

This sed script has rapidly diminishing speed returns depending on the size of the file. Taking samples from a random log file I had lying around, at 100 lines:

```
lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt

real    0m0.056s

user    0m0.054s

sys     0m0.002s

lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt                          

real    0m0.003s

user    0m0.002s

sys     0m0.002s
```

At 200 lines:

```
lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt

real    0m0.161s

user    0m0.160s

sys     0m0.001s

lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt                          

real    0m0.003s

user    0m0.000s

sys     0m0.003s
```

And at 1000 lines:

```
lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' bar.txt > sed.txt

real    0m7.739s

user    0m7.460s

sys     0m0.280s

lazarus@massacre ~ $ time sort bar.txt | uniq > su.txt                          

real    0m0.005s

user    0m0.001s

sys     0m0.004s
```

Sort and uniq are not exactly slow (and certainly a lot more readable  :Wink: ).

----------

## devsk

 *AssociateX wrote:*   

> 
> 
> Also, how long is:
> 
> ```
> ...

 use 

```
ps -aef|grep -v grep|grep sample-init-process
```

 instead. You are just seeing the process which is grepping. The actual sample-init-process is long gone.

----------

## devsk

 *lazarusrat wrote:*   

> 
> 
> And at 1000 lines:
> 
> ```
> ...

 this can be a problem because sample interval should be definitely smaller than 8 seconds. Otherwise you won't catch many useful files and hence the results that you see (wherein cache boot is longer). I think the another way would be to do this 'sedding' only on the final output and/or do 'sedding' in the while loop every 3 or so iterations. Sample collector phase can be slow overall, but not when its taking the samples.

----------

## lazarusrat

 *devsk wrote:*   

>  *lazarusrat wrote:*   
> 
> And at 1000 lines:
> 
> ```
> ...

 

Well, these times should probably be taken with a grain of salt. The tests were done with a text file with massively varying line lengths and contents, probably nothing close to the format of what's actually being uniqued by these scripts. Is it likely that what's being uniqued will have 1000 lines or more?

Edit: Even still, using a different file (ls /usr/lib /lib > foo.txt) with 1300 lines, sort | uniq is faster than the sed script:

```
lazarus@massacre ~ $ time sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' foo.txt > sed.txt

real    0m0.807s

user    0m0.802s

sys     0m0.006s

lazarus@massacre ~ $ time sort foo.txt | uniq > su.txt                          

real    0m0.003s

user    0m0.000s

sys     0m0.005s
```

----------

## devsk

the idea was to not sort at all because he wanted  to preserve the order of loading of files to be same as the order that lsof found when sampling. And he has a point. But what do we pay for it? If useful results are lost while uniquing, then no point.

----------

## AssociateX

 *devsk wrote:*   

> use 
> 
> ```
> ps -aef|grep -v grep|grep sample-init-process
> ```
> ...

 

Ah man.....dang it for not seeing that. Thank you.

----------

## zxy

I'm just experimenting with uniquing only at the end. I think it will sample much better.

Though the final sed-ing takes time (athlon64 - 7 min and counting)

--- EDIT

sed-ing lasted for 10 mins ... going to reboot now...  :Question: 

----------

## lazarusrat

Ah, I misunderstood. Here is a little awk script that is much faster than the sed script and does the same thing (stored in file ~/neek for this example):

```
#!/bin/sh

/usr/bin/awk '{

if ($0 in stored_lines)

        x=1

else

        print

        stored_lines[$0]=1

}' $1

```

```
lazarus@massacre ~ $ wc -l foo.txt

41207 foo.txt

lazarus@massacre ~ $ ls -l foo.txt

-rw-------  1 lazarus users 2987383 Aug 18 09:56 foo.txt

lazarus@massacre ~ $ time ~/neek foo.txt > awk.txt

real    0m0.083s

user    0m0.068s

sys     0m0.014s

lazarus@massacre ~ $
```

If I weren't so rusty with awk I would attempt to get that into a single script file that could just be made executable.  :Smile: 

Edit: Just made it a shell script that invokes awk instead of a flat awk script.

----------

## AssociateX

OK, I have this now:

```
athlon ~ # wc -l /etc/conf.d/sofile-list.load

554 /etc/conf.d/sofile-list.load
```

Yippy!

Although my system stops during boot at:

 *Quote:*   

> Starting syslog-ng ...

 

But it continues if I hit the space-bar. That is minor but would be nice to know whats causing it.

Also, is it true that I need to open the apps that I want to be cached (if that's the term) before this gets done:

```
while true; do ps -aef|grep -v grep|grep sample-init-process; sleep 1; echo done; done
```

Or is this supposed to update at some point else also?

Running this with the 554 entries:

```
cat `cat /etc/conf.d/sofile-list.load` > /dev/null
```

gives back:

 *Quote:*   

> cat: /home/me/.config/menus: Is a directory
> 
> cat: /home/me/.kde/share/apps/kconf_update: Is a directory
> 
> cat: /tmp/gconfd-me/lock/0t1155919871ut751148u1000p13434r1702137469k3213382076: No such file or directory
> ...

 

Will that matter?

----------

## devsk

 *AssociateX wrote:*   

> 
> 
>  *Quote:*   
> 
> cat: /usr/share/apps/kconf_update: Is a directory 
> ...

 nope. Ca't cat a directory. that's exactly why the script does 2>&1.... :Smile: 

----------

## devsk

 *lazarusrat wrote:*   

> 
> 
> lazarus@massacre ~ $ time ~/neek foo.txt > awk.txt
> 
> real    0m0.083s
> ...

 

that's fast.

 *lazarusrat wrote:*   

> 
> 
> Edit: Just made it a shell script that invokes awk instead of a flat awk script.

 luckily, we need it as a script and not  as an executable.

----------

## zxy

I just included the awk script lazarusrat gave (thank you very much)

It's quick.

I have put sleep to 0.05 and  total_samples_per_boot to 800. So the sampling is accurate. Maybe I will even throw sleep out. 

Results are cool.

I think that it is goot to include login managers icons and background in right place in the list of files too, because while computer caches for example firefox or OO it is blockig access to this images and login manager is waiting for them.

What do you think, which files should be in the list too?

-----------------------

P.S. I turned of chrootkit,  it loads a ton of files. Maybe now I can put it back.

----------

## devsk

I just modified the original post with the discussions so far.

----------

## zxy

ROYALE   :Cool: 

Very good!

--- EDIT ---

You should add

```
 rm /usr/bin/uniquer
```

to the uninstall instructions.

----------

## AssociateX

I think I have it working so I thought I would stop back by with some numbers.

I changed the samples from 200 to 400 since you went from sleep 1 to .5 just so I could get enough stuff open in time.

I opened:

kinfocenter

kcontrol

Enemy Territory

Enemy Territory True Combat

The Gimp

Amsn

Mozilla

Mozilla -mail

gFTP

konqueror

xchat

grip

k3b

xmms

oowriter2

kwrite

ksensors

konsole

lshw

top

ls

I checked the word count when sample-init-process was done:

 *Quote:*   

> athlon ~ # wc -l /etc/conf.d/sofile-list.load*
> 
>  1097 /etc/conf.d/sofile-list.load
> 
>     0 /etc/conf.d/sofile-list.load0
> ...

 

That's about double what I had before... which is fine for me, I want this fast.

On the reboot I checked the memory:

 *Quote:*   

> me@athlon ~ $ top | grep -i mem
> 
> Mem:   1555020k total,  1120000k used,   435020k free,    15808k buffers

 

I can now start and close mozilla this fast:

 *Quote:*   

> me@athlon ~ $ time mozilla
> 
> No running windows found
> 
> real    0m2.190s

 

And for OpenOffice Writer to open then close:

 *Quote:*   

> me@athlon ~ $ time oowriter2
> 
> real    0m3.977s
> 
> user    0m2.684s
> ...

 

The machine is:

product: A7N8X-X

vendor: ASUSTeK Computer INC.

product: AMD Athlon(tm) XP 2800+

*-memory 1536MB

Thank you devsk for giving my extra gig of ram something to do. This is fun.

----------

## zxy

Check this post out.

https://forums.gentoo.org/viewtopic-t-463204-start-0-postdays-0-postorder-asc-highlight-.html

It's a cool defragmenter, though I didn't run it yet, but it looks very promising. It defragments in a very cool way, so it should be a fantastic combination of caching boot and defragmenting the files we need for startup. Then the booting should be done in minimal time. 

I can't wait to try the combination. Will post the results.

----------

## wpoely86

I have a problem with the script: it dies and i don't know why.

I have added echo lines to the script and so i found out it does run and gets into

the sampler loop. But then it dies after +/- 20 loops on the rule "sleep 0.5"

I don't get any error message or so. The moment the script dies is more or less at the same 

time with the init scripts end. Anyone got a clue why it dies?

If i run it while the pc is already fully booted, it doesn't die.

----------

## dberkholz

Could you integrate this into sys-apps/readahead-list and drop a patch to the author?

----------

## devsk

I did file a bug to include this in the readahead-list and start some useful discussions but it was quickly closed as a duplicate of another bug, which is almost dead. devs closing enhancements fast is a major putoff... :Smile:  And if you try to reopen, they threaten to ban you... :Smile:  So, I didn't press much this time.

https://bugs.gentoo.org/show_bug.cgi?id=142338

I have no interest in seeing it going into readahead-list anymore. Its not such a big deal to get a few seconds faster boot up or app startup. It was just an idea that occurred to me and I tried it out. If someone feels that it absolutely rocks and should be integrated, please raise your hand and volunteer to put it there... :Wink: 

----------

## gerardo

Shouldn't the my-readahead script be placed in the boot runlevel instead of the default?

This question was asked before, but I haven't seen an answer to it

----------

## albright

I've followed the instructions in the first post but this

does not work for me.

lsof *is* installed (honest and truly).

When I boot two files are created in /etc/conf.d

They are sofile-list.load and sofile-list.load0

They are both empty. The sample-init-process script

seems to run only for a split second ... It does not show

up in ps even immediately after boot.

If I try to run the sample-init-process on its own from a konsole,

it craps out with this error:

/usr/sbin/sample-init-process: line 63: : No such file or directory

That may not be relevant.

Anyway, any advice would be appreciated.

----------

## maximan

I'm using the first script. Is there any change or update of this script ??

M.

----------

## gerardo

 *albright wrote:*   

> I've followed the instructions in the first post but this
> 
> does not work for me.
> 
> lsof *is* installed (honest and truly).
> ...

 

Just to be sure, have you copied everything exactly? Attention for linebreaks...

Have you made the uniquer script executable ?

Check that you have /usr/bin/awk

If not, emerge sys-apps/gawk

----------

## gerardo

 *maximan wrote:*   

> I'm using the first script. Is there any change or update of this script ??
> 
> M.

 It would be great to have it in portage or perhaps in an overlay...

But the dev doesn't like it apparently, as the bug was closed immediatly...

I'm afraid we'll have to check this post regularly  :Rolling Eyes: 

----------

## albright

 *Quote:*   

> Just to be sure, have you copied everything exactly? Attention for linebreaks... 
> 
>  Have you made the uniquer script executable ? 
> 
>  Check that you have /usr/bin/awk 
> ...

 

Thanks for that but that's all ok ... can't imagine what's wrong

----------

## trevorj

 *gerardo wrote:*   

> It would be great to have it in portage or perhaps in an overlay...
> 
> But the dev doesn't like it apparently, as the bug was closed immediatly...
> 
> I'm afraid we'll have to check this post regularly 

 

It's not that the dev didn't like it, he just marked it as a dupe of another bug; Another way of saying 'Hey, this was here before! Continue on in the original bug so we don't have to go back and forth'

This set of scripts reminds me of preload actually. http://preload.sf.net if anyone's interested.

I think readahead-list should be used instead of catting the files

----------

## nowinter

same as allbright, got a couple of files /etc/conf.d/sofile-list.load and /etc/conf.d/sofile-list.load0.. seems that there would be more as the script runs.. no doubt it runs, i see it in processes, as well as plenty of its greps, lsofs and a sed process. but the files are empty, even after many hours. i tried to rm them, touch /forcesampler and re-done all the howto steps actuallly. nada.

----------

## nowinter

although there are files in /tmp, i mean init_sample1 and 2, full of file names.. why don't they get to /etc/conf.d/sofile-list.load ?

----------

## nowinter

disregard two previous posts of mine, after app. 6 hours there is a bunch of content in /etc/conf.d/sofile-list.load. 

Although it consists in a big part of just regular files, apparently brought there by amule/amarok etc. By no means they would fit in memory altogether  :Smile:   :Smile: 

Should I just drop them from the list? Or may be I need to add a grep -v option to the script so they don't pop up again... as well as changing .mozilla to .opera so the stored mails won't raise to memory..

Anyway, thanks, I think I've finally got the point of the readahead work now   :Idea:  can't wait to the next boot.

----------

## XAvAX

I was just thinking, maybe speed-daemon would be a good name for it, and we could name the sampler speed-daemon-sampler for readability. If we change uniquer to non-sorting-uniquer, that might be nice for readability too, but tell me if I'm taking it too far. That's just how I named them on my system so I wouldn't forget what they did. But because I'll probably forget anyway, I wrote this MAN page.

Save it as /usr/share/man/man1/speed-daemon.1 and then gzip it, and you're good to go! (I must apologize, I don't know how to bold in manpages - But at least I got the line wrap length right! [or at least, it matches convert's]) Feel free to correct my numbers, spelling, etc. as you see fit.

I also made one edit to the initscript, I made it invoke its own stop function before eend, and since its stop function is eend, it works out nicely (With the side benefit of a millisecond improvement in shutdown times   :Very Happy:  )

```
speed-daemon(1)

NAME

        /etc/init.d/speed-daemon  -  speed booting and application startup by preloading accessed

         files.

SYNOPSIS

        /etc/init.d/speed-daemon [start|stop]

OVERVIEW

        The speed-daemon init script functions by calling cat() to read a list of files, and then

        it invokes cat() again, to read the files referenced by that list to /dev/null,

        effectively preloading them through caching.

OPTIONS

        start   Executes the script. If /resample exists, it triggers it to rebuild its list of

                files to cache.

        stop    Normally this would stop a script, but as it completes when run, this is

                unnecessary. It is preserved so that the rc-script system doesn't get confused.

NOTES

        It is probably most useful if placed in the "boot" runlevel, as it then preloads all of

        your init process, as well as possibly [x|k|g]dm. If you wish to resample the list of

        files, simply touch() /resample and reboot. If it does not seem to work, try to open

        /etc/conf.d/sofile-list.load with your favorite pager. It should have somewhere in the

        vicinity of 200 files if you are preloading a graphical login, maybe 100 more if you

        sample the login/applications starting process as well, and maybe 100 less if you only

        preload to the console. If there seem to be unusually few files listed, try decreasing the

        "sleep" timer near the end of /usr/sbin/speed-daemon-sampler, as your computer is likely

        too fast. If, on the other hand, it seems to bog down your entire system, this is somewhat

        normal as creating the file list is rather resource-intensive. If, however, the computer

        seems to return to normal operation too early, ending before some programs you wished to

        preload ran, I will note that the sleep timer multiplied by the total number of samples

        (right at the top) is how long it will actually run. The script itself depends on lsof.

        Also, it uses the aptly named /usr/bin/non-sorting-uniquer to ensure that nothing is

        preloaded twice, while retaining them in the order they are called. It is very fast (It

        can unique a list of 41,207 words in 0m0.083 seconds, which was essential in this 

        application). It depends on awk.

CREDIT

        Credit for the initscript and sampler goes to devsk of the gentoo forums.

        Credit for the uniquer goes to lazarusrat of the same.

        This comes from the forum thread at: http://forums.gentoo.org/viewtopic-t-478491.html
```

----------

