# kernel performance is getting lower

## gigel

kernel 3.x.x it's been available for quite some time, but in terms of performance and usability (on a laptop) I would say that it belongs to beta RC.

besides some random wlan disconnections I would say that my vanilla 2.6.38.8 kernel is quite good, while the newer versions are much slower than the previous mentioned kernel - no wlan disconnections but total unusable usb mouse, mobile phone modem and webcam.

for example, in terms of booting time

2.6.38.8 takes 13-14 seconds to boot until the login prompt, 35-40 seconds to fully start kdm, and 45-50 second to fully load KDE. 

all the 3.x.x kernels are way slower with the same options enabled

20 seconds til login prompt, 55-60 til kdm, and 75-80 to fully load KDE. 

maybe this thread does not belong here, so feel free to move it to the proper place, but I'm curious if other users have experienced this performance losses. 

I know that booting times does not tell anything about actual performance, but I don't know any reliable benchmark that tests things properly, so this is the only objective measure I found so far, besides the subjective opinion that 2.6.38 feels snappier. 

excuse the  bad English, please

----------

## albright

can't say I've noticed any slowdown

For what it is worth (not much), my thinkpad x300 with 3.1.1 pf-sources kernel

boots to kde desktop (auto-login) in 33 seconds from cold boot. And, of course,

kde is a real pig ...

----------

## gigel

thanx for your input. I guess that maybe the slowdown is related to the scheduler and the preemption model (I use for both kernel the voluntary kernel preemption). If I select the preemptible kernel then the bootup process increases by 50% roughly, and the overall performance is actually worse(yes, I don't get the random audacious music interrupts, but those don't occur often)

I guess I'm gonna give this 3.1.1-pf-kernel a try, and move to the con's BFS. 3.1.3 breaks so many things on my system...

----------

## dE_logics

You did oldconfig to configure the new kernel?

Also I haven't noticed any difference.

----------

## gigel

 *dE_logics wrote:*   

> You did oldconfig to configure the new kernel?
> 
> Also I haven't noticed any difference.

 

of course

I tried the pf sources, and changed to BFS scheduler - still behind 2.6.38.8 in terms of booting time but not by much. 

but, 3.1.1 and above breaks things, unusable USB mouse, not working usb camera, mobile phone as modem, etc...

the main drawback in performance is seen until kdm + kde starts completely, where 10-20 seconds in startup time makes a difference. after this, performance is more or less the same(I would say that it is still better in 2.6.38.8 but I could be just subjective with that)

I have 2 Gb of ram, but if chromium is eating at a moment a ton of memory and I try to compile something in between, the system becomes unresponsive and I must press the power button. with 2.6.* I can still spawn a new console and kill chromium. I didn't recall having these problems back in those days when I compiled gentoo on my old 850 mhz amd duron.

----------

## dE_logics

I think you should try configuring the kernel freshly. Also, what is the version of linux-headers? I use custom Linux headers to avoid version mismatch.

Also try recompiling all X drivers and X itself; this's cause I think this may be a Linux headers issue.

----------

## gigel

 *dE_logics wrote:*   

> I think you should try configuring the kernel freshly. Also, what is the version of linux-headers? I use custom Linux headers to avoid version mismatch.
> 
> Also try recompiling all X drivers and X itself; this's cause I think this may be a Linux headers issue.

 

I would say that X is pretty much snappy. The KDE startup takes more than half of the booting time. 

I have the 2.6.39 linux headers, but is that the one used during compilation of the newer kernel, as I really doubt it. ?

Even if I'm configuring a fresh kernel I'll end up having the same .config file, so there is no point in doing so.

After some tuning I would say that 3.1.1-pf, with the BFS scheduler gives almost the same performance as 2.6.38.8 with the CFQ scheduler, they both boot in almost 1 minute, and behave more or less the same. 

I'll install a 3.1.1 version of linux headers and then recompile the pf sources once more.

----------

## dE_logics

 *mortix wrote:*   

>  *dE_logics wrote:*   I think you should try configuring the kernel freshly. Also, what is the version of linux-headers? I use custom Linux headers to avoid version mismatch.
> 
> Also try recompiling all X drivers and X itself; this's cause I think this may be a Linux headers issue. 
> 
> I would say that X is pretty much snappy. The KDE startup takes more than half of the booting time. 
> ...

 

Linux headers is used by a lot of other packages, it provides an API for userspace apps (?), including DRM, thus is needed for xf86-video-ati (?).

equery depends linux-headers

 * These packages depend on linux-headers:

app-emulation/wine-1.3.32 (>=sys-kernel/linux-headers-2.6)

media-video/ffmpeg-9999 (v4l ? sys-kernel/linux-headers)

sys-libs/glibc-2.12.2 (>=sys-kernel/linux-headers-2.6.9)

virtual/os-headers-0 (!prefix ? sys-kernel/linux-headers:0)

x11-drivers/xf86-input-evdev-9999 (>=sys-kernel/linux-headers-2.6)

But unfortunately xf86-video-ati doesn't depend on it.  :Sad:  But compiling package without linux-headers (and injecting it to portage) results in most packages failing to compile. This suggests it's a very basic requirement for many packages to build.

Also making oldconfig may give unexpected results, thus I suggest configuring it freshly. The configurations are not the same.

----------

## gigel

 *dE_logics wrote:*   

> 
> 
> Linux headers is used by a lot of other packages, it provides an API for userspace apps (?), including DRM, thus is needed for xf86-video-ati (?).
> 
> equery depends linux-headers 
> ...

 

thank you for pointing that out.

as from 3.1.1 and above usb mouse it rather doesn't work than in does work. so installing a newer version of linux headers might fix things + re-emerge of evdev. 

 *Quote:*   

> 
> 
> Also making oldconfig may give unexpected results, thus I suggest configuring it freshly. The configurations are not the same.

 

after an oldconfig, I alwas give a rough look trough make menuconfig, + diff-ing to the older version so I will not omit some stuff.

----------

## wcg

Try

```

equery depends virtual/os-headers

```

as well as searching direct dependencies on linux-headers.

(Some packages depend on them but use the virtual package

for the dependency name.)

----------

## gigel

 *wcg wrote:*   

> Try
> 
> ```
> 
> equery depends virtual/os-headers
> ...

 

unfortunately latest version of linux-headers available in portage is 2.6.39  :Very Happy: 

----------

## dE_logics

[quote="gigel"] *dE_logics wrote:*   

> 
> 
> after an oldconfig, I alwas give a rough look trough make menuconfig, + diff-ing to the older version so I will not omit some stuff.

 

May things get added unknowingly, which may cause dependency issues. That's why I made a new kernel for 3 series.

 *Quote:*   

> unfortunately latest version of linux-headers available in portage is 2.6.39

 

I use custom headers using make headers_install in the kernel source, then copying/linking fomr usr/include to /usr/include.

This requires an unmerge of linux-headers package and it's injection using package.provided. That way I get headers version exactly equal to the kernel, i.e. just the perfect API.

A lot of people here complained that installing old headers causes issues, although officially, it's not meant to happen.

----------

## gigel

 *dE_logics wrote:*   

> 
> 
> This requires an unmerge of linux-headers package and it's injection using package.provided. That way I get headers version exactly equal to the kernel, i.e. just the perfect API.
> 
> A lot of people here complained that installing old headers causes issues, although officially, it's not meant to happen.

 

there is no need to unmerge linux-headers package firstly (it is quite not good to do this actually), since, if you remove kernel headers you can't install the new ones. I did this but fortunately I had another machines from which I could take again the binary linux-headers package.

the proper way to do this is just overwriting the files in /usr/include - which I ended up doing. 

apart from this, I still can't see why I still have 10-15 seconds behind the 2.6.38.8 kernel, in term of booting time, but I'll live with that, since now all my USB devices work again with 3.1.4 and 3.2rc3 series.

----------

## gigel

I've removed xdm from default runlevel. 

added an uptime job in local.d to indicate the uptime after every boot.

kernel 2.6.38.8 - 13 seconds

2.6.39.4 - 15 seconds

3.1.4 - 20 seconds.

7 seconds is not much, but in these terms is almost 50% worse than 2.6.38.8.

----------

## dE_logics

 *gigel wrote:*   

>  *dE_logics wrote:*   
> 
> This requires an unmerge of linux-headers package and it's injection using package.provided. That way I get headers version exactly equal to the kernel, i.e. just the perfect API.
> 
> A lot of people here complained that installing old headers causes issues, although officially, it's not meant to happen. 
> ...

 

I had run make headers_install first before unmerging linux-headers; for safety, I made a package off the linux-headers package.

Is KDE still giving an issue?

----------

## gigel

 *dE_logics wrote:*   

> 
> 
> I had run make headers_install first before unmerging linux-headers; for safety, I made a package off the linux-headers package.
> 
> Is KDE still giving an issue?

 

this is strange, because you overwrite the linux headers and than unmerge them? 

but I'm guessing portage woun't remove files which were modified. 

kde is still slow (read: starts slower than with a 2.6. kernel  - cause otherwise, slowness canot be so easily be reproduced, but it occurs when the system is under heavy load - compiling, browsing, etc..)

to me, it's quite surprising that the newer 3.x kernel are slower by 50% than the 2.6 ones, only regarding the console stuff. 

maybe it's a disk scheduler thing, because I would say that during boot alot of I/O is performed.

----------

## dE_logics

 *gigel wrote:*   

>  *dE_logics wrote:*   
> 
> I had run make headers_install first before unmerging linux-headers; for safety, I made a package off the linux-headers package.
> 
> Is KDE still giving an issue? 
> ...

 

When you run make headers_install, it installs the headers to usr/include of the kernel source tree not from the root FS. Then you have to copy over the headers to /usr/include.

----------

## gigel

 *dE_logics wrote:*   

> 
> 
> When you run make headers_install, it installs the headers to usr/include of the kernel source tree not from the root FS. Then you have to copy over the headers to /usr/include.

 

already did that. I have the 3.1.4 kernel headers in /usr/include

now I'm using the ck sources, and on top of that I also patched the kernel to get the BFQ IO scheduler. 

now the bootup speed increased and applications start a little bit faster.

2.6.38.8 - 13 seconds to console, whereas the above 3.1.4-ck+bfq boots in 17 ( to fully load kde it takes 65-70 seconds) - the best performance from a 3.1. kernel I've achieved so far. 

I really don't think I'm the only one experiencing slow down with this kernel...

----------

## ulenrich

New versions of linux-headers usally do additions to kernel-abi, but ecxeption:v4l2. And these additions are regarded, when a new glibc version is released. So I doubt there is a performance issue in having a slightly older kernel-headers package installed.

The importance of new headers is for new features: kvm,xen!

I don't regret a ten seconds more boot time of my pc, if it is more reliable than before:

Look at all the issues have been solved since 2.6.38  :Smile: 

And I also do have better experience using Con Kolivas BFS scheduler. But I don't need his mm-patches from the ck-patchset. And I don't trust BFQ for patching IO. This is just too complicated for simple out of main trunc patches  :Sad: 

----------

## gigel

 *ulenrich wrote:*   

> 
> 
> Look at all the issues have been solved since 2.6.38 
> 
> 

 

I agree, the random wlan disconnects which I'm experiencing with this kernel is rather frustrating. but this doesn't happen often though.

 *Quote:*   

> 
> 
> And I also do have better experience using Con Kolivas BFS scheduler. But I don't need his mm-patches from the ck-patchset. And I don't trust BFQ for patching IO. This is just too complicated for simple out of main trunc patches 

 

can you give one reason for not trusting the BFQ scheduler?

for me, every app starts faster (firefox, chromium, etc..)

----------

## dE_logics

BFQ? Patch please... For 3.0.4 preferably.

----------

## gigel

 *dE_logics wrote:*   

> BFQ? Patch please... For 3.0.4 preferably.

 

http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php

You can grab the source for 3.0.0 kernel. 

I'm still waiting for ulenrich to give some comments about  why BFQ is not trustworthy. Looking at the tests it seems that BFQ performs better than CFQ.

----------

## asturm

Forget the stuff about linux-headers, that's got nothing to do with your boot performance at all.

Activating rc_parallel in your rc.conf might provide you with a significant decrease in bootup time if you haven't already.

Anyway, I would rather assume some config problem with your system. Nothing of that kind observed here. Have you looked at your dmesg output recently?

----------

## gigel

 *genstorm wrote:*   

> Forget the stuff about linux-headers, that's got nothing to do with your boot performance at all.
> 
> Activating rc_parallel in your rc.conf might provide you with a significant decrease in bootup time if you haven't already.
> 
> Anyway, I would rather assume some config problem with your system. Nothing of that kind observed here. Have you looked at your dmesg output recently?

 

I'm assuring you that there are no problems with kernel config, or whatever other config. 

I looked and relooked at the menuconfig and .config file several times. I have some errors in 2.6.38.8(this is the fast kernel) that look like this

```

[Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored

[Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored

[Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored

....

and so on for like 300 lines

```

while in 3.1.4-ck + bfq the line appears only once

from time to time I also get the following error 

```

[drm:ironlake_update_pch_refclk] *ERROR* enabling SSC on PCH
```

but speed is not affected by that

about the parallel startup, I've disabled it, since it's not that safe. but I'll enable it and test how it performs

----------

## asturm

A full dmesg log should be interesting to see.

 *gigel wrote:*   

> about the parallel startup, I've disabled it, since it's not that safe. but I'll enable it and test how it performs

 

In the two years I've been using it I never ever had rc_parallel issues except for sameday-emerging new ~arch openrc releases (made me plug in my usb boot stick twice so far).

For the record, I had stuttering sound with ck-patchset and questionable - I'd say esotheric - responsiveness improvements back in 2.6.3something which is why I'm not using it - using rc_parallel is definitely not less safe than ck.

----------

## gigel

so far, fastest 3.x.x kernel - ck-sources + bfq IO scheduler (boots 10 seconds faster than vanilla sources with same options enabled)

http://pastebin.com/73NApApr

good old 2.6.38.8

http://pastebin.com/kc6hDwW9

parallel startup gives no improvement at all. 17-18 seconds no parallel, 18 seconds with parallel enabled. 

and after that, kdm+kde takes till 65-70 seconds. 

I know it's not much, but it is so strange that with that old kernel the system boots up and feels faster. 

with bfq scheduler I really see that chromium and firefox start faster.  I still didn't test the system under load, but I don't usually stress the system too much

----------

## asturm

Interesting facts. Also, KDE needs exceptionally long time to start on your system imo.

Please set CONFIG_PRINTK_TIME=y in section 'Kernel Hacking' for meaningful output and paste again. Only then can we identify possible kernel regressions at some point(s). Right now I can only tell that your old kernel makes a lot more noise than the 3.1.4 one.

PS: I'm also on CONFIG_PREEMPT_VOLUNTARY=y (CONFIG_DEFAULT_IOSCHED="cfq") and there haven't been any huge changes in 3.0. Would be interesting to see how sys-kernel/vanilla-sources-2.6.39.4 performs on your system.

----------

## gigel

 *genstorm wrote:*   

> Interesting facts. Also, KDE needs exceptionally long time to start on your system imo.
> 
> Please set CONFIG_PRINTK_TIME=y in section 'Kernel Hacking' for meaningful output and paste again. Only then can we identify possible kernel regressions at some point(s). Right now I can only tell that your old kernel makes a lot more noise than the 3.1.4 one.
> 
> PS: I'm also on CONFIG_PREEMPT_VOLUNTARY=y (CONFIG_DEFAULT_IOSCHED="cfq") and there haven't been any huge changes in 3.0. Would be interesting to see how sys-kernel/vanilla-sources-2.6.39.4 performs on your system.

 

I know the 2.6.38 is more noisy, but somehow it is faster. setting the voluntary preemption to Y will increase the boot time , cause the kernel is trying to figure wich application that is being started is more important, so it checks them all but it doesnt give an improvement in the end. 

but I will reconfig both kernel with the same options. I can say that my kde starts as fast as on my other laptop which has a 1.66 ghz CPU, but the same 5400 rpm HDD. So you simply cannot squeeze more performance out of these disks. 

to the command prompt on the 1.66 ghz laptop the system boots in 36 seconds, whereas in the 2ghz one in half of that. 

but the 2.6.38 is damn fast 13 seconds. Maybe this kernel has other bugs (I mentioned one before) but other than that I didn't experienced any problems. 

thanx again all of you for your time. I'll post the updated dmesg during the day.

----------

## f4c3m3l70r

How do I log my startup screen?

Ive been also playing around with gentoo & pf sources and would like to post some results.

----------

## gigel

314ck+bfq

http://pastebin.com/4Yyayky9

2.6.38.8

http://pastebin.com/vaR7khFv

relevant differences

[   27.813482] ata1.00: configured for UDMA/133

[   27.813491] ata1: EH complete

[   33.006298] ata1.00: configured for UDMA/133

[   33.006303] ata1: EH complete

so, newer kernel is 5 seconds slower. and that adds to several seconds more in terms of kde startup

@f4c3m3l70r, I have no ideea how to do that. I've just put an uptime script in local.d that prints the uptime before the usual login prompt.

but feel free to post your results here

----------

## asturm

That's quite a notable regression. What other kernels have you tried so far?

----------

## gigel

 *genstorm wrote:*   

> That's quite a notable regression. What other kernels have you tried so far?
> 
> Could you post some more lines around those ata1 messages? Anything regarding some 'link is slow to respond' stuff?

 

nothing like that. the full dmesg are on pastebin. 

if I don't start kde the ata messages don't show up, but I'm not sure about this. 

dmesg2638:[    0.387909] TCP: Hash tables configured (established 262144 bind 65536)

dmesg2638:[   11.839376] ip_tables: (C) 2000-2006 Netfilter Core Team

dmesg314:[    0.400037] TCP: Hash tables configured (established 262144 bind 65536)

dmesg314:[   15.942741] ip_tables: (C) 2000-2006 Netfilter Core Team

so it seems that almost everything starts slower in the newer kernel. I'll also post the results with vanilla. to make a proper comparison.

----------

