# thrashing with ck-sources r5

## puntium

Hi, I'm trying out the new ck-sources r5 based on the ck5 patchset.

I tried it with some settings that I thought were pretty default for the patch.. 500HZ, preempt, low latency all on.. (I also have xfs and reiserfs among other things) When I first booted, seemed pretty good, but then when I got mozilla and a few other programs going, with some compilation in the background.. things started to really go downhill..

After a while, could barely get it to respond. Trying to kill X resulted in a hard freeze. (using nvidia 4349 drivers)

Went and checked on the homepage for the patchset, and saw a mention of AA's vm extensions.. Wondering if these could be the culprit, I also applied the rmap patch found on this site, which rips out the AA patch's changes.

When I tried to compile, I got the __br_lock_usage_bug() (from mm.o) error so I had to disable the kernel preemption. Then it compiled fine and things seem to be working pretty well.

Got a compile going in the background right nowand seems to be holding up.

Also had the __br_lock_usage_bug() problem with the xfs-r2 kernel that I was running before. Had to disable the option there too.

has anyone else had any excessive thrashing with the ck5 kernel?

(i just noticed, I had the "enable low latency with sysctl" setting on.. which means the low latency modification wasn't taking effect last time I ran it.. but should this change how the VM works, should it?)

Ken

----------

## dbezona

Well, glad to see I am not alone!

Seems like a memory leak - watching GKrellm, I could see it was simply using more and more RAM, and never releasing it, just digging deeper into swap as everything slowed down to a crawl.

I just reverted back to my working ck4 kernel, so I didn't spend any real time troubleshooting.

----------

## puntium

If you go ahead and apply this patch at

http://members.optusnet.com.au/ckolivas/kernel/rmap15e_030410_ck5_2.4.20.patch.bz2

and turn off kernel preemption then all seems to work well.

Maybe its not hte ideal solution, but the interactivity update is well worth it for me.

----------

## swat

 *puntium wrote:*   

> If you go ahead and apply this patch at
> 
> http://members.optusnet.com.au/ckolivas/kernel/rmap15e_030410_ck5_2.4.20.patch.bz2
> 
> and turn off kernel preemption then all seems to work well.
> ...

 

I thought pre-empt was better? On my box, i'm not experiencing thrashing but rather xmms starts cutting out when any activity happens.

Simon

----------

## dizzogg

I've been experiencing the same thrashing.  I was compiling something and browising the web, and it just became unresponsive.  I guess I'll try the VM patch, although I'm skeptical of not using preempt.... But I love the ck kernel, so I'll give anything a try.  I'll post back once I'm running it.

----------

## puntium

Actually, I'm not recommending the disabling of preemption for any other reason than that on my box, it makes "make bzImage" fail, with an undefined reference to __br_lock_usage_bug() which is apparently some kind of deliberate failure when something doesn't get called with a correct parameter.

The only way I could get to compile at all was to unset the kernel preemption.. but  maybe this only happens on my config.

----------

## puntium

Reporting back..

so it seems like the rmap patch doesn't seem to fix things at all.. maybe only delays the matter.. after running the machine for 2 days of light use.. it started to thrash all over the place again. "top" showed 6 megs of free physical memory.

xosview showed that about 490/512mb was "used"

I looked at ps aux output, and the numbers don't seem to add up. I could roughly sum up maybe 100M being used by the processes shown by ps, but top still reported 400M+ usage..

Just now switched back ot the xfs kernel.. top shows 200M free.. though I've been only using it for a few minutes as of yet

xosview shows about 1/3 of the memory being used is file cache.. 

so i think there is something definitely funky going on with ck-sources r5.. somehow doesn't give all the memory back when processes exit.. 

Hope someone figures out the problem  :Wink:  Maybe I'll write con kolivas directly..

----------

## Lovechild

Please try out an mm-sources kernel instead.. -ck is just to buggy

----------

## puntium

Some news from Con Kolivas himself.

I reported the strange behaviour to him directly. He wrote me back saying he had not seen anything, but then soon after, he got another message reporting similar symptoms.

Seems like the common characteristic among people who have hte problem is the enabling of XFS. Are the people posting on this thread all using XFS? 

Can anyone try disabling XFS? I unfortunately won't be ablt to to do this as on e of my key partitions is in xfs..

----------

## piquadrat

I have also problems with ck5. I tried to emerge the kde-3.1.1a patches, but sometimes in the middle of the compiling, the system gets totally unresponsive. I went back to ck4, problems gone  :Smile: 

BTW: I've enabled xfs support

----------

## lurid

This reminds me a lot of what happened when I tried the wolk kernel.  Massive amounts of disk thrashing, extremely slow desktop when compiling.  I'm not sure what the problem was.  I recompiled it 4 times with different options each time (to see if it was my fault) and it never got better.  I'm assuming the ck5 kernel probably has similar (if not the same) patches that wolk has.  In my experience ck4 is the best kernel right now.  I get no slow downs at all, even when emerging large packages.

----------

## swat

I have problems - not unlike the ones you are having - basically XMMS skips when I do anything in galeon!

 :Sad: 

Simon

----------

## Vancouverite

I am also switching back to CK4. Responsivness is great but I just had my first xmms skip in a very long time. Also my memory just keeps getting gobbled up, I'll be swapping soon. Not a big deal because CK4 is a dynamite kernel.

----------

## zeb

I have serious memory problems with this kernel as well. I've compiled kde twice with ck4 without any problems, but compiling kde-3.1.1a with ck5 is impossible... The compilation chokes with massive trashing after about an hour (this is with 832 MB ram).

Too bad, as the responsiveness of this kernel was fantastic  :Sad: 

Oh well, back to ck4, which was rock solid.

----------

## swat

IF you look at CKs site, you can see he has reconised the problems with the patch. I have just compiled a kernel using all the seperat patches+vanilla sources and it works like a dream  :Smile: 

Simnon

----------

## Megaptera

 *Quote:*   

> 
> 
> Can anyone try disabling XFS? I unfortunately won't be ablt to to do this as on e of my key partitions is in xfs 

 

I don't have XFS enabled and I had the same problems other people are describing with ck5.  Massive slowdown when compiling anything and sound would skip, even from loading a web page.  CK4 works like a charm, though.

----------

## asimon

 *Lovechild wrote:*   

> Please try out an mm-sources kernel instead.. -ck is just to buggy

 

Do you think the mm-kernel is not full of errors? mm-2.5.66-r3 corrupted four of my LVM volumes and leaved them in such a sad state that I needed to restore a full backup. Gernerally, I think it is not very reasonable to recommend unstable software to other people at all. People should get it straight that using development kernels could destroy data, not to mention the wasted time and money.

----------

## Zalator

Same probelm; using ck4 kernel again...

Not everybody wants to use development kernels, personally I want the best performance and stability combination, and the ck kernel set provides this for me. Using development kernels, while maybe having cool features, will give me unnecessary stability difficulties.

Incedentally,

Does anybody know a fix?

----------

## ferringb

I haven't confirmed this, but the ck5 source probably is just using the ck all-in-one patch.  I've had the exact problems as described in this thread.

What I have gotten working though, is to use the individual patches available at http://members.optusnet.com.au/ckolivas/kernel/ against a vanilla kernel.

This takes care of the problem- I've yet to been able to reproduce the thrashing/crashing that the earlier ck5 source was inducing.  The worst thing I've gotten is a minor skippage w/ xmms using esd while compiling a kernel, and moving large files around (in general trying to break it).

This went away as soon as I switched to xmms-alsa, so I'd say the individual patches work quite nicely.

If I get time tonight, I'll pull down the ck5 *all-in-one* patch and try it out to see if it contains the fixes.

Also, I've noticed free memory decreasing, but it doesn't seem to be an issue- basically not sure if it's a leak or kernel/processes normal expansion.

----------

## swat

 *ferringb wrote:*   

> I haven't confirmed this, but the ck5 source probably is just using the ck all-in-one patch.  I've had the exact problems as described in this thread.
> 
> What I have gotten working though, is to use the individual patches available at http://members.optusnet.com.au/ckolivas/kernel/ against a vanilla kernel.
> 
> 

 

Yes - because the inidividual patches do not include the interactivity patch - which CK5 does. This is the patch that causes the problems as you will see if you look on CKs site.

Simon

----------

## zeb

Has anyone tried the new ck6pre-patch that is now available? This one apparently includes a new version of the interactivity patch.

 *Con Kolivas wrote:*   

> "use the preview for a newer better patch"

 

----------

## ferringb

 *swat wrote:*   

>  *ferringb wrote:*   I haven't confirmed this, but the ck5 source probably is just using the ck all-in-one patch.  I've had the exact problems as described in this thread.
> 
> What I have gotten working though, is to use the individual patches available at http://members.optusnet.com.au/ckolivas/kernel/ against a vanilla kernel.
> 
>  
> ...

 

Where is the specific 'interactivity' patch?  I was under the impression from what I'd read that the *all-in-one* patch was composed of the split patches (that's what his page says).  I'm guessing scheduler tunables/desktop tuning are what you are specifically refering to as the interactivity portion of the patch (O(1), pe, ll would fall under it also obviously).

Specific url?  At the moment I'm diffing between the all-in-one patch and the split patches to see if there is some other patch not posted on his website...

----------

## ferringb

 *zeb wrote:*   

> Has anyone tried the new ck6pre-patch that is now available? This one apparently includes a new version of the interactivity patch.
> 
>  *Con Kolivas wrote:*   "use the preview for a newer better patch" 

 

I've been running a kernel based on the split patches since about 1 am, and I haven't had any issues with it- the above posts are in regards to this kernel.

Also, after doing a quick diff between my patched kernel and ck6_pre, the only difference is some settings in regards to timeslices- hz was 100, now 200 for instance.

Either way, running ck6 now for just over an hour and everything is behaving quite nicely- I tried the usual kernel compile in the background running xmms w/ esd (and alsa) and it behaved, no thrashing/crashing, memory behaving much better then before- it's been hovering around 250mb free rather then the 4-20mb free I'd seen earlier in the night...

----------

## lurid

Yup, same here.  ckpre6 is behaving very nicely.

----------

## puntium

lurid, ferringb, are you using XFS? In a response I got from Con, he said he suspected the XFS snapshot that he used in ck5

 *Quote:*   

> Also, after doing a quick diff between my patched kernel and ck6_pre, the only difference is some settings in regards to timeslices- hz was 100, now 200 for instance. 

 

Which patched kernel are you referring to? the split patches vs pre6 all in one? I ask because it seems like just changing a config parameter sounds like a bit of a hack or a "magic number".. if there is a bad interaction with some other part of the kernel, I guess I want to know before I go running it on my main machine with a big xfs data partition.

I thought the interactivity patch was the new stuff they came up with for 2.5.65 (i think it was). there was a post on slashdot or linuxtoday or something about it. Was a long mailing list discussion about whether renicing X was the right thing to do etc. Interesting that there's no difference that shows up in your diffs tho..

----------

## lurid

I downloaded the all-in-one patch.gz from the page and applied it to a vanilla kernel.  I don't use XFS on any machine I own, so its never turned on in the kernel anyway.  I can't say if this is the problem or not..  all I can say is that ckpre6 works very well.

Speaking of renicing X, this is right off Con's webpage:

 *Con Kolivas wrote:*   

> NOTE: YOU MUST NOT RENICE X WITH THESE PATCHES. READ FAQ
> 
> ---
> 
> Renicing X? Many distributions (eg Mandrake) start X by default at a nice of -10 to make it more responsive. This is a workaround for the old scheduler limitations and the new interactivity changes make this unecessary, and will actually decrease performance with this kernel.

 

I saw this today when downloading the ckpre6 patch.  I admit that I was actually renicing X by using a wrapper that called X and automatically set its nice to -10.  I removed this and reset everything to call X how it should be, then compiled the kernel.  Everything has been peachy since.

----------

## ferringb

 *puntium wrote:*   

> lurid, ferringb, are you using XFS? In a response I got from Con, he said he suspected the XFS snapshot that he used in ck5

 

In my original playing around w/ the split patches, I was applying the xfs patch.  why?  It updated portions of the vfs.  I honestly have no clue what it was doing, but I figured it was backported updates to the vfs code (finer spinlocks for instance).

 *puntium wrote:*   

> Which patched kernel are you referring to? the split patches vs pre6 all in one? I ask because it seems like just changing a config parameter sounds like a bit of a hack or a "magic number".. if there is a bad interaction with some other part of the kernel, I guess I want to know before I go running it on my main machine with a big xfs data partition.

 

I started w/ a standard 2.4.20 kernel originally pulled from kernel.org- I would presume it's exactly the same as the vanilla-source, but being thorough...  in the last 24 hours I've ran both a split patched kernel, and the pre6 patched kernel (currently running pre6).  I think one thing to note, is that by the time I started in w/ the split patches the Makefile was claiming it was ck6 from the patches.

Basically, I think about the time I started on the split patches, whatever issue that ck5 had was resolved in the splits, cause I had none of the ck5 'all-in-one' patches problems.

As for changing a configure parameter (specifically killing preemption according to swat) to fix the problem, I agree.  It seems as though preemption brought out some very nasty behavior with the ck5 patch.  I was running w/ the same config for the split kernel and preck6- preemption enabled, low-latency enabled, O(1) enabled, ll via sysctl disabled, and xfs disabled (I don't use it).  XFS's behaviour under the preck6 patch I can't tell you- I don't run XFS (somebody else care to answer that question?).

 *puntium wrote:*   

> I thought the interactivity patch was the new stuff they came up with for 2.5.65 (i think it was). there was a post on slashdot or linuxtoday or something about it. Was a long mailing list discussion about whether renicing X was the right thing to do etc.

 

The interactivity patch was a backport of 65 features, off the top of my head (and not authorative), preemption, O(1) scheduler, low-latency, vm addons, tunable hz/scheduler and what is called desktop tuning.  Note I haven't yet gone through to see what the desktop-tuning actually modifies- I assume it tweaks the default values of the scheduler/preemption/hz, but again, I haven't looked.

Going by ck's statements on his website, renicing X is a bad thing under the new patches, and since the last time I tried it was w/ my previous kernel patched with ac1, preempt, and O(1) kernel- it didn't seem to help, so I stopped renicing it.

 *puntium wrote:*   

> Interesting that there's no difference that shows up in your diffs tho..

 

Note those diffs were a comparison between the split and preck6 at around 7-8am this morning... I did the diff to see what was different between my current split kernel (using split patches from around midnight) and the new preck6 to see if there was any major changes.

I did this mainly since my split kernel behaved fine, and to see if preck6 was likely to do the same- since then I've been using the preck6 without any issues.

It's actually behaved better in terms of memory, which I'm guessing is due to the removal of some updates/tweaks to nr_request in ll_rw_blk - my system now hovers w/ an expected/acceptable amount of free memory, rather then 4-20mb...

----------

## puntium

Hmm, sounds a bit more encourging.. the reason I ask about XFS is that when I originally reported the problem to Con, he had said that his patches worked fine on his test machines, but that he had gotten reports from other XFS users that had the same problems I had seen.

But if you guys had seen similar problems w/o xfs (i mean actually mounting an xfs partition, not just applying the patch) then maybe it doesn't have to do with xfs at all.

Anyways, i guess i'll wait around till maybe someone writes an ebuild.. looks like on Con's page ck6 isn't "pre" anymore, but he also doesn't mention anything about interactivity.. (i wonder if its stil there or not) I guess I can live with the xfs-kernel for a while longer.. i just really want the interactivity stuff w/o jumping to 2.5.. with all these emerges going on the b/g its a pain to have to wait for them to finish to be able to use my machine for anything else..

Anyhow.. good luck, i'll report back if I eventually get stuff working..

----------

## jaloha

It is causing a memory leak in the kernel.  Originally I though it was associated with the NVidia driver, but I loaded the software one instead and the problem still exist.  Has anyone tried ck-sources-2.4.20-r6 I'll install it this evening and post back to the forum if it fixes things.

UPDATE:  I have been running the ck-source-2.4.20-r6 for about 12 hours now (I think) and it just freaked out again.  The memory usage jumped from about @130 to 437.  The good thing is that the system froze for about 5-10 seconds and then recovered (before it became to slow to use).  Right now it is as responsive as normal, just the memory usage is really large.  Also, as I open large programs, such as mozilla, they eat large amounts of RAM as usual 100 Megs or so, but the total memory usage does not increase.  As I use the system the memory usage decreases (down to 399) as I type this.  Maybe it is just not reporting it as free.  If someone who knows the memory manager better than me can explain this, that would be great, but other than hanging for a few seconds, the new sources seem great.Last edited by jaloha on Fri Apr 18, 2003 10:43 am; edited 1 time in total

----------

## dizzogg

I'm running ck6 at the moment and it is running much better than ck5.  no problems yet....   :Very Happy: 

----------

## Vancouverite

I too have been running CK6 all day with no problems.

----------

## puntium

Reporting back..

Built ck6 from the portage tree wiht 200hz preempt, low latency, and xfs..  xosview shows a more sane memory usage with my typical usage .. about 50% cache not 100% user+share.

and its nice and fast  :Wink: 

----------

## Edgaer

I just upgraded to -ck6 and I'm experienceing the thrashing, then X goes funny and my box hardlocks and I've had to go back to -gentoo-r2, though I've experienced the thrashing with every kernel I've tried (gentoo, vanilla, ck4, and ck6). With the gentoo line being the most stable. With vanilla I'd even see kernel crashes after so much ram would get eaten up.

So far the two main culprits have been the blackdown JDK 1.3.whatever, some kde programs and screen savers, and big emerges. Things have gotten a little better with the new 1.4.1 JDK but I'm still having problems. Anybody got any suggestions (I'd like to be able to actually use the ck sources rather than booting it up and then reverting to the gentoo kernel when ckX is too unstable)

----------

## jaloha

 *Quote:*   

> I've experienced the thrashing with every kernel I've tried (gentoo, vanilla, ck4, and ck6). With the gentoo line being the most stable.

 

It has been my experience that the Gentoo kernel does the best job when the memory gets tight.  However, that is not the problem we have been discussing here.  It sounds like you have a different memory leak in a kernel driver (it if was in an application, you could run top, see which one and kill it and then the problem should go away).  Do you have any unusual hardware or strange file systems mounted?

----------

## puntium

I just check Con Kolivas' page and he now has an extra patch to add the interactivity update to the O(1) scheduler. I guess this means that the ck6 patch does not include the update..

Has anyone that uses XFS tried the extra interactivity patch?

----------

## Edgaer

 *jaloha wrote:*   

> It has been my experience that the Gentoo kernel does the best job when the memory gets tight.  However, that is not the problem we have been discussing here.  It sounds like you have a different memory leak in a kernel driver (it if was in an application, you could run top, see which one and kill it and then the problem should go away).  Do you have any unusual hardware or strange file systems mounted?

 

I don't know if it's a kernel issue or not. How would I go about finding out? And I shouldn't have anything weird going with the kernel. Ohter than the standard weirdness of having to compile scsi emulation in to burn cd's, the only file systems I keep open regularly are resiserfs, procfs, devfs, and tempfs. I frequently use vfat though I don't keep that mounted all the time anymore. And when I need at my /boot partition I add ext3 to the mix. And occasionally I'll have msdos or iso9660 file systems when I'm hunting through cd's and floppys.

And I noticed the thrashing start up today, so I fired up top and watched my free memory go from like 16M to 3-4M, and this was after closing a program that was eating up like 20% of my memory. Though that might have been conincindental. Though my system seems to have recovered this time as I'm typing this after shutting a few programs down. Though do you think this could be a video driver issue? I'm using the nvidia drivers though I may have also experienced this issue with the kernel's built in i810 driver as well (I don't recall if I did or not off hand)

----------

## jaloha

Edgaer,

It might just be that your system is very limited on memory.  I don't know your system specifications or if the applications you use are memory intensive.

I would first enable the "nv" driver in your XF86Config file and confirm that the problem still exist.

Then, open top and look at the values in the VIRT and RES columns.  I am not exceptionally knowledgeable about this type of thing, but I think RES is the amount of memory the application is using, and VIRT is the amount allocated to it.  If the sum of the values in the RES is greater than the total amount of memory in your system less the size of you kernel memory usage, the system has to swap to disk.  Most likely you will be able to find another application with excessive memory usage.

Also, make sure X is not reniced.

btw, how much memory do you have and what window manager are you running?

----------

## Edgaer

 *jaloha wrote:*   

> Edgaer,
> 
> It might just be that your system is very limited on memory.  I don't know your system specifications or if the applications you use are memory intensive.
> 
> 

 

Actually I was playing the other day and noticed that rather than my memory being at issue I get an extremely large load average. It was like 5.xx, 3.xx, and I think either 1.xx or 2.xx

 *jaloha wrote:*   

> 
> 
> I would first enable the "nv" driver in your XF86Config file and confirm that the problem still exist.
> 
> Then, open top and look at the values in the VIRT and RES columns.  I am not exceptionally knowledgeable about this type of thing, but I think RES is the amount of memory the application is using, and VIRT is the amount allocated to it.  If the sum of the values in the RES is greater than the total amount of memory in your system less the size of you kernel memory usage, the system has to swap to disk.  Most likely you will be able to find another application with excessive memory usage.
> ...

 

I might try that. And where do I find out the kernel memory usage?

 *jaloha wrote:*   

> 
> 
> Also, make sure X is not reniced.
> 
> 

 

Shouldn't be. Especially now that I've got the new gentoo sources with some of the ck patch stuff installed

 *jaloha wrote:*   

> 
> 
> btw, how much memory do you have and what window manager are you running?

 

128Meg and kde or fluxbox depending on needs. Also, where can I get a sane layman's guide to which memory management patch tree to use?

----------

## jaloha

 *Edgaer wrote:*   

> 
> 
> And where do I find out the kernel memory usage?
> 
> 

 

I am not really the person to ask about this, but I think just run free and then subtract your system memory from the value in total.

EXAMPLE:

```

josh@aloha josh $ free

             total       used       free     shared    buffers     cached

Mem:        515004     507720       7284          0      33276     244208

-/+ buffers/cache:     230236     284768

Swap:       257032          0     257032

```

I have 512 Megs of RAM in my computer, so this should mean that the kernel is using

((512 * 1024) - 515004) / 1024  = (approx.) 9 Megs of memory.

 *Edgaer wrote:*   

> 
> 
> 128Meg and kde or fluxbox depending on needs. Also, where can I get a sane layman's guide to which memory management patch tree to use?

 

Under KDE with an average workload (which is not small for me) my system uses about 200-300 megs of memory.  This is just what the applications require.  Do you have gtop?  This program give a good breakdown of current memory usage.  The issue is that if your programs are using 200 Megs of RAM and the system only has 128, you will experience fairly intense swapping.

To answer you second question and further address this issue, I highly recommend the new 2.5.* mm sources (on the desktop). Configuring them and debugging problems with them can be more difficult, but they is a definate performance advantage.  Use the CFQ scheduler.  The default memory manager for this kernel seems really good for the desktop.  A lot of people seem to enjoy it.

----------

