# New low-latency work, 2.6.8-rc1-ck6 and reiser revelations

## kerframil

Hi all,

EDIT 10/08/2004: The latest branch is now 2.6.8-rc4-ck2.

firstly I'd like to announce that snapshots of Con Kolivas' ck patchset are available against 2.6.8-rc1 (the latest being 2.6.8-rc1-ck5-0407142235 at the time of writing, although it identifies itself simply as 2.6.8-rc1-ck5).

Apart from the excellent work that has been going on in the mainline 2.6 branch, the development of this snapshot has drawn attention once more to areas of high latency in the kernel with such luminaries as Andrew Morton, William Lee Irwin III and Nick Piggin becoming involved in the ensuing discussion. Throughout this time, ordinary users have been making a great effort to test for areas of high latency (thanks to a patch by WLI specifically designed to diagnose these areas).

The upshot of this is that Andrew Morton has recently committed a handful of latency fixes to the tree, all of which (if I'm not mistaken) can be found in the aforementioned ck snapshot   :Very Happy: 

Some of you may also have heard of Ingo Molnar's "voluntary preempt" patch which basically adds more preemption points to the kernel without taking the "lock, stock and barrel" approach of the standard CONFIG_PREEMPT option. This was present in the last few releases of the ck snapshot, but has recently been dropped because it is (a) somewhat immature (b) hasn't proven to be of significant value in its current from ... however the patch is still floating around for those who want to try it.

For those interested in the low-latency work, I strongly recommend that they review the individual patches in the ck branch and check out the following ... as well as trying ck of course  :Wink: Linux: Low Latency and Filesystems (Kerneltrap)ck mailing listWARNING FOR REISERFS LOVERS: Much to my horror, a lot of evidence seems to have arisen during latency testing that suggests that reiserfs (3) has suffered some serious regression in terms of latency compared to 2.4 kernels; indeed ext3 seems to wipe the floor with it in terms of scheduling latency.

And, for those that think that resiser4 is the natural answer to all of our performance concerns I would strongly suggest that they consider the hype very carefully before making the jump ... (admittedly that is just one person's set of experiences but I found myself even more incredulous than I was before to the idea that reiser4 is going to solve the world's problems at this point in time).

So, I hope you enjoy testing the new -ck branch and I wonder if any of you other patchset authors out there would be interested in including these new latency fixups   :Cool: 

Please note that, as always, I recommend that all typical desktop users do not make use of the CONFIG_PREEMPT option unless they have a specific requirement for it (e.g. pro audio processing, JackIt, live video processing and so forth). This advice stands for all recent 2.6 kernels, not just -ck.

Furthermore, as I recommended on a previous -ck related post, if you experience poor scheduling performance with a particular program (in particular, winex / cedega) then it is probably because this program is buggy and makes poor use of yielding (which doesn't sit very well with the staircase scheduling policy). The simple solution, believe it or not, is to run the program in question with a nice level of +19.Last edited by kerframil on Tue Aug 10, 2004 1:08 pm; edited 4 times in total

----------

## Gentii

Ha, I understand now. When I saw the difference between ck and mm or love in winex, I thought ck was really bad. Now it explains the high difference  :Smile: 

I've to try ck again, with that nice level for winex. But why 19 and not 18 or 20 ?

----------

## butters

nice takes values that range from -19 - 19, so 20 won't work, and 19 with be the "most" nice.

I wouldn't jump to any conclusions that poor reiserfs (3) latency in 2.6 indicates a potential problem for reiser4.  I expect, instead, some unexpected and unrelated areas of high latency due to its immaturity.  After all, it's mostly a complete rewrite.

Perhaps vfs should have hooks to let the scheduler and fs implementation communicate and work together?  Also, perhaps some software analog of branch prediction can be employed to buffer reads and writes from threads likely to be scheduled next.

Nonetheless, I would hope that reiser4, developed primarily for 2.6, would at least outperform ext3!

----------

## DiskBreaker

Running it now... I must say that's one awesome kernel! Con even included SuSE's old writeback latency patch. Apparently CFQ has been fixed a bit, too. Way to go for staircase! 

Since we've had selectable IO-Schedulers on boot for some time now, I think once its development stabilizes a bit it will be time to merge staircase into the mainline 2.6 kernel and have selectable Process Schedulers, too.  :Wink: 

In the meantime CPUSE, the CPU Scheduler Evaluation Patches (https://sourceforge.net/projects/cpuse/) already support switching Process Schedulers at runtime and even have a GUI for finetuning them...

Cheers,

DiskBreaker

----------

## barry

I've had no luck with this kernel. It compiles fine, but performance is terrible. An emerge sync causes the mouse pointer to freeze. Back to -ck5 for me, no problems there (I add in the SUSE patch manually).

----------

## kerframil

 *barry wrote:*   

> I've had no luck with this kernel. It compiles fine, but performance is terrible. An emerge sync causes the mouse pointer to freeze.

 

Hmm, too bad   :Confused: 

Just out of interest, what filesystem are you using?

----------

## kerframil

 *butters wrote:*   

> I wouldn't jump to any conclusions that poor reiserfs (3) latency in 2.6 indicates a potential problem for reiser4.  I expect, instead, some unexpected and unrelated areas of high latency due to its immaturity.  After all, it's mostly a complete rewrite.

 

I wasn't inferring that there was a relationship between the two - after all the two implementations are quite different. As a staunch reiserfs user, the information about the latency regression was a little upsetting! The reiser4 information obviously is concerning performance in general and I just thought I'd mention it.

 *Quote:*   

> Perhaps vfs should have hooks to let the scheduler and fs implementation communicate and work together?  Also, perhaps some software analog of branch prediction can be employed to buffer reads and writes from threads likely to be scheduled next.

 

Interesting notions. There is also the concept of I/O priorities; I believe Jens Axboe implemented it in a cutting-edge branch of CFQ but I don't think it is present in the current mainline implementation. Also, have you tried out the write barriers feature (which can be found in -mm)?

 *Quote:*   

> Nonetheless, I would hope that reiser4, developed primarily for 2.6, would at least outperform ext3!

 

Heh, indeed  :Wink: 

----------

## barry

 *kerframil wrote:*   

> 
> 
> Just out of interest, what filesystem are you using?

 

Reiserfs. I've not seen problems like this with earlier CK kernels, though.

A quick note to add I'm using the current snapshot (snapshot-2.6.8-rc1-ck6-0407151120.bz2) and not the one above.

----------

## fallow

the latests ck patchsets becoming as the most greatestful of all  :Smile: 

I`m using Staircase since 2.6.4 and since  versions 6.5+ I`m very delighted.

greetings

----------

## fazto

Kerframil, are you refering to the lkml discussion about reiserfs latency concerns? I am planning a new box and, by default, I tend to choose reiserfs (3). Now, i'm not so sure anymore. I'm planning to run the -ck or -love kernel on it so maybe I should go to ext3. Do you know if Hans Reiser has commented on this latency issue's? I browsed the namesys mailinglist, but could not find anything.

Switching filesystems after install is a lot of work , so I want to make a well informed choice.

----------

## kerframil

 *Quote:*   

> Kerframil, are you refering to the lkml discussion about reiserfs latency concerns? I am planning a new box and, by default, I tend to choose reiserfs (3). Now, i'm not so sure anymore. I'm planning to run the -ck or -love kernel on it so maybe I should go to ext3.

 

Yes, I was referring to said discussion but I don't want to blow the problem out of proportion. Naturally, there are more factors that govern performance than latency alone. In this case, I don't think that latency time == service time (service time meaning the length of time it takes for a request to be fully serviced), rather it refers to scheduling latency for transactions fielded through the reiserfs code.

The kind of folks who would be really concerned by it would probably be the same kind who genuinely need preempt; those who really need predictable and low mean average scheduling latencies throughout all aspects of system performance. Frankly, that is a small proportion of us.

However, I will say this: I feel quite certain that reiserfs performance has somewhat regressed throughout the 2.6 development period. Quite how, I cannot say - but I have also seen comments supporting that view now and again on the reiser list.

 *Quote:*   

> Do you know if Hans Reiser has commented on this latency issue's?

 

Probably not. Frankly, I don't think he cares very much about it; he wants everyone to jump on the reiser4 bandwagon as soon as possible ("it's production ready ... because I say so!") and only Chris Mason seems to be doing any serious maintenance on the reiserfs codebase these days. I recall the incredible fuss that Hans kicked up when Chris came up with the (rather neat) patches to provide extended attribute and POSIX ACL support ... features that people actually want and expect. That provided me with a somewhat displeasing insight into the way Hans thinks, even if he is a visionary and/or genius.

I still like reiserfs and will probably stick to it for server usage (it's served me well and seems to suit my usage patterns). However, I intend to ditch it for workstation use and have no intention of moving to reiser4 yet. Personally, I am going to move to XFS. The ext3 filesystem may wipe the floor with reiserfs in terms of scheduling latency, but I still consider it to be a low performance filesystem in general (and overrated).

This is probably an opportune time to mention something else that may be very important when it comes to filesystem performance. An interesting thread ensued on the ck mailing list recently which raised several points I was not previously aware of:

Ext3 really works well in conjuction with the anticipatory scheduler (I suppose they are both the "standards" within their respective realms).

XFS really does not work well with the anticipatory scheduler (better to use elevator=cfq instead or deadline if you don't trust cfq for some reason).

The anticipatory scheduler can be a performance impediment for RAID arrays that use a heavy amount of mirroringThe big eye-opener for me is that the choice of filesystem should have implications in terms of one's choice of I/O scheduler - and it is not a topic that I have seen discussed before.

Anyway, my choice is going to be XFS henceforth. I have been of the opinion that it is easily the most scaleable journalling filesystem bar none for some time but had reservations about its maturity (at least, in the Linux kernel). These reservations have diminished somewhat now and I hear very good things about it. Unfortunately, I do not know much about its scheduling latency but I suspect that it is good (after all, it actually has a hard realtime option - CONFIG_XFS_RT and, given SGI's background, one would assume that they are aware of the need for deterministic performance in general).Last edited by kerframil on Tue Aug 10, 2004 6:10 pm; edited 1 time in total

----------

## fazto

You don't answer fast, but boy, do you answer thorough   :Laughing: 

Thanks!! I will look into XFS more seriously, I didn't know it was such a good option nowadays.

----------

## Isaiah

 *fazto wrote:*   

> Thanks!! I will look into XFS more seriously, I didn't know it was such a good option nowadays.

 

Ditto dat - has yours truly convinced to give XFS a shot also  :Cool: 

----------

## GentooBox

 *Isaiah wrote:*   

>  *fazto wrote:*   Thanks!! I will look into XFS more seriously, I didn't know it was such a good option nowadays. 
> 
> Ditto dat - has yours truly convinced to give XFS a shot also 

 

I had it like that for a long time, and when i came to reinstall my gentoo, xfsprogs dident work (on the livecd) witch sucked..

but i still prefer Reiser4, it seems like the future. (just like winfs)

----------

## kerframil

Regarding my vague complaints about reiserfs regression, I just saw a post from Hans on the list, touching upon apparent problems with the block allocator algorithm (I remember Chris doing some work on it not so long ago). That post, and its parents, make for very interesting reading. I hope they get it sorted out quickly because reiserfs was one of the best filesystems in terms of long-term fragmentation issues. I suspect that any patches would probably appear in Andrew Morton's -mm tree first. 

Of course, another thing to remember is not to use tails because they are (and always have been) dog slow (a tails=small option was added not so long ago as a compromise, but I suspect the the reduction in filesystem slack is still not worth it).

----------

## DocterD

CK-2.6.8.1 is out. But this release doesn't work for me. When starting X i get a Kernel Panic. 

CK1-2.6.8.1, Xorg-DRM Radeon and NVNet

Has anyone a ebuild?

----------

## Isaiah

 *GentooBox wrote:*   

> ...but i still prefer Reiser4, it seems like the future. 

 

I get the future part - giving xfs a shot anyway (it was certainly easy enough to do)  :Cool: 

----------

## teutzz

that hype about reiser4 mentioned here is quite old and outdated, for instance even the binary format of the partition changed twice (no sorry 3 times) at least since that date and not to mention the kernel patches and how much that changed since then.. so if anything i think that that test must be considered as obsolete

i haven't done tests with reiser4 (even thow i've been using it on my root and boot partitions for the past several months) but if not for anything else it feels quite snappy (and as for space efficiancy -> awsome))

----------

## boroshan

 *teutzz wrote:*   

> i haven't done tests with reiser4 (even thow i've been using it on my root and boot partitions for the past several months) but if not for anything else it feels quite snappy (and as for space efficiancy -> awsome))

 

Glad it's working for you. I didn't notice any particular improvement, and suffered a lot of R4 related kernel problems. I started using reiser 3, upgraded to R4 and then switched again to xfs, mainly on the strength of this thread.

So far I haven't missed them a bit. I'm looking forward to reiser4 maturing a bit, and I think it has tremendous potential. It's just not ready for prime time yet.

----------

