# Kernel not able to keep time and latency issues.

## silent_Walker

I compiled a kernel, 4.0.9.  The tree can be found at :

https://github.com/volkswagen/mycustomkernel

The problem that I am having is that after about 15 minutes ( could be more or less ) the time becomes off, it seems that the system thinks that 1 minute in  real life is 1 second.  The mobo. battery is a new battery.  Also when the computer forgets what a second is, I get really bad latency, it takes about 30 seconds for a menue to pop up while I am hovering over it.  I can force the menue open by highlighting it and hiting the enter key.  Keystrokes also begin to lag behind.  Since latency is horrible multitasking is impossible.  But what is weird is during this time throughput is high.  As reported by time ( maybe not the best tool) but the sys line under time is like .20 when the system can keep time but when the system can not keep time and latency goes down sys is like .07 or .01.

I did edit cfq-iosched.c  I also editied fair.c radeon.h ( I get way better fps about 700 increase according to glx gears  1200 after the edit about 500 before the edit.) sysctl.c workqueue.c prio.h sched.h Kernel hz is at 300.  I am using HPET.

 I tried this kernel under Gentoo as well as other distros and other hardware and get the same result thats how I know that it is a kernel problem.  The hardware I am running is a bit old.  Intel p4 HT with 512mb 500GB sata I hdd .  

Any help is really helpful!

----------

## Roman_Gruber

check those rtc /(real time clock) and watchdog things in the kernel and build those as modules.

Therefore you need to rebuild one more the kernel and boot up that new kernel.

----------

## silent_Walker

 *tw04l124 wrote:*   

> check those rtc /(real time clock) and watchdog things in the kernel and build those as modules.
> 
> Therefore you need to rebuild one more the kernel and boot up that new kernel.

 

Thank You, I will try that, I always build the kernel without watchdog.

EDIT: I updated git to reflect watchdog changes

----------

## Buffoon

Is there anything in dmesg that can shed light on this?

----------

## silent_Walker

 *Buffoon wrote:*   

> Is there anything in dmesg that can shed light on this?

 

The only thing that stands out to me is:

```
>>>WARNING<<< Wrong ufstype may corrupt your filesystem, default is ufstype=old

[  177.873750] hfs: can't find a HFS filesystem on dev sda2

[  346.660488] workqueue: max_active 8900 requested for writeback is out of range, clamping between 1 and 512

[  347.783922] tee (16172): drop_caches: 1

[  348.325122] tee (16200): drop_caches: 1

```

Would that be able to bring the system to a crawl?  Or is this normal?

----------

## kernelOfTruth

@silent_Walker:

don't bother doing the diff

I'm uploading the branch & diff as of now

there only seem to be a picture file and the kernel configs you posted

so it's not a kernel source issue but most likely a kernel config

or more precisely a P4 processor related issue (could also be a regression)

Why 4.0.9 btw ?

Did you try out older or newer kernels (like 4.1.6 or even 4.2) ?

There has been some changes to older core 2 (and also p4 ?) processors recently which could be related

https://github.com/kernelOfTruth/linux/commits/b27e12262751b9d47e70fcd61b0cff8923c0ede0

related to the error message in dmesg:

http://www.mailbrowse.com/linux-kernel/990785.html [PATCH 28/35] workqueue: increase max_active of keventd and kill current_is_keventd()

----------

## silent_Walker

 *kernelOfTruth wrote:*   

> @silent_Walker:
> 
> Why 4.0.9 btw ?
> 
> Did you try out older or newer kernels (like 4.1.6 or even 4.2) ?
> ...

 

I started to build this kernel a few weeks ago now and at that time that was the latest release.

 *Quote:*   

> don't bother doing the diff
> 
> I'm uploading the branch & diff as of now 

 

Your to fast man, you beat me to it.

 *Quote:*   

> 
> 
> https://github.com/kernelOfTruth/linux/commits/b27e12262751b9d47e70fcd61b0cff8923c0ede0
> 
> related to the error message in dmesg:
> ...

 

So what was in dmesg was not from me changing values in cfq-iocshed, then.  I will apply that patch and see what happens in terms of performance.

----------

## kernelOfTruth

no no no,

it's already included in the source (2010),

the changes you did to the scheduler and CFQ don't seem to be there (on github) ?

What exactly to do you want to get with those changes of the scheduler ?

better responsiveness - low latency ?

you know of BFQ ?

and BFS ?

if you're striving for low latency try those out:

http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php

http://ck-hack.blogspot.co.at/

----------

## kernelOfTruth

 *silent_Walker wrote:*   

> I compiled a kernel, 4.0.9.  The tree can be found at :
> 
> [snip]
> 
> I did edit cfq-iosched.c  I also editied fair.c  Kernel hz is at 300.  I am using HPET.
> ...

 

that's really weird   :Confused: 

I suggest trying with a new kernel source and then applying either CFQ or CFS

you could start anew with the kernel config of SysRescueCD or e.g. Pappy's Kernel Seeds (https://forums-web2.gentoo.org/viewtopic-t-942572-postdays-0-postorder-asc-start-0.html?sid=95423dcba19b11bc8ecae688c0575971)

then slim down the kernel (binary) from there

Unfortunately I don't have any i686 kernel config available right now

I tended to have them for an HP Microserver but it seems I deleted or lost them some time ago   :Confused: 

anyway - I'm "off" for some time

hope this helps

----------

## silent_Walker

 *kernelOfTruth wrote:*   

> no no no,
> 
> it's already included in the source (2010),
> 
> 

 

Oops, did not look at the year

 *Quote:*   

> 
> 
> the changes you did to the scheduler and CFQ don't seem to be there (on github) ?

 

They are there look under linux-4.0.9/block/ and then look for cfq-iosched.c, I just edited values I did not add anything or remove anything.

 *Quote:*   

> 
> 
> What exactly to do you want to get with those changes of the scheduler ?
> 
> better responsiveness - low latency ?

 

I just wanted to see if that I could increase throughput while decreasing latency a bit or decrease throughput and get the latency really low.  I also did not want any delay, as soon as one task finishes I want the next one to start right then, not a little after.

 *Quote:*   

> 
> 
> you know of BFQ ?
> 
> and BFS ?
> ...

 

I have heard of those, but where is the fun in using some one else's ideas.  

Basically what I am doing is instead of having a sysctl.conf I am hard coding the values that I want the kernel to use.

----------

## silent_Walker

After leaving the machine on for 30 minutes with a webpage open I checked dmesg

and I get:

```

 1920.836649] CPU0: Core temperature/speed normal

[ 1920.836652] CPU1: Core temperature/speed normal

[ 1949.996079] mce: [Hardware Error]: Machine check events logged

[ 2266.291725] [sched_delayed] sched: RT throttling activated

[ 2359.055686] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1413)

[ 2359.056145] CPU1: Core temperature/speed normal

[ 2359.056147] CPU0: Core temperature/speed normal

[ 2399.996085] mce: [Hardware Error]: Machine check events logged

[ 2845.148090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 1428)

[ 2845.148098] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1428)

[ 2845.148657] CPU1: Core temperature/speed normal

[ 2845.148659] CPU0: Core temperature/speed normal

[ 2849.996060] mce: [Hardware Error]: Machine check events logged

[ 2965.646424] psmouse serio1: Wheel Mouse at isa0060/serio1/input0 lost synchronization, throwing 1 bytes away.

[ 7192.197732] CPU1: Core temperature above threshold, cpu clock throttled (total events = 10229)

[ 7192.197740] CPU0: Core temperature above threshold, cpu clock throttled (total events = 10229)

[ 7192.198232] CPU1: Core temperature/speed normal

[ 7192.198234] CPU0: Core temperature/speed normal

[ 7505.684125] CPU1: Core temperature above threshold, cpu clock throttled (total events = 11741)

[ 7505.684135] CPU0: Core temperature above threshold, cpu clock throttled (total events = 11741)

[ 7505.684517] CPU1: Core temperature/speed normal

[ 7505.684518] CPU0: Core temperature/speed normal

[ 7810.376090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 12469)

[ 7810.376095] CPU0: Core temperature above threshold, cpu clock throttled (total events = 12469)

[ 7810.376672] CPU1: Core temperature/speed normal

[ 7810.376671] CPU0: Core temperature/speed normal

[ 8113.106502] CPU1: Core temperature above threshold, cpu clock throttled (total events = 16218)

[ 8113.106515] CPU0: Core temperature above threshold, cpu clock throttled (total events = 16218)

[ 8113.107078] CPU0: Core temperature/speed normal

[ 8113.107080] CPU1: Core temperature/speed normal

[ 8413.692102] CPU0: Core temperature above threshold, cpu clock throttled (total events = 20604)

[ 8413.692114] CPU1: Core temperature above threshold, cpu clock throttled (total events = 20604)

[ 8413.692561] CPU1: Core temperature/speed normal

[ 8413.692561] CPU0: Core temperature/speed normal

[ 8910.520094] CPU0: Core temperature above threshold, cpu clock throttled (total events = 24043)

[ 8910.520104] CPU1: Core temperature above threshold, cpu clock throttled (total events = 24043)

[ 8910.520637] CPU1: Core temperature/speed normal

[ 8910.520639] CPU0: Core temperature/speed normal

[ 9452.168097] CPU0: Core temperature above threshold, cpu clock throttled (total events = 27060)

[ 9452.168107] CPU1: Core temperature above threshold, cpu clock throttled (total events = 27060)

[ 9452.168674] CPU1: Core temperature/speed normal

[ 9452.168677] CPU0: Core temperature/speed normal

[ 9752.389374] CPU1: Core temperature above threshold, cpu clock throttled (total events = 28735)

[ 9752.389381] CPU0: Core temperature above threshold, cpu clock throttled (total events = 28735)

[ 9752.389954] CPU0: Core temperature/speed normal

[ 9752.389956] CPU1: Core temperature/speed normal

[10065.584118] CPU0: Core temperature above threshold, cpu clock throttled (total events = 32286)

[10065.584126] CPU1: Core temperature above threshold, cpu clock throttled (total events = 32286)

[10065.584617] CPU0: Core temperature/speed normal

[10065.584619] CPU1: Core temperature/speed normal

[10366.447957] CPU1: Core temperature above threshold, cpu clock throttled (total events = 36493)

[10366.447964] CPU0: Core temperature above threshold, cpu clock throttled (total events = 36493)

[10366.448454] CPU1: Core temperature/speed normal

[10366.448454] CPU0: Core temperature/speed normal

[10668.685251] CPU1: Core temperature above threshold, cpu clock throttled (total events = 37678)

[10668.685258] CPU0: Core temperature above threshold, cpu clock throttled (total events = 37678)

[10668.685715] CPU0: Core temperature/speed normal

[10668.685717] CPU1: Core temperature/speed normal

[11035.508090] CPU1: Core temperature above threshold, cpu clock throttled (total events = 39158)

[11035.508098] CPU0: Core temperature above threshold, cpu clock throttled (total events = 39158)

[11035.508555] CPU0: Core temperature/speed normal

[11035.508557] CPU1: Core temperature/speed normal

[11341.756081] CPU1: Core temperature above threshold, cpu clock throttled (total events = 39257)

[11341.756088] CPU0: Core temperature above threshold, cpu clock throttled (total events = 39257)

```

But this can not be the case I am putting my finger on the cooler and it is cold, I do not burn my finger at all.  So how would I turn off cpu throtling even if the cpu gets "too hot" which it  is not because my finger is on the cooler and did not get burned.  Why is my computer lying to me?

----------

## kernelOfTruth

 *Quote:*   

> mce: [Hardware Error]: Machine check events logged 

 

well, there's something happening

install mcelog

```
emerge mcelog
```

after that see what the program says about that hardware error

also what do lm_sensors say ?

```
emerge lm_sensors
```

configure it with sensors-detect

then fire it up

```
/etc/init.d/lm_sensors start
```

```
sensors
```

----------

## krinn

 *kernelOfTruth wrote:*   

>  *Quote:*   mce: [Hardware Error]: Machine check events logged  
> 
> well, there's something happening
> 
> 

 

Overheat are report also thru mce ; so that should be that. He should really look at temperature trouble.

----------

## kernelOfTruth

@krinn:

roger that - forgot about that connection,

thanks   :Smile: 

@silent_Walker:

first make sure you have a stable base system (or kernel base) then you can tweak to your hearts liking

(also make sure to have a backup of that stable system then)

----------

## silent_Walker

 *krinn wrote:*   

>  *kernelOfTruth wrote:*    *Quote:*   mce: [Hardware Error]: Machine check events logged  
> 
> well, there's something happening
> 
>  
> ...

 

My thermal paste was dry ( that was the stock paste from 2004), so most likely that is why the cooler was not getting warm since the heat was not transferring well.

 However I still get horrible latency.  So I do not think that it is a heat issue.  I built the kernel again with 1000 hz and right after boot into X the clock did not work and task switching and latency were poor.  I heard of the divider command via grub and set it for 25, so I was running on 40hz, it did boot, but because of the low hz it was way slower then 1000 with the horrible latency.  I tried setting it to divider=10 for 100hz but every time the computer tried to mount the HDD it would freeze and the scroll and caps lock will just flash.  I will try to build the kernel without pre-emption and see if that would fix it.  

If not I will rebuild the kernel but only add one of my modified files, till I get the bad performance then I know which file caused the bad latency.

I also built the kernel with debugging and everything came back with an OK or PASSED . Maybe the hardware I am testing on is just to old to handle the edits I  made.

----------

## silent_Walker

 *kernelOfTruth wrote:*   

> 
> 
> you know of BFQ ?
> 
> and BFS ?
> ...

 

Do you know of any other schedulers that are out there that work with the mainline kernel?  Besides deadline BFS CFS.

How well do BFS CFQ work together.

----------

## kernelOfTruth

 *silent_Walker wrote:*   

> 
> 
> Do you know of any other schedulers that are out there that work with the mainline kernel?  Besides deadline BFS CFS.
> 
> 

 

I can't consciously list them,

in the past years I've come across a few which were in the proving stage - meaning:

they work for one kernel releases and are a technical previews (mainly developed to achieve a thesis, master degree - at universities)

but aren't maintained.

If you're up to it do some googling and see what comes up, then go from there

 *silent_Walker wrote:*   

> 
> 
> How well do BFS CFQ work together.

 

they're fine,

you don't necessarily need BFQ but the latency is somewhat lower than CFQ with high i/o

also with BFS you have SMTNICE which works with SMT capable processors,

CFS doesn't really consider it (well)

Consider using 

```
schedtool

chrt
```

tools to raise priority of tasks and/or IRQs - this can cut down latency,

also use a preemptive kernel instead of voluntary preemption or non preemptive,

there's also the RCU settings (boosting) that you can tinker with

also boot up the kernel with appending

threadirqs

that - simply said - the IRQs, etc. get "split up" to have a finer granular tuning working surface (more knobs to tweak)

----------

## Buffoon

Gentoo is a Linux you build for yourself. 

I would test this hardware with something that is built by somebody else, just to see if it is your Gentoo that can't handle the hardware or the hardware itself is expiring on you.

There are times when CMOS reset can help to get rid of weird issues, my 2¢.

----------

## silent_Walker

 *Buffoon wrote:*   

> Gentoo is a Linux you build for yourself. 
> 
> I would test this hardware with something that is built by somebody else, just to see if it is your Gentoo that can't handle the hardware or the hardware itself is expiring on you.
> 
> There are times when CMOS reset can help to get rid of weird issues, my 2¢.

 

Thanks.

I took the CMOS battery out and left it out for 30 minutes but still get the same problems.  I will try maybe slitaz, as it is supposed to be great on older hardware.

At first I thought I knew what the problem was, I was looking thru the files that I edited and saw that I changed ( under sysctl.c)

```
 #ifdef CONFIG_PRINTK

 static int ten_thousand = 10000;
```

to

```
 #ifdef CONFIG_PRINTK

static int ten_thousand = 1000;
```

But apparently the value ranges from 0 to 10,000 and PRINTK does something else then what I thought it did.  Before I did my research on PRINTK I thought that 10,000 was the perfect value for latency and throughput.  I was going to put it  back to 10k and seen if that would fix it ( the problem) and then go higher for better latency at the risk of lower throughput.  Unfortunately what I thought what would happen wont happen.  

I was looking at this webpage describing CFS, so I think that now may values may be out of sync. under fair.c and sysctl.c

my sysctl.c has these values:

```

static int min_sched_granularity_ns = 10000;      /* 10 usecs */

static int max_sched_granularity_ns = 80000;   /* 80 usecond */

static int min_wakeup_granularity_ns;         /* 0 usecs */

static int max_wakeup_granularity_ns = 25000;   /* 25 usecond */

```

from the default of:

```

263 static int min_sched_granularity_ns = 100000;           /* 100 usecs */

264 static int max_sched_granularity_ns = NSEC_PER_SEC;     /* 1 second */

265 static int min_wakeup_granularity_ns;                   /* 0 usecs */

266 static int max_wakeup_granularity_ns = NSEC_PER_SEC;    /* 1 second */

```

and my fair.c has:

```

unsigned int sysctl_sched_latency =200000ULL; ( do I need the space after the "=" sign? )

unsigned int normalized_sysctl_sched_latency = 75500000ULL;  ( woah,  I am an idiot, I set this number way to high.  I put it to 75.5 seconds.  Following "stock" this should be the same as sysctl_sched_latency. so 200000 or .2 seconds)

unsigned int sysctl_sched_min_granularity = 150000ULL;

unsigned int normalized_sysctl_sched_min_granularity = 200000ULL;

static unsigned int sched_nr_latency = 20;

unsigned int sysctl_sched_wakeup_granularity = 0UL;

unsigned int normalized_sysctl_sched_wakeup_granularity = 80000UL;

const_debug unsigned int sysctl_sched_migration_cost = 695UL;  ( 0.000695 seconds or .695 ms ) 

nsigned int sysctl_sched_cfs_bandwidth_slice = 166000UL

```

from the default of :

```

unsigned int sysctl_sched_latency = 6000000ULL;

unsigned int normalized_sysctl_sched_latency = 6000000ULL;

unsigned int sysctl_sched_min_granularity = 750000ULL;

unsigned int normalized_sysctl_sched_min_granularity = 750000ULL;

static unsigned int sched_nr_latency = 8;

unsigned int sysctl_sched_wakeup_granularity = 1000000UL;

unsigned int normalized_sysctl_sched_wakeup_granularity = 1000000UL;

const_debug unsigned int sysctl_sched_migration_cost = 500000UL; ( 0.5 seconds)

unsigned int sysctl_sched_cfs_bandwidth_slice = 5000UL;

```

So I am thinking that maybe the time-slice is off and that tasks are not being allowed to run.

I am probably wrong but the sysctl_sched_min_granularity decides the minimum amount of time it runs before being pre-empted I changed the value from 75 milliseconds to 15, so things are getting pre-empted faster.  So if they are getting pre-empted faster that means the cpu is taking on more things at once and can not cope and is killing off some tasks like the clock because it has lower priority.  I have no idea.

But when running at 1000hz more things go thru the CFS right?  And things get pre-empted even faster thats why right after boot on 1000hz the system crawls, while on 300hz it takes time to reach that point where latency "drops dead".

Throughput does increase tho opening thunar when latency goes down time reads:  Maybe this is from normalized_sysctl_sched_latency = 75500000ULL?

```

real   0m1.255s

user   0m0.050s

sys   0m0.008s

And here is the output from when latency is good

real   0m2.025s

user   0m0.414s

sys   0m0.054s

```

So it opens 1.2 seconds faster.  So for a headless machine this would be awesome, not so much  for a desktop tho.

EDIT:  Found an old email  from 2007 http://lkml.iu.edu/hypermail/linux/kernel/0711.0/1207.html

 *Quote:*   

>  kernel_sched_nr_latency should always be set to kernel_sched_latency/kernel_sched_min_granularity. (it's not a free tunable)

 

mine is not. It should be set from 20 to 13.3333333, I will just put that to 13. ( or should I round up?)

EDIT2: Found something from Fedora, maybe this applies to me as well.

 *Quote:*   

> Description of problem:
> 
> Big virtualization hosts can experience high contention at the run queue lock, please see bug 960816 for more details.
> 
> Setting kernel.sched_migration_cost tunable to 5000000 (10X its default value) eliminates the contention.

 

Mine is waayyy below .5 seconds so increase the number to 5000000 ( 5 seconds ) may help or at least restoring it to the stock of .5s

I also read somewhere else that 

```
/proc/sys/kernel/sched_autogroup_enabled
```

should be set to 0 for performance I have mine on 1.  I do not think that that is causing the problem but it may be a way to get around the problem.  I will recompile the kernel with 

```

static unsigned int sched_nr_latency = 20; 

to

static unsigned int sched_nr_latency = 13;

and 

normalized_sysctl_sched_latency = 75500000 (75 seconds)

to

normalized_sysctl_sched_latency = 200000 (0.2 seconds)

```

Then if that does not work I will increase the value of 

```
 

sysctl_sched_migration_cost = 695 (0.000695 seconds) 
```

to stock of 

```

sysctl_sched_migration_cost = 500000 (5ms)
```

----------

## kernelOfTruth

Keep in mind that

lower latency could equal lower performance & throughput

Take a look at the following settings:

http://www.overclock.net/t/1519964/experiment-overclocking-the-linux-os-through-the-sysclt-proc-sys-sys-block-and-other-settings#post_23094293

----------

## silent_Walker

 *kernelOfTruth wrote:*   

> Keep in mind that
> 
> lower latency could equal lower performance & throughput
> 
> 

 

So having normalized_sysctl_sched_latency = 75500000ULL or 75 seconds is why throughput is high.  Because higher latency could equal higher performance and throughput.

 *Quote:*   

> 
> 
> #Given value and default was 950000
> 
> #At it's default of 950000 RTs can take 95% of CPU usage/sec
> ...

 

Is disabling this really a safe option.

----------

## kernelOfTruth

 *silent_Walker wrote:*   

> 
> 
>  *Quote:*   
> 
> #Given value and default was 950000
> ...

 

I'd keep it around 95 - 97% since there were some lockups with having it disabled

----------

## Ant P.

Have you benchmarked any of these changes you've made using tools like latencytop and perf?

I'd love to know what kind of a performance boost you're seeing with edits like these.

----------

## silent_Walker

 *Ant P. wrote:*   

> Have you benchmarked any of these changes you've made using tools like latencytop and perf?
> 
> I'd love to know what kind of a performance boost you're seeing with edits like these.

 

I have not been able to run latency top and perf yet.  All of my kernels on my p4 get to a kernel panic, about 2 minutes into boot.  And my laptop charger is barley working ( thats why I am on this ancient p4)

```
 Kernel panic - not syncing: CPU context corruption
```

Unfortunately thats all it says, and the computer locks up.

But as far as my edits to radeon.  They have been some what good.  With my p4 machine I have an old RV280SE, that was not able to play 1080p fluidly, I was only getting anywhere from 6-9 ( 9 for a non action scene and 6 for an action scene) fps with about 1 fps drop every 5 seconds just about, and with the new 4k resolution, videos would play at 1 fps.  When I edited, the radeon.h

I could get 1080p videos to play at about 25 fps but get about 3 dropped frames a second on a non action scene, like some one walking around or talking to another person, but with an action scene I would get 11 fps with about 10 dropped frames a second, so it would be laggy.  I tried to find a reason for this and I edited the radeon clocks file, this does not overclock the card as far as I know.  But dropped fps went down about 2%. 

Also when I did not edit the radeon clocks file, and ran glx gears, the fps would start at 1200 ( up from the 600 I usually get) but it would keep climbing, if I wanted to I could of gotten a million fps, if I left it on for an hour, after editing the clocks file, the result is more constantly around 1200 fps.  That is with no other windows open, and I am running Openbox not something with a lot of eye candy to take away from glxgears.

I am about to edit the radeon.h one more time, because I want to see if I can lower the dropped fps more by increasing the timeout.  If you look the stock is 100 ms timeout I changed that to 3ms timeout.  So I will change it to 60ms timeout and will run glx gears and youtube stats for nerds to see whether or not fps is increased or decreased by the timeout and also see if dropped fps goes down.

This is with flash and not html5 as the default viewer for youtube as my hardware does not fully support features of html5.

Without and with the edits I can not play full screen videos via youtube at all, its all lag.  This is why I have not run any games on it yet, as I think that the game would fail horribly performance wise on this machine, although once I can get around the kernel panic I will try nexuiz in a window mode and in full screen and report the fps count.

Also if you are wondering about the "y" that is in there at line 5 on radeon_clocks.c I added that for no reason.

Of course some may say these results are not 100% true, because glxgears is not a true benchmark tool but that is what I had available at that time.  The same for youtube's stats for nerds.

----------

## kernelOfTruth

 *Ant P. wrote:*   

> Have you benchmarked any of these changes you've made using tools like latencytop and perf?
> 
> I'd love to know what kind of a performance boost you're seeing with edits like these.

 

Ant P., thanks for the hint,

I'm out   :Surprised: 

----------

