# [HOWTO] Fun with io priorities aka ionice

## pijalu

Sometimes, when my sweet gentoo do some heavy background tasks on my laptop (eg: big fat emerge, update-db, prelink, etc...), my io usage becomes annoying (no cpu use... just hd is out for lunch)...and this, even if the tasks are niced 

==> My desktop starts to lags... opening a console takes way too much time....

If you already experience this, here comes a tip to keep a pleasant desktop: messing with io priorities...

As usual, this is not garanteed, if you break something, get hacked, loose your job or even start to miss windows, don't complain to me  :Wink: .... This won't make your desktop go faster with IO... just reschedule things to keep a snappy desktop

So, let's start !

Check kernel settings:

* Be sure you have a PREEMT enabled kernel

```

$ zcat /proc/config.gz | grep PREEMT

# CONFIG_PREEMPT_NONE is not set

CONFIG_PREEMPT_VOLUNTARY=y

# CONFIG_PREEMPT is not set

```

(normal or voluntary should do)

* be sure CFQ iosched is enabled

```

$ zcat /proc/config.gz | grep IOSCHED

CONFIG_IOSCHED_NOOP=y

CONFIG_IOSCHED_AS=y

CONFIG_IOSCHED_DEADLINE=y

CONFIG_IOSCHED_CFQ=y

CONFIG_DEFAULT_IOSCHED="anticipatory"

```

* if CONFIG_IOSCHED_CFQ is set but CONFIG_DEFAULT_IOSCHED is not CFG (like here)

add to /etc/conf.d/local.start:

```

   for drive in  /sys/block/*/queue/scheduler 

   do

      echo cfq > $drive

   done

```

(restart local)

* install ionice

a) get the source from kernel docs:

```

sed -n -e "/snip ionice.c/,/snip ionice.c/ p" /usr/src/linux/Documentation/block/ioprio.txt | sed "s/.\+snip ionice.\+//g" > ionice.c

```

If you want all users to have access to ionice, it will need to bet set uid root ==> this mean security hazard..... the following patch should avoid normal user setting higher ioprio than default and, last but not least, avoid ionice to start programs as root.... This is not bullet proof and can lead to security problems... jump to point e and skip f if you plan to use ionice as root only)

b) Patch it => create a file called ionice.patch and copy the following content

```

--- ionice.c.ori   2006-05-14 13:59:53.000000000 +0200

+++ ionice.c   2006-05-14 13:56:32.000000000 +0200

@@ -1,5 +1,3 @@

-

-

 #include <stdio.h>

 #include <stdlib.h>

 #include <errno.h>

@@ -99,14 +97,23 @@

          printf("%s: prio %d\n", to_prio[ioprio_class], ioprio);

       }

    } else {

+      if (getuid()!=0 && (ioprio_class < IOPRIO_CLASS_BE || ioprio < 4)  ) {

+         fprintf(stderr,"User cannot set higher ioprio\n");

+         return 1;

+      }

       if (ioprio_set(IOPRIO_WHO_PROCESS, pid, ioprio | ioprio_class << IOPRIO_CLASS_SHIFT) == -1) {

          perror("ioprio_set");

          return 1;

       }

 

-      if (argv[optind])

+      if (argv[optind]) {

+          if (seteuid(getuid())!=0 || setegid(getgid())!=0) {

+         perror("Error setting real user");

+         return 1;

+          }

          execvp(argv[optind], &argv[optind]);

    }

+   }

 

    return 0;

 }

```

c) check if patch is fine

```

patch --dry-run -l < ionice.patch

```

d) if fine, apply

```

patch -l < ionice.patch

```

e) compile

```

gcc -o ionice ionice.c

```

f) set rights (only if you patched ionice...):

```

# chown root:root ionice && chmod u+s,-rw ionice

```

g) copy to /usr/bin (/usr/sbin for a root only version) 

```

# mv ionice /usr/bin

```

* Play with it:

I use ionice on every heavy io cron jobs.... Simply change the cron job scripts (/etc/cron.*) to prefix the calls to program with ionice...( do some backups first... :Wink:  )

Some infos:

* Can be used with nice

* 3 prio:

	RT (real time)  ==> -c1 (take it no matter what)

	BE (BEST EFFORT)==> -c2 (default)

	IDLE            ==> -c3 (disk will be used only if "free")

* Level: available for RT and BE, use via -nX 

Levels allows you a finer tunning of the class (prio between same class) - this is not used for idle...

Default prio are BE level 4

Check /usr/src/linux/Documentation/block/ioprio.txt for more info  :Wink: 

Examples

eg: Start an emerge sync as low priorities 

```

# ionice -c3 emerge --sync

```

Not annoying cron slocate (update-db) - /etc/cron.daily/slocate :

```

#! /bin/sh

if [ -x /usr/bin/updatedb ]

then

   if [ -f /etc/updatedb.conf ]

   then

      /usr/bin/ionice -c3 nice /usr/bin/updatedb

   else

      /usr/bin/ionice -c3 nice /usr/bin/updatedb -f proc

   fi

   chown root:locate /var/lib/slocate/slocate.db

fi

```

Have fun....

----------

## johntash

Nice..  I think I'll try this on my gentoo desktop later this week.  Looks good  :Smile: 

----------

## JuNix

Nice post!

----------

## JuNix

Just to follow up on this, this really works, very nicely indeed. So so useful

So I ran a benchmark

Completely idle disks

```
gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:

 Timing cached reads:   2012 MB in  2.00 seconds = 1004.70 MB/sec

 Timing buffered disk reads:  282 MB in  3.02 seconds =  93.47 MB/sec

```

Lots of normally prioritised diskio using chkfrag.pl

```
gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:

 Timing cached reads:   2188 MB in  2.00 seconds = 1092.19 MB/sec

 Timing buffered disk reads:   40 MB in  3.13 seconds =  12.77 MB/sec

```

So you can immediately see that while chkfrag.pl is running, I'm only getting 12.77 MB/s, so I have really degraded disk io bandwidth available, lots of expensive head seeks

and finally, chkfrag.pl is invoked with ionice -c3 ....

```
gatekeeper ~ # hdparm -T -t /dev/md2

/dev/md2:

 Timing cached reads:   2732 MB in  2.00 seconds = 1365.29 MB/sec

 Timing buffered disk reads:  310 MB in  3.02 seconds = 102.81 MB/sec

```

and for the curious

chkfrag.pl

```
#!/usr/bin/perl -w

#this script search for frag on a fs

use strict;

#number of files

my $files = 0;

#number of fragment

my $fragments = 0;

#number of fragmented files

my $fragfiles = 0;

#search fs for all file

open (FILES, "find " . $ARGV[0] . " -xdev -type f -print0 |");

$/ = "\0";

while (defined (my $file = <FILES>)) {

        PrivoxyWindowOpen(FRAG, "-|", "filefrag", $file);

        my $res = <FRAG>;

        if ($res =~ m/.*:\s+(\d+) extents? found/) {

                my $fragment = $1;

                $fragments += $fragment;

                if ($fragment > 1) {

                        $fragfiles++;

                }

                $files++;

        } else {

                print ("$res : not understand for $file.\n");

        }

        close (FRAG);

}

close (FILES);

print ( $fragfiles / $files * 100 . "% non contiguous files, " . $fragments / $files . " average fragments.\n");

```

----------

## hans0r

great howto! works like a charm   :Smile: 

----------

## pussi

ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though.

----------

## pijalu

 *pussi wrote:*   

> ionice is also avalibe from portage from sys-apps/util-linux-2.13_pre7, which is currently hard masked though.

 

Yep, but i didn't suceeded to compile this package on my system => the reason why I choose to take it directly from the kernel doc

----------

## Cinquero

Isn't there any other scheduler supporting io priorities? The CFQ scheduler produces EXTREME lags on the desktop. For example, when editing within vi, the response can take up to a minute or more.... so that solution is far from optimal, but better than nothing  :Smile: . M;aybe that is due to ext3's incapability to maintain a high total throughput rate in high disk load situations (unlike XFS)?

If you are using fcrontab, you should consider running your jobs via:

```
%hourly *        rm -f /var/spool/cron/lastrun/cron.hourly

%daily * 5-23       rm -f /var/spool/cron/lastrun/cron.daily

%weekly * 5-23      rm -f /var/spool/cron/lastrun/cron.weekly

%monthly * * *   rm -f /var/spool/cron/lastrun/cron.monthly

@ 10 if /usr/bin/test -x /usr/bin/ionice; then /usr/bin/ionice -c3 /usr/sbin/run-crons; else /usr/sbin/run-crons; fi

```

BTW: the ionice utility is also available via my scripting tools package: https://stier.dynu.com/~myportage/sys-libs/mscript-tools/

----------

## neuron

ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly.

----------

## Cinquero

 *neuron wrote:*   

> ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly.

 

Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO).

----------

## neuron

 *Cinquero wrote:*   

>  *neuron wrote:*   ionice -c3 seems a bit screwy.. I've seen it go ti 99% iowait and move EXTREAMLY slowly (keeping extreamly low latency desktop while doing it though), while -c2 -p7 works flawlessly. 
> 
> Uhm, isn't that what -c3 says? To delay any io operations as long as there are higher-prio IOs? 99% iowait tells you that there is IO going on, so that seems perfect to me (as long as there actually IS some non -c3 IO).

 

yes, but it's a tad bit extreme.

Like in I can copy stuff in about 100kb/sec, while downloading something at 1mb/sec.  It should not go anywhere near that slow.  And ionice -c2 -p7 gives me 20-30mb/sec instead  :Wink: 

----------

## bludger

This looks great. Any idea when it will be unmasked in portage?

----------

## Cinquero

Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it.

https://stier.dynu.com/~myportage/sys-libs/mscript-tools/

Question:

does anyone else experience the following problem?

1.) ionice -c3 emerge ... some pkg

2.) rsync / target (in parallel to 1)

3.) some GUI (ie. tvtime) with regular io and cpu priorities hangs as long as rsync builds the filelist... that is really ANNOYING and definitely a bug.

----------

## neuron

 *Cinquero wrote:*   

> Install my scripting tools package. It includes the ionice binary. Don't wait for Gentoo. You have no guarantee that they will ever support it.

 

it's in sys-apps/util-linux-2.13_pre7  :Wink: 

----------

## Cinquero

 *neuron wrote:*   

> it's in sys-apps/util-linux-2.13_pre7 

 

Not unmasked.

----------

## neuron

 *Cinquero wrote:*   

>  *neuron wrote:*   it's in sys-apps/util-linux-2.13_pre7  
> 
> Not unmasked.

 

no but "You have no guarantee that they will ever support it.", that's fairly unlikely  :Wink: 

----------

## Cinquero

 *neuron wrote:*   

> no but "You have no guarantee that they will ever support it.", that's fairly unlikely 

 

Conversation considered useless.

----------

## fgrosshans

Thanks a lot. It makes emerge-by-working practical on my laptop !

It just lacks the way to (io)renice an already started process. For a started emerge, for exemple, you can find the relevant PID by

```
 # ps -A | grep emerge

22066 tty2     00:00:19 emerge

```

and then ...

```
 ionice -c3 -p2206
```

----------

## pijalu

Something like this script may help

```

#!/bin/bash

cmd="$@"

if [ -z "$cmd" ]

 then

    echo "feed me something..."

    exit 0

fi

# get all options

params=`echo $@ | sed -e "s/ /\n/g" | sed -n "/^-/p"`

# get rid of extra \n

params=`echo $params`

# set a def val if needed

if [ -z "$params" ]

then

    params="-c3"

fi

# for each given prog name

for item in `echo $@ | sed "s/-.*//g" | sed "s/\ /\n/g"`

do

  # for each process hitting this

  for pid in `ps -A | grep $item | sed "s/[^ 0-9].*//g"`

  do

      echo "ionicing $item ($pid) with $params"

      ionice $params -p$pid

  done

done

```

usage: iorenice [partial prog name] [ionice options]

Note: This will take ALL hit to partial prog name (multiple should work) and apply the give class/prio (-c3 by def)

----------

## Cinquero

Uhm, my mscript-tools script allows to nice a complete process hierarchy. Just give it the parent process id:

mscript-tools NiceAll `pidof emerge`

You just need to replace nice with ionice in it.

----------

## MaratIK1

Request for ebuild added:

https://bugs.gentoo.org/show_bug.cgi?id=148519

----------

## Cinquero

Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS.

----------

## bludger

 *MaratIK1 wrote:*   

> Request for ebuild added:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=148519

 According to this link, ionice is included in schedutils 1.5.0, which appears to be already in portage.  Has anyone tried this out?Last edited by bludger on Mon Oct 02, 2006 10:10 am; edited 1 time in total

----------

## bludger

 *Cinquero wrote:*   

> Summary of my experience with the CFQ scheduler: H-O-R-R-I-B-L-E. Just plain useless. When I do not use io prios, a simple "rm -rf /var/tmp/ccache" may lock the entire desktop until it has finished.... even NFS transfers have stalled completely. I'm switching back to AS.

 

I haven't tried it out myself, so I can't really judge, but it seems contrary to the theory of CFQ as far as I understand it.  According to this link, io should be distributed fairly between all processes.  Can someone else confirm this phenomenon?  I find it strange that Redhat have made this the default scheduler, if it really is so bad.  Can you confirm that everything is running with IOPRIO_CLASS_BE (Best effort) and that nothing is running with IOPRIO_CLASS_RT (Real time)?

Maybe I'll have to try it out for myself.  BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default?  I would rather not recompile my kernel just for this test.

----------

## bludger

 *bludger wrote:*   

> Maybe I'll have to try it out for myself.  BTW if I reboot with the boot flag "elevator=cfq" will that select cfq as default?  I would rather not recompile my kernel just for this test.

 

Just found this: /usr/src/linux/Documentation/block/switching-sched.txt.  It is possible to change the scheduler on the fly with:

echo cfq > /sys/block/<device>/queue/scheduler

----------

## Cinquero

 *bludger wrote:*   

> ...

 

The problems occurred with kernel 2.6.16.29. I'm now using 2.6.18 (CFQ again because without it it is even worse on my desktop because I regularly create checksums of quite a few GB of data in the background), and, because there were significant changes to the CFQ scheduler code, I did test it again: download via NFS and du -sh /usr /var as root. And just as I started to write this message, firefox did not get any read access any more until I terminated the du process...

----------

## MaratIK

 *bludger wrote:*   

>  *MaratIK1 wrote:*   Request for ebuild added:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=148519 According to this link, ionice is included in schedutils 1.5.0, which appears to be already in portage.  Has anyone tried this out?

 

Ugly. W/o ionice.patch   :Confused: 

----------

## Cinquero

Ok, here is a screenshot of the problem that I think is due to CFQ:

https://stier.dynu.com/~moinmoin/MarksWiki/ScreenShots/200610?webnail=scheduler-insanity&action=show

All three avidemux2 processes have the same ionice priority (default, no ionice command used). However, the one with the lowest nice level (4) has the lowest CPU consumption for quite some time. I'm processing raw video data here (on an external USB2 disk, and we know that USB2 disks are rather limited by the USB2 protocol). That problems resembles exactly the one I have with "rm -rf /var/tmp/ccache" blocking firefox/desktop IO etc.

----------

## boniek

 *Con Kolivas wrote:*   

> CFQ has builtin ionice support which is automatically linked to nice level 
> 
> unless you explicitly set ionice separately.(...) The ionice support is for reads only. Due to the complex
> 
> path writes take to get to the io scheduler there is no ionice support for
> ...

 

Pretty important details about ionice from guru himself plus interesting link as a bonus: http://ckrm.sourceforge.net/

----------

## rich0

I've been using ionice for a few months now - it almost completely eliminated the IOBOUND problems I was having with my mythtv backend.  I run the backend as -c1 and my writes are essentially guaranteed to get to disk before the buffer blocks.  Likewise I can run emerge and other intensive tasks as -c3.

However, I recently started moving most of my data to a software raid device (running under lvm2).  I don't think the kernel raid drivers pay any attention to ioniceness.  I was having tons of IOBOUND errors on myth while doing disk-intensive operations like large copies.  Doing an emerge --sync or eix update also killed whatever was recording at the time.

Has anybody used ionice with linux software rid / mdadm?

----------

## kokoko3k

 *boniek wrote:*   

>  *Con Kolivas wrote:*   CFQ has builtin ionice support which is automatically linked to nice level 
> 
> unless you explicitly set ionice separately.(...) The ionice support is for reads only. Due to the complex
> 
> path writes take to get to the io scheduler there is no ionice support for
> ...

 

...Even ionice manual said:

 *Quote:*   

> Best effort.  This is the default scheduling class for any process that hasn't asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion.

 

But i really can't make it work; i wrote a simple bash script which make some benchmark using dd and different nice/ionice combinations.

The results i got are identical when comparing dd launched (in parallel and on different sectors of course) with nice -19  and nice +19, while differs significantly when using ionice.

Here is the script, to run root privileges are needed, as well as 2 partitions, just edit PART1 and PART2 variables.

http://pastebin.com/m73845a16

And here is my result:

http://pastebin.com/f6a37a60f

Is there anything i should do to 'link' nice and ionice priorities (apart from wrapper scripts of course) as stated in ionice manpage?

I'm using 2.6.22-gentoo-r1

Thanks in advance.

----------

