# Kernel upgrade + High disk I/O

## Daivil

Hello everyone,

I'm encountering a weird issue with one of my MySQL/NFS server within a web infrastructure.

I've upgraded the kernel of this server from 2.6.34 to 3.6.11 (yes, I know, long version gap...).

Anyway, this upgrade has caused a major Disk I/O behaviour as you can see on this graph :

http://94.125.167.135/diskstats_utilization-week.png

Here is the difference between my 2 kernel config files :

 *Quote:*   

> 2,4c2,3
> 
> < # Automatically generated make config: don't edit
> 
> < # Linux kernel version: 2.6.34-gentoo-r6
> ...

 

(I've exluded (is not set lines from output).

What could have caused such issue? Any idea anyone?

Thanks,

----------

## Daivil

Clue : iotop -o seems to report a high I/O usage from kjournald. This is ext3.

Is there a kernel option (or feature) that could explain such difference between 2.6.34 and 3.6.11 regarding ext3 performance/journal ?

----------

## KaZeR

Hi,

I've seen similar issues in the past which where related to the chosen IO scheduler.

Which scheduler are you using?

----------

## aCOSwt

which options do you mount your fs with ? (data= ? atime ?...)

Did you change anything about this ?

I can't be absolutely sure about this but I think I remember that sometimes since 2.6.32... possibly 2.6.34 some mount option was dropped from the default set of options.

EDIT : Yess ! it was journal-checksum

----------

## Daivil

Hi,

Thanks for your answers.

Here are my mount options :

```
rootfs on / type rootfs (rw)

/dev/root on / type ext3 (rw,noatime,errors=continue,barrier=1,data=writeback)

devtmpfs on /dev type devtmpfs (rw,relatime,size=6146948k,nr_inodes=1536737,mode=755)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)

cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)

openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)

cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)

cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)

freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

/dev/sda4 on /var type ext3 (rw,noatime)

nfsd on /proc/fs/nfsd type nfsd (rw,noexec,nosuid,nodev)

binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)

/dev/sda1 on /boot type ext2 (rw,noatime)
```

The scheduler options i'm using is :

```
#

# IO Schedulers

#

CONFIG_IOSCHED_NOOP=y

CONFIG_IOSCHED_DEADLINE=y

CONFIG_IOSCHED_CFQ=y

# CONFIG_DEFAULT_DEADLINE is not set

CONFIG_DEFAULT_CFQ=y

# CONFIG_DEFAULT_NOOP is not set

CONFIG_DEFAULT_IOSCHED="cfq"

# CONFIG_INLINE_SPIN_TRYLOCK is not set

# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set

# CONFIG_INLINE_SPIN_LOCK is not set

# CONFIG_INLINE_SPIN_LOCK_BH is not set

# CONFIG_INLINE_SPIN_LOCK_IRQ is not set

# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set

# CONFIG_INLINE_SPIN_UNLOCK_BH is not set

CONFIG_INLINE_SPIN_UNLOCK_IRQ=y

# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set

# CONFIG_INLINE_READ_TRYLOCK is not set

# CONFIG_INLINE_READ_LOCK is not set

# CONFIG_INLINE_READ_LOCK_BH is not set

# CONFIG_INLINE_READ_LOCK_IRQ is not set

# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set

CONFIG_INLINE_READ_UNLOCK=y

# CONFIG_INLINE_READ_UNLOCK_BH is not set

CONFIG_INLINE_READ_UNLOCK_IRQ=y

# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set

# CONFIG_INLINE_WRITE_TRYLOCK is not set

# CONFIG_INLINE_WRITE_LOCK is not set

# CONFIG_INLINE_WRITE_LOCK_BH is not set

# CONFIG_INLINE_WRITE_LOCK_IRQ is not set

# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set

CONFIG_INLINE_WRITE_UNLOCK=y

# CONFIG_INLINE_WRITE_UNLOCK_BH is not set

CONFIG_INLINE_WRITE_UNLOCK_IRQ=y

# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set

CONFIG_MUTEX_SPIN_ON_OWNER=y

CONFIG_FREEZER=y
```

----------

## aCOSwt

 *Daivil wrote:*   

> 
> 
> ```
> /dev/root on / type ext3 (rw,noatime,errors=continue,barrier=1,data=writeback)
> 
> ...

 

Because the graphics you posted in the OP concerns /dev/sda, can you tell which fs (root or var) is the most concerned ?

BTW you are aware of the impacts of setting barrier=1.

Did you pass from the default nobarrier to barrier =1 coincidentally with your kernel upgrade ?

----------

## smlbstcbr

I think you've finally found yourself with this:

https://forums.gentoo.org/viewtopic-t-482731-postdays-0-postorder-asc-start-0.html

https://forums.gentoo.org/viewtopic-t-793263-postdays-0-postorder-asc-start-0.html

As an advice, I've been using the Deadline scheduler and seems to be a decent solution to the unresponsiveness of the system during disk access.

----------

## Daivil

 *aCOSwt wrote:*   

>  *Daivil wrote:*   
> 
> ```
> /dev/root on / type ext3 (rw,noatime,errors=continue,barrier=1,data=writeback)
> 
> ...

 

It concerns /var/.

I'm not aware of the impact of barrier=1. What's the impact?

----------

## _______0

which program keeps the statistics and makes that graph?

----------

## Chiitoo

 *_______0 wrote:*   

> which program keeps the statistics and makes that graph?

 

I was interested of that as well, and right now, am emerging something with the faint 'Munin 2.0.8' around the bottom-centre of the image working as a clue.  ^^

----------

## KaZeR

 *Daivil wrote:*   

> 
> 
> I'm not aware of the impact of barrier=1. What's the impact?

 

Barriers are known to have a nasty impact on performances : http://lwn.net/Articles/283164/

----------

## Daivil

 *Chiitoo wrote:*   

>  *_______0 wrote:*   which program keeps the statistics and makes that graph? 
> 
> I was interested of that as well, and right now, am emerging something with the faint 'Munin 2.0.8' around the bottom-centre of the image working as a clue.  ^^

 

Yes, it's munin !

----------

## Daivil

 *smlbstcbr wrote:*   

> I think you've finally found yourself with this:
> 
> https://forums.gentoo.org/viewtopic-t-482731-postdays-0-postorder-asc-start-0.html
> 
> https://forums.gentoo.org/viewtopic-t-793263-postdays-0-postorder-asc-start-0.html
> ...

 

Changing to Deadline scheduler hasn't changed anything (at all).

----------

## Daivil

I've changed /var/ to ext2 instead of ext3 in my fstab and rebooted my server. Disk IO/Load has fallen to 30/40% instead of 90/100%.

No I need to understand why my 3.6.11 kernel is consuming so much IO/ressources for kJournald with ext3.

Anyone ?

----------

