# PE 2650 aacraid custom kernel very SLOW Write performance

## mr_friend

Hello Everybody,

I have been pulling my hair out for the last few days trying to get the system running normally. This is not my first custom gentoo kernel...

From the liveCD 2007.0 the write performance is good(afaik):

```

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write          40 MBs |    0.4 s | 102.049 MB/s |   0.0 %  | 326.6 % |

| Random Write   16 MBs |    0.6 s |  26.419 MB/s |   0.0 %  |  60.9 % |

| Read           40 MBs |    0.0 s | 2173.204 MB/s | 217.3 %  | 652.0 % |

| Random Read    16 MBs |    0.0 s | 2073.656 MB/s |   0.0 %  |   0.0 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.030 ms |        0.081 ms |  0.00000 |   0.00000 |

| Random Write |        0.020 ms |        0.074 ms |  0.00000 |   0.00000 |

| Read         |        0.007 ms |        0.064 ms |  0.00000 |   0.00000 |

| Random Read  |        0.006 ms |        0.024 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.017 ms |        0.081 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

```

But, from my kernel 2.6.23-gentoo-r3 it is SLOW

```

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write          40 MBs |   12.4 s |   3.223 MB/s |   0.3 %  |   4.9 % |

| Random Write   16 MBs |   18.8 s |   0.831 MB/s |   0.0 %  |   2.0 % |

| Read           40 MBs |    0.0 s | 2000.000 MB/s |   0.0 %  | 600.0 % |

| Random Read    16 MBs |    0.0 s | 1562.344 MB/s |   0.0 %  | 400.0 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.015 ms |       40.001 ms |  0.00000 |   0.00000 |

| Random Write |        0.005 ms |       10.000 ms |  0.00000 |   0.00000 |

| Read         |        0.004 ms |       10.000 ms |  0.00000 |   0.00000 |

| Random Read  |        0.003 ms |       10.001 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.008 ms |       40.001 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

```

It seems ODD that only the write performance is suffering... everything slows  down to a crawl...

Here is the kernel .config 2.6.23-gentoo-r3

Here is my older kernel config that used to work fine: .config 2.6.17-gentoo-r4

Here are the modules that get loaded:

```

rtc                    14232  0 

ata_generic             8708  0 

pata_serverworks       10624  0 

evdev                  10880  0 

tg3                   102164  0 

ohci_hcd               30860  0 

usbcore               136016  2 ohci_hcd

8250_pci               23040  0 

8250                   32900  1 8250_pci

serial_core            22296  1 8250

serverworks             8068  0 [permanent]

sworks_agp             10784  0 

agpgart                33288  1 sworks_agp

```

Here is the lspci -v output. 

Here is the dmesg output, with all kinds of debug info.

Somebody, PLEASE HELP ME. I am going crazy...

I will paypal $50 to whoever gets this resolved for me  :Smile:  as a token of appreciation!!!.

----------

## mr_friend

Hello everybody,

I got ~100 views of the thread  :Smile:  but, no replies... 

Here's how I resolved the issue... take a quess

I went back to the 2.6.19-gentoo-r5 kernel, which has an older version of the aacraid driver...  :Smile: 

----------

## mendo

I'm having the exact same problem. I have an Adaptec 31205 SAS raid controller and are using the aacraid driver. I test the speed by just unpacking the linux kernel source and delete it again. I have found that the change happened between linux-2.6.22 and linux-2.6.23.

2.6.22:

Unpack: 22.3 seconds.

Delete: 1.8 seconds.

2.6.23:

Unpack: 327 seconds.

Delete: 258 seconds.

The times include flush to disk with the sync command.

I have just tested again with gentoo-sources-2.6.24-r3. That's the same as with 2.6.23.

I have noticed the following difference in the log:

2.6.22:

Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)

Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming Write Enabled

Feb 25 12:52:42 dexter kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through

2.6.24:

Mar  6 13:55:01 dexter kernel: Driver 'sd' needs updating - please use bus_type methods

Mar  6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] 2925506560 512-byte hardware sectors (1497859 MB)

Mar  6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write Protect is off

Mar  6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00

Mar  6 13:55:01 dexter kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA

My controller is installed in a dual opteron dualcore machine. What about yours?

What firmware version do you have on your controller? I have 5.2-0 (15323).

I'm going to try vanilla-sources too.

----------

## mendo

I have now tried linux kernel 2.6.25-rc4 (vanilla from kernel.org) to be sure that it hasn't anything with gentoo to do. I also upgraded my controller firmware to build 15611 (the latest from Adaptec) but nothing helped.

I have however found out something else. I'm using the xfs filesystem and I found that if i mount the filesystem with the "nobarrier" option the performance is back to normal. It could seem that the barrier thing in xfs is relying on a feature which has disappeared or got broken in later versions of aacraid.

Actually I should have thought of this earlier. For some time ago I worked with a new iSCSI based raid array. Here I ran into the same problem.

It seems to be a problem with many harddrive controllers. Even on my laptop I have run into this problem. I believe the problem first appeared in kernel 2.6.17. At that time I didn't know about the nobarrier option but now I just tried it and it was in fact that problem again.

I think I can see in your dmesg output that you're also using xfs. Try the nobarrier option!

----------

## Poseidon

FFS   :Shocked:  , I have two poweredge 2650 servers and have done everything in my power to get them working up to speed.  I was sure the PERC controller or memory on it was fried at this point.  I of course use XFS with Raid-5 for all our media / file server operations for performance reasons.  It slowed down awhile ago (prob when I upgraded the kernel), but couldn't figure it out.

You just saved me two servers.  I wish I'd read this sooner or you'd have saved me a lot of diagnosing time and hours of sleep.

 :Cool: 

----------

