# Performance tuning 3ware 9000 Series Cards

## Ast0r

If you have a 3ware 9000 series card, you might be missing out on some of the performance that your card/drives are capable of. For the last 4 months I have been very frustrated becasue the read-speed on my RAID5 arrays was abysmal. On my 1.2TB array (5x300GB in RAID5 on a 3ware 9550SX- :Cool:  I was getting only about 65MB/s on sequential reads (via "hdparm -tT /dev/sda"). On my 900GB array (4x300GB in RAID5 on a 3ware 9000S) I was getting around 90MB/s. Obviously something was wrong, but I could never figure out what it was!

Well, now I know: I needed to adjust the kernel's read-ahead cache settings for the device. According to the 3ware white-paper that I downloaded, you can do this as follows:

```
blockdev --setra 16384 /dev/sda
```

Once you do this, add the same line to /etc/conf.d/local.start so that the change will persist across reboots.

Remeber how I said I was getting 65MB/s on the 5 disk array? Check it out now:

```
yang ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   3532 MB in  2.00 seconds = 1765.89 MB/sec

 Timing buffered disk reads:  656 MB in  3.06 seconds = 214.65 MB/sec
```

Sweet!Last edited by Ast0r on Tue Oct 31, 2006 10:20 pm; edited 1 time in total

----------

## nianderson

But 3ware claims faster speeds.

The 3ware 9550SX PCI-X to SATA II RAID controller delivers over 800MB/sec RAID 5 reads and exceeds 380MB/sec RAID 5 writes, making it 200% faster than the industry-leading 3ware 9500S RAID controller. 

Ive got a customer with a 9550sx and he is claiming this output.

```

 hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   3684 MB in  2.00 seconds = 1840.44 MB/sec

 Timing buffered disk reads:  296 MB in  3.01 seconds =  98.48 MB/sec

```

----------

## Ast0r

 *nianderson wrote:*   

> But 3ware claims faster speeds.
> 
> The 3ware 9550SX PCI-X to SATA II RAID controller delivers over 800MB/sec RAID 5 reads and exceeds 380MB/sec RAID 5 writes, making it 200% faster than the industry-leading 3ware 9500S RAID controller. 
> 
> Ive got a customer with a 9550sx and he is claiming this output.
> ...

 

How many drives? Keep in mind that the 800MB/s top read speed is the total capability of the card. Right now my setup is limited by the speed of the disks (7200rpm) and the fact that I am using only 5 (whereas it supports 8). If you used (8) 15k RPM disks, your transfer rate should be a LOT better. Obviously, your mileage will vary according to the hardware that you connect to it.

So did your client try this? Did he get any improvement?

----------

## nianderson

the blockdev --setra we have been using for a long time. It does indeed increase performance. 

Im just trying to figure out why he has one golden machine that performs 2 times as good as others we have sent him.

----------

## Ast0r

 *nianderson wrote:*   

> the blockdev --setra we have been using for a long time. It does indeed increase performance. 
> 
> Im just trying to figure out why he has one golden machine that performs 2 times as good as others we have sent him.

 

Maybe the hard drives are better in that particular machine?

----------

## truE_

I'm getting unbearable performence from my 3ware 9550SX-4LP. I'm using 3ware's latest firmware (as of today), kernel 2.6.18.3's built-in scsi driver, and Seagate ST3750640AS 750GB drives in a raid5 configuration. (1.6TB) I'm getting write speeds of 3MB/s, and read speeds of around 25MB/s:

hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   2064 MB in  2.00 seconds = 1031.50 MB/sec

 Timing buffered disk reads:   74 MB in  3.03 seconds =  24.39 MB/sec

I've replaced the backplane, tried other drives (same model), and multiple different kernel versions. Same thing...

I did the blockdev command as a test and speeds did not change.

Anyone have any ideas? I'm wondering if its these drives...

----------

## Ast0r

Perhaps you have a configuration problem on your card? You have you tried experimenting with the settings in the 3ware card's BIOS?

----------

## nianderson

Do you have  a BBU?

If you dont, I am not positive that turning write-caching on will actually do anything. Ive seen that before but anyway try the following

Download both tw_cli and 3dm2. Check for errors in 3dm.

In tw_cli set storsave to perform, set queuing to off.

What motherboard is this in? Are you using a riser?

----------

## truE_

Thanks for all of your help and suggestions -- I am working dilegently to resolve this.

I was getting very poor performance with the default BIOS options (I simply did a Firmware upgrade when first installed).

I am still getting very poor performance after doing what you guys suggested:

hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   2088 MB in  2.00 seconds = 1042.61 MB/sec

 Timing buffered disk reads:   90 MB in  3.03 seconds =  29.71 MB/sec

This is with storsave set to "Performance", Queuing "Off", and write-caching "off". Since I don't have a BBU (I ordered one, but got shipped the 9500S one), write caching should really make no performance differences.

I'm using an Intel SR1450 Chassis, with an Intel SE7520JR2DDR2 motherboard and an Intel PCIX riser (ADWPCIXR).

Like I said, I've tried different drive sets (of the same model drive), another backplane, and all new SATA cables. Maybe the riser is hindering performance?

Any other ideas from anyone? I am using the exact same configuration with SCSI drives and LSILogic raid controllers, and the machines are smoking fast. I am also getting tons of CPU when doing writes. I'll include a few more examples below:

My RAID1 LSI SCSI machines:

time dd if=/dev/zero of=testfile bs=1024k count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 2.16581 seconds, 496 MB/s

real    0m2.168s

user    0m0.000s

sys     0m2.170s

My RAID5 3WARE SATA PCI-X

Still running after 10+ MINUTES (took 2 seconds with the other machine!!) w/ an 8+ CPU Load....  Will update this thread if/when it finishes.

Of course I expect a drastic difference between SCSI and Raid 1 vs. 5 -- but this is tragic, something hardware-wise must not be in good shape...

Thank you very much.

----------

## nianderson

I would try bypassing the riser, we have seen numerous issues with risers.

3ware does reccomend active risers, is that riser passive or active?

In fact I have a 3ware 9550sx series with 4 drives in our lab right now.

Its in the middle of a test but Ill get the hdparm output for ya as well as the drive models

----------

## truE_

 *nianderson wrote:*   

> I would try bypassing the riser, we have seen numerous issues with risers.
> 
> 3ware does reccomend active risers, is that riser passive or active?
> 
> In fact I have a 3ware 9550sx series with 4 drives in our lab right now.
> ...

 

Thanks again, I appreciate it.

I will try and remove the riser and see if there is any difference. I could not find if it was an Active or Passive riser -- for reference, here is a link to it through newegg.com (for picture reasons)

http://www.newegg.com/Product/Product.asp?Item=N82E16816117029

Lastly, I would greatly appreciate an hdparm w/ raid 5 using that controller.  (also, is it PCI-X?)

----------

## nianderson

That looks like a passive riser to me.

The machine I have in the lab is still running heavy disk tests.

so this is under heavy disk load.

 hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   10200 MB in  2.00 seconds = 5100.78 MB/sec

 Timing buffered disk reads:   54 MB in  3.01 seconds =  17.92 MB/sec

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC

------------------------------------------------------------------------------

u0    RAID-5    VERIFYING      18     64K     419.065   OFF    OFF      OFF     

Port   Status           Unit   Size        Blocks        Serial

---------------------------------------------------------------

p0     OK               u0     139.73 GB   293046768     WD-WMAP41189376

p1     OK               u0     139.73 GB   293046768     WD-WMAP41142747

p2     OK               u0     139.73 GB   293046768     WD-WMAP41146945

p3     OK               u0     139.73 GB   293046768     WD-WMAP41184828

as you notice its also verifying so that bench is pretty much useless  sorry  :Smile: 

Ill try to remeber to get the bench and post it before we ship it out.

----------

## -Craig-

You *REALLY* need to enable write caching, it makes a *BIG* difference!!!!

----------

## WhiteSpade

I've been looking around 3ware's site and I found this:

 *Quote:*   

> blockdev --setra 16384 /dev/sda
> 
> (Note: 16384 is just an example value.  You will have to do testing to determine the optimal value for your system).  The OS will read-ahead X pages, and throughput will be higher. 

 

So... I am looking forward to experimenting with a few different values.  I know some people have achieved enormous gains using the suggested value while others have only seen modest gains.  I wonder if different values will help.  I'll let you all know if I find anything.

---Alex

----------

## Ast0r

 *WhiteSpade wrote:*   

> I've been looking around 3ware's site and I found this:
> 
>  *Quote:*   blockdev --setra 16384 /dev/sda
> 
> (Note: 16384 is just an example value.  You will have to do testing to determine the optimal value for your system).  The OS will read-ahead X pages, and throughput will be higher.  
> ...

 

A friend of mine wrote a shell script to try different values and figure out which is best. What we determined was that 16384 is hard to beat in terms of overall performance, at least on the 9xxx series cards that we have.

If you are interested, here is the shell script that he wrote:

```
#!/bin/bash

START=0

INCREMENT=16

ALLOWDRIFT="50.0"

speed=0

try=$(( $START - $INCREMENT ))

keepgoing=true

topspeed=0

topra=0

while [[ "${keepgoing}" == "true" ]]; do

   try=$(( $try + $INCREMENT ))

   echo -n "Trying $try readahead..."

   blockdev --setra $try /dev/sda

   speed=$(hdparm -t /dev/sda | tail -n 1 | awk '{print $11}')

   echo -n " ${speed} MB/sec"

# Hmm...

   if [ "$(echo "scale=3; ${speed} > ${topspeed}" | bc )" -eq '1' ]; then

      echo -e "\t** TOP SPEED"

      topspeed="${speed}"

      topra="${try}"

      keepgoing=true

   else

      percdiff=$(echo -e "scale=3\n100 - ( ( ${speed} / ${topspeed} ) * 100 )\nquit" | bc)

      if [ "$(echo "scale=3; ${percdiff} < ${ALLOWDRIFT}" | bc)" -eq '1' ]; then

         echo -e "\t${percdiff}% slower"

         keepgoing=true

      else

         echo -e "\t${percdiff}% slower, bailing..."

         keepgoing=false

      fi

   fi

done

echo -e "Looks like the best speed of ${topspeed} MB/sec is achieved with a readahead of ${topra}.\n"
```

I would be interested if you find that another value works better for you. If so, please post which card you are using, how many drives, array configuration, etc.

Also, keep in mind that this script only increments by 16 blocks each recursion and the hdparm takes about 12 seconds per run, so you need about 12 * 2000 seconds just to hit 32768 (double the recommended value from 3ware). You will either want to change the granularity or run this over night (I recommend the latter if you are serious about finding the best value).

----------

## -Craig-

For me, "blockdev --setra 16384 /dev/sda" doesn't make any difference.

----------

## dtlgc

My before and after on 3ware card - plain mirrored drives

ic conf.d # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   4308 MB in  2.00 seconds = 2153.87 MB/sec

 Timing buffered disk reads:  164 MB in  3.02 seconds =  54.37 MB/sec

ic conf.d # blockdev --setra 16384 /dev/sda

ic conf.d # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   4328 MB in  2.00 seconds = 2163.86 MB/sec

 Timing buffered disk reads:  240 MB in  3.01 seconds =  79.68 MB/sec

----------

## col

 *-Craig- wrote:*   

> You *REALLY* need to enable write caching, it makes a *BIG* difference!!!!

 

I remember reading somewhere that this can be very dangerous if your using a journaling file system ?

----------

## Bo_Oris

Verschoben

----------

## MentholMoose

Thanks for the tip. 9550SX-12 with 7x 160GB Seagate drives.

root@knightboat ~ # blockdev --getra /dev/sda

256

root@knightboat ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   1864 MB in  2.00 seconds = 932.62 MB/sec

 Timing buffered disk reads:  368 MB in  3.00 seconds = 122.61 MB/sec

root@knightboat ~ # blockdev --setra 16384 /dev/sda

root@knightboat ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   1882 MB in  2.00 seconds = 941.10 MB/sec

 Timing buffered disk reads:  672 MB in  3.01 seconds = 223.39 MB/sec

----------

## NightMonkey

Some very interesting legwork done here:

* http://www.makarevitch.org/rant/3ware/

And some interesting discussion and comparative benchmarks here:

* http://forums.storagereview.net/index.php?showtopic=25923&pid=245225&mode=threaded&start=#entry245225

----------

## fixinko

I've using 9550SXU-4 with 4xWDC5001ABYS (RAID-5). Read performance is quite good, but on database load or moving big files (write operations), load goes to 5-12 (big i/o wait), and the whole system is lazy. I've read this http://www.3ware.com/KB/article.aspx?id=11050 article, but those settings, doesn't solve my problem. What did I wrong? My actual settings:

hdparm:

```

phoenix ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   6388 MB in  2.00 seconds = 3196.67 MB/sec

 Timing buffered disk reads:  336 MB in  3.02 seconds = 111.25 MB/sec

```

tw_cli:

```

//phoenix> /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy

------------------------------------------------------------------------------

u0    RAID-5    OK             -       -       64K     1396.95   ON     OFF

Port   Status           Unit   Size        Blocks        Serial

---------------------------------------------------------------

p0     OK               u0     465.76 GB   976773168     WD-WCAS83789493

p1     OK               u0     465.76 GB   976773168     WD-WCAS83782829

p2     OK               u0     465.76 GB   976773168     WD-WCAS83799487

p3     OK               u0     465.76 GB   976773168     WD-WCAS85160709

//phoenix> /c0 show all

/c0 Driver Version = 2.26.02.010

/c0 Model = 9550SXU-4LP

/c0 Available Memory = 112MB

/c0 Firmware Version = FE9X 3.04.00.005

/c0 Bios Version = BE9X 3.04.00.002

/c0 Boot Loader Version = BL9X 3.02.00.001

/c0 Serial Number = L320910A7441596

/c0 PCB Version = Rev 032

/c0 PCHIP Version = 1.60

/c0 ACHIP Version = 1.90

/c0 Number of Ports = 4

/c0 Number of Drives = 4

/c0 Number of Units = 1

/c0 Total Optimal Units = 1

/c0 Not Optimal Units = 0

/c0 JBOD Export Policy = off

/c0 Disk Spinup Policy = 1

/c0 Spinup Stagger Time Policy (sec) = 2

/c0 Auto-Carving Policy = off

/c0 Auto-Carving Size = 2048 GB

/c0 Auto-Rebuild Policy = on

/c0 Controller Bus Type = PCIX

/c0 Controller Bus Width = 64 bits

/c0 Controller Bus Speed = 133 Mhz

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy

------------------------------------------------------------------------------

u0    RAID-5    OK             -       -       64K     1396.95   ON     OFF

Port   Status           Unit   Size        Blocks        Serial

---------------------------------------------------------------

p0     OK               u0     465.76 GB   976773168     WD-WCAS83789493

p1     OK               u0     465.76 GB   976773168     WD-WCAS83782829

p2     OK               u0     465.76 GB   976773168     WD-WCAS83799487

p3     OK               u0     465.76 GB   976773168     WD-WCAS85160709

```

blockdev and /sys:

```

phoenix ~ # cat /sys/block/sda/queue/max_sectors_kb

128

phoenix ~ # cat /sys/block/sda/queue/nr_requests

8192

phoenix ~ # cat /proc/sys/vm/page-cluster

3

phoenix ~ # blockdev --report /dev/sda

RO    RA   SSZ   BSZ   StartSec     Size    Device

rw 16384   512  4096          0 2929625088  /dev/sda

phoenix ~ # cat /proc/sys/vm/dirty_background_ratio

20

phoenix ~ # cat /proc/sys/vm/dirty_ratio

60

phoenix ~ # cat /sys/block/sda/queue/scheduler

noop anticipatory [deadline]

```

/proc/interrupts:

```

phoenix ~ # cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3

  0:   58053879          0          0          0   IO-APIC-edge      timer

  1:          2          0          0          0   IO-APIC-edge      i8042

  4:        165          0          0          0   IO-APIC-edge      serial

  8:          0          0          0          0   IO-APIC-edge      rtc

  9:          0          0          0          0   IO-APIC-fasteoi   acpi

 12:          3          0          0          0   IO-APIC-edge      i8042

 14:         78          0          0          0   IO-APIC-edge      ide0

 19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3

 23:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2

 24:    6256880          0          0          0   IO-APIC-fasteoi   3w-9xxx

380:   65061197          0          0          0   PCI-MSI-edge      eth0

NMI:          0          0          0          0

LOC:   57333954   57333952   57534123   57533999

ERR:          0

```

uname:

```

phoenix ~ # uname -a

Linux phoenix 2.6.23-hardened-r8 #1 SMP Mon Mar 17 08:10:52 CET 2008 x86_64 Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux

```

----------

## NightMonkey

Try using IOZone as your benchmarking tool. It is in Portage. All of the tools you've used are read-only tools. You have no data shown about your write performance.

FYI: I just kicked some 3ware cards out of the servers I was deploying, and replaced them with PCI-Express Areca cards. Incredible improvement in performance, both write and read, especially random write performance, which was our problem area (RAID 5). It doesn't work miracles, but it doesn't put much in the way between Linux and the disks.  If I have a choice, I'll not choose 3ware again until I hear they've cleaned up their act.

 *fixinko wrote:*   

> I've using 9550SXU-4 with 4xWDC5001ABYS (RAID-5). Read performance is quite good, but on database load or moving big files (write operations), load goes to 5-12 (big i/o wait), and the whole system is lazy. I've read this http://www.3ware.com/KB/article.aspx?id=11050 article, but those settings, doesn't solve my problem. What did I wrong? My actual settings:
> 
> hdparm:
> 
> ```
> ...

 

----------

## fixinko

 *NightMonkey wrote:*   

> Try using IOZone as your benchmarking tool. It is in Portage. All of the tools you've used are read-only tools. You have no data shown about your write performance.
> 
> FYI: I just kicked some 3ware cards out of the servers I was deploying, and replaced them with PCI-Express Areca cards. Incredible improvement in performance, both write and read, especially random write performance, which was our problem area (RAID 5). It doesn't work miracles, but it doesn't put much in the way between Linux and the disks.  If I have a choice, I'll not choose 3ware again until I hear they've cleaned up their act.
> 
> 

 

I haven't problem with write performance, but with big i/o waits on write operations, so therefore I didn't show data about my write performance...

----------

## NightMonkey

If you haven't already, see my links posted above ( http://forums.storagereview.net/index.php?showtopic=25923&pid=249105&mode=threaded&start=#entry249105 ) . Seems to deal specifically with the symptoms you are describing.

 *fixinko wrote:*   

>  *NightMonkey wrote:*   Try using IOZone as your benchmarking tool. It is in Portage. All of the tools you've used are read-only tools. You have no data shown about your write performance.
> 
> FYI: I just kicked some 3ware cards out of the servers I was deploying, and replaced them with PCI-Express Areca cards. Incredible improvement in performance, both write and read, especially random write performance, which was our problem area (RAID 5). It doesn't work miracles, but it doesn't put much in the way between Linux and the disks.  If I have a choice, I'll not choose 3ware again until I hear they've cleaned up their act.
> 
>  
> ...

 

----------

## fixinko

 *NightMonkey wrote:*   

> If you haven't already, see my links posted above ( http://forums.storagereview.net/index.php?showtopic=25923&pid=249105&mode=threaded&start=#entry249105 ) . Seems to deal specifically with the symptoms you are describing.
> 
> 

 

I've tried set --setra to 0, but with no effect... 3ware support give me advise to upgrade firmware, so I'm going tomorrow upgrade the firmware. I hope it will be ok  :Smile: 

----------

