# 3ware 9650 performance [need help]

## nianderson

Ive got 6 3 Gb/sec drives that I had configured in RAID6. Things seemed to be going a bit slow so I ran bonnie++ on it with a 4G run (I have 2G memory) I was seeing under 5Mb/s! I switched to RAID5+HS and now Im seeing a whopping 42ishMb/s. I have the readahead set and I am running 2.6.19-gentoo-r5 sources. Does anyone have any ideas as to what the slowdown could be?Last edited by nianderson on Tue May 01, 2007 1:24 am; edited 1 time in total

----------

## nianderson

So Ive tried 2.6.20-gentoo-r7 now also with no real difference. I also took the driver out of the kernel downloaded the source from 3wares site and tried building several versions by hand and loading those. I really saw no performance difference.

Somewhere along the line I either miscalculated or I got a small increase in performance.

```

dd if=/dev/zero of=11G_file bs=65536 count=163840

163840+0 records in

163840+0 records out

10737418240 bytes (11 GB) copied, 156.309 s, 68.7 MB/s

dd if=/dev/zero of=11G_file bs=65536 count=163840

163840+0 records in

163840+0 records out

10737418240 bytes (11 GB) copied, 145.184 s, 74.0 MB/s

```

So on creation of 11GB im getting between 68.7MB/s and 75MB/s first with queuing off, second with queuing on.

I've got 5 drives in raid 5. If you assume 80% efficiency of their advertised max thats 60MB/s sustained. So 4*60MB/s is 240MB/s assuming no slow down due to raid. However if you consider that you might get 80% efficiency due to raid thats around 192MB/s. Am I any where close on this or am I just pulling these numbers out of my @ss?

----------

## drescherjm

 *Quote:*   

> However if you consider that you might get 80% efficiency due to raid thats around 192MB/s. Am I any where close on this or am I just pulling these numbers out of my @ss?

 

I get around 267MB/s (hdparm buffered disk read) on a linux software raid 6 with 6 320GB Seagate 7200.10 on an asus am2 mobo with Athlon64 3200 processor with 2 GB DDR2 memory at work. Writes are around 80MB/s for large files which I believe is a bug as the cpu usage is less that 7% writes.

----------

## nianderson

```

hdparm -tT /dev/sdc

/dev/sdc:

 Timing cached reads:   1644 MB in  2.00 seconds = 822.47 MB/sec

 Timing buffered disk reads:  498 MB in  3.00 seconds = 165.76 MB/sec

```

this still seems slow to me.

Im hoping the drives are jumpered to 1.5Gb/s and it just was not caught by the system builder. 

What driver version are you using?

----------

## drescherjm

I remember the ASUS AM2 board having 2500 MB/s in the cached result but that is more a function of system/memory speed and not disk or raid speed. 

For me the SATA1 / SATA2 setting does not make much of a performance difference since no single drive can operate anywhere near at 150MB/s the added bandwidth of 3GB / s ready does not do much. However I do admit SATA2 does have other non bandwidth benefits such as NCQ

BTW, I do remember reading (I believe on the mythtv-users list) about poor write performance on the 3ware cards being caused by the hardware cache on the card being off by default.

----------

## nianderson

So I have learned a bit more on this now.

Aparently some motherboards ship with an improper pci setting. 

according to a 3ware knowledgebase article found here: 

http://www.3ware.com/KB/article.aspx?id=12546

if Memory Write and Invalidate Enable is not set write performance will be 

suboptimal.

According to lspci -vvv 

02:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID (rev 01)

        Subsystem: 3ware Inc 9650SE SATA-II RAID

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 

Stepping- SERR+ FastB2B-

This shows MemWINV- which I am thnking means Memory Write and Invalidate 

disable.

----------

## nianderson

yes in case you were possibly wondering I am *still* working on this issue.

I have been through the paces with 3ware, tyan, asus, and seagate.

I have tried switching to an asus kfn4-dre board with no success ..... speeds still hovering around 60-70MB/s

I am not crazy there is something awry here.

Ive tried a plethora of kernels and tweaks in the 3ware KB all to no avail.

Dont shoot me here but now im even on debian etch with the machine back in the vendor lab trying to resolve the issue.  As of yet 3ware has been of little help. I have a new contact there that I am hoping will shed new light.

Anyone else care to take a stab?

Yes writecaching is on .... without it im getting 8.2 MB/s

----------

## nianderson

Ok .... slowly makeing progress on this if anyone cares .....

It might be an unconfirmed firmware bug. If anyone else is having issues we should get in touch so we can compare notes and run it up the flag pole.

----------

## kinkozmasta

I think I might be having a similar problem.

https://forums.gentoo.org/viewtopic-t-535223-highlight-3ware.html

I gave up trying to fix it a long time ago however. If you've made any progress or found anything out I would be very interested to know.

----------

## snIP3r

hi all!

i am also very interessted in speeding up my hw raid5.

i run a 3ware 9650SE-4LP (hw cache is on, firmware/bios is FE9X 3.06.00.003, driver 2.26.02.008) with 3 WD3200RE 320GB harddrives on an AMD64 EE 3800+ with an  ASUS M2N SLI-Deluxe Board (0903 bios) and 2gig ddr800 ram. kernel is 2.6.20-gentoo-r8.

as suggested on a 3ware doc i set some options:

```

echo "64" > /sys/block/sda/queue/max_sectors_kb

echo "512" > /sys/block/sda/queue/nr_requests

echo "deadline" > /sys/block/sda/queue/scheduler

blockdev --setra 16384 /dev/sda

```

my root was formated xfs and hdparm -tT /dev/sda3 (the root partition) gives me:

```

area52 ~ # hdparm

hdparm -Tt /dev/sda3

/dev/sda3:

 Timing cached reads:   2400 MB in  2.00 seconds = 1200.62 MB/sec

 Timing buffered disk reads:  368 MB in  3.03 seconds = 121.42 MB/sec

```

and bonnie++ gives me:

```

Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

area52           4G   564  98 78020  18 31634  14   856  99 115089  26 288.5   8

Latency             27343us    2608ms     733ms   27384us     115ms     165ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

area52              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16   198   2 +++++ +++   225   2   207   2 +++++ +++   158   1

Latency               417ms    1071us     349ms     286ms     619us    1164ms

```

i think thats a

write 78 mb/s

rewrite 31 mb/s

read 115 mb/s

?????

lspci-vvv returns:

```

02:00.0 RAID bus controller: 3ware Inc Unknown device 1004 (rev 01)

        Subsystem: 3ware Inc Unknown device 1004

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-

        Latency: 0, Cache Line Size: 32 bytes

        Interrupt: pin A routed to IRQ 16

        Region 0: Memory at fa000000 (64-bit, prefetchable) [size=32M]

        Region 2: Memory at fdeff000 (64-bit, non-prefetchable) [size=4K]

        Region 4: I/O ports at 9c00 [size=256]

        Region 5: Memory at fdefe000 (32-bit, non-prefetchable) [size=4K]

        Expansion ROM at fdec0000 [disabled] [size=128K]

        Capabilities: [40] Power Management version 2

                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/5 Enable-

                Address: 0000000000000000  Data: 0000

        Capabilities: [70] Express Legacy Endpoint IRQ 0

                Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-

                Device: Latency L0s <128ns, L1 <2us

                Device: AtnBtn- AtnInd- PwrInd-

                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

                Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+

                Device: MaxPayload 256 bytes, MaxReadReq 512 bytes

                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0

                Link: Latency L0s <512ns, L1 <64us

                Link: ASPM Disabled RCB 128 bytes CommClk- ExtSynch-

                Link: Speed 2.5Gb/s, Width x4

        Capabilities: [100] Advanced Error Reporting

```

3ware has released a new firmware (http://www.3ware.com/download/Escalade9650SE-Series/9.4.1.2/9.4.1.2-9xxx-Upgrade.zip). can i update to this firmware without getting problems with my gentoo installation (especially the kernel driver)???

greets

snIP3r

----------

## mmikrut

Hello All,

I was wrestling with the same problem for a while now.   Although I am running Debian Etch (sorry for trespassing on your forums!  :Smile:  )  the symptoms are the same.  I am running Raid5 with 3 segate 500GB drives on 2.6.21.6 kernel, however my write performance is hovering at 60Mb/s at its best.

I believe the problem is in 3ware firmware (or Linux driver) code and it is related to kernel versions above 2.6.18.    

When booting with 2.6.18 kernel, I am getting write speeds between 90 and 110Mb/s  which I believe is acceptable on my current setup.  Booting with 2.6.20 and 2.6.21 kernels, my write speed varies from 30 to 60Mb/s.

The test results also vary between kernel versions 2.6.18 & 2.6.21 vary depending on amount of RAM available for caching.   

The best results  were achieved with  max_sectors_kb = 128 nr_requests = 8192  and blockdev  --setra 16384, but this may depend on hardware configuration. I also set Storsave Policy = performance on the controler, this gives an extra 10-20% imrprovement  at a slight risk of loosing data in event of power failure, if you run without backup battery unit:)

If someone can confirm this performance degradation in Kernels > 2.6.18 then I think it would be a good idea to point this out to 3ware people and ask for a fix.   Can someone please run some tests to confirm that?

Here are my results with kernel 2.6.18 & kernel bootparam mem=128M  to limit caching of results in RAM.

# hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   7282 MB in  2.00 seconds = 3644.75 MB/sec

 Timing buffered disk reads:  284 MB in  3.01 seconds =  94.42 MB/sec

# ./hdReadWriteTest / sda 128 8192 16384 1M 3K

Settings are:

max_hw_sectors_kb = 128 max_sectors_kb = 128 nr_requests = 8192 read_ahead_kb = 8192 16384

WRITE SPEED TEST

sync; time { dd if=/dev/zero of=//testfile bs=1M count=3K; sync; }

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 28.3321 seconds, 114 MB/s

real    0m29.727s

user    0m0.008s

sys     0m4.416s

READ SPEED TEST

time dd if=//testfile of=/dev/null bs=1M count=3K

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 40.7132 seconds, 79.1 MB/s

real    0m40.787s

user    0m0.004s

sys     0m2.700s

Here are some tests with kernel 2.6.18 & full 4GB RAM

# hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   9598 MB in  2.00 seconds = 4805.53 MB/sec

 Timing buffered disk reads:  436 MB in  3.00 seconds = 145.15 MB/sec

# ./hdReadWriteTest / sda 128 8192 16384 1M 3K

Settings are:

max_hw_sectors_kb = 128 max_sectors_kb = 128 nr_requests = 8192 read_ahead_kb = 8192 16384

WRITE SPEED TEST

sync; time { dd if=/dev/zero of=//testfile bs=1M count=3K; sync; }

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 15.1917 seconds, 212 MB/s

real    0m29.533s

user    0m0.008s

sys     0m4.276s

READ SPEED TEST

time dd if=//testfile of=/dev/null bs=1M count=3K

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 1.03761 seconds, 3.1 GB/s

real    0m1.039s

user    0m0.000s

sys     0m1.040s

Now, test results when booting with kernel 2.6.21.6, mem=128M

# hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   7316 MB in  2.00 seconds = 3663.08 MB/sec

 Timing buffered disk reads:  104 MB in  3.45 seconds =  30.16 MB/sec

# ./hdReadWriteTest / sda 128 8192 16384 1M 3K

Settings are:

max_hw_sectors_kb = 128 max_sectors_kb = 128 nr_requests = 8192 read_ahead_kb = 8192 16384

WRITE SPEED TEST

sync; time { dd if=/dev/zero of=//testfile bs=1M count=3K; sync; }

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 62.134 seconds, 51.8 MB/s

real    1m4.097s

user    0m0.004s

sys     0m4.300s

READ SPEED TEST

time dd if=//testfile of=/dev/null bs=1M count=3K

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 31.7436 seconds, 101 MB/s

real    0m32.056s

user    0m0.003s

sys     0m3.089s

Test results when booting with kernel 2.6.21.6 & 4GB RAM

# hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   7862 MB in  2.00 seconds = 3936.98 MB/sec

 Timing buffered disk reads:  436 MB in  3.00 seconds = 145.19 MB/sec

# ./hdReadWriteTest / sda 128 8192 16384 1M 3K

Settings are:

max_hw_sectors_kb = 128 max_sectors_kb = 128 nr_requests = 8192 read_ahead_kb = 8192 16384

WRITE SPEED TEST

sync; time { dd if=/dev/zero of=//testfile bs=1M count=3K; sync; }

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 55.8994 seconds, 57.6 MB/s

real    1m5.844s

user    0m0.003s

sys     0m4.237s

READ SPEED TEST

time dd if=//testfile of=/dev/null bs=1M count=3K

3072+0 records in

3072+0 records out

3221225472 bytes (3.2 GB) copied, 1.04852 seconds, 3.1 GB/s

real    0m1.050s

user    0m0.001s

sys     0m1.049s

Finally, here are my bios/firmware/driver version numbers: 

/c0 Driver Version = 2.26.06.002-2.6.19

/c0 Model = 9650SE-4LPML

/c0 Memory Installed  = 224MB

/c0 Firmware Version = FE9X 3.08.02.005

/c0 Bios Version = BE9X 3.08.00.002

/c0 Monitor Version = BL9X 3.05.00.002

/c0 Serial Number = L326001A7010371

/c0 PCB Version = Rev 032

/c0 PCHIP Version = 2.00

/c0 ACHIP Version = 1.90

/c0 Number of Ports = 4

/c0 Number of Units = 1

/c0 Number of Drives = 3

/c0 Total Optimal Units = 1

/c0 Not Optimal Units = 0

/c0 JBOD Export Policy = off

/c0 Disk Spinup Policy = 1

/c0 Spinup Stagger Time Policy (sec) = 1

/c0 Auto-Carving Policy = off

/c0 Auto-Carving Size = 2048 GB

/c0 Auto-Rebuild Policy = on

/c0 Controller Bus Type = PCIe

/c0 Controller Bus Width = 4 lanes

/c0 Controller Bus Speed = 2.5 Gbps/lane

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy

------------------------------------------------------------------------------

u0    RAID-5    OK             -       -       256K    931.303   ON     OFF

Port   Status           Unit   Size        Blocks        Serial

---------------------------------------------------------------

p0     NOT-PRESENT      -      -           -             -

p1     OK               u0     465.76 GB   976773168     9QG0HJRT

p2     OK               u0     465.76 GB   976773168     9QG1NN9Y

p3     OK               u0     465.76 GB   976773168     9QG1S8V2

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest

---------------------------------------------------------------------------

bbu   On           Yes       OK        OK       OK       0      xx-xxx-xxxx

/c0/u0 status = OK

/c0/u0 is not rebuilding, its current state is OK

/c0/u0 is not verifying, its current state is OK

/c0/u0 is initialized.

/c0/u0 Cache State = on

/c0/u0 volume(s) = 1

/c0/u0 name =

/c0/u0 serial number = 9QG0HJRT906AC1001FF5

/c0/u0 Ignore ECC policy = off

/c0/u0 Auto Verify Policy = off

/c0/u0 Storsave Policy = performance

/c0/u0 Command Queuing Policy = on

Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)

------------------------------------------------------------------------

u0       RAID-5    OK             -       -       -     256K    931.303

u0-0     DISK      OK             -       -       p3    -       465.651

u0-1     DISK      OK             -       -       p2    -       465.651

u0-2     DISK      OK             -       -       p1    -       465.651

u0/v0    Volume    -              -       -       -     -       931.303

----------

## nianderson

 *snIP3r wrote:*   

> 
> 
> 3ware has released a new firmware (http://www.3ware.com/download/Escalade9650SE-Series/9.4.1.2/9.4.1.2-9xxx-Upgrade.zip). can i update to this firmware without getting problems with my gentoo installation (especially the kernel driver)???
> 
> 

 

You can although I updated to the latest firmware and driver and I did notice a slight decrease in performance. However, it fixes a bug in RAID6 that when one drive fails you may lose data.

----------

## nianderson

 *mmikrut wrote:*   

> Hello All,
> 
> I was wrestling with the same problem for a while now.   Although I am running Debian Etch (sorry for trespassing on your forums!  )  the symptoms are the same.  I am running Raid5 with 3 segate 500GB drives on 2.6.21.6 kernel, however my write performance is hovering at 60Mb/s at its best.
> 
> I believe the problem is in 3ware firmware (or Linux driver) code and it is related to kernel versions above 2.6.18.    
> ...

 

No worries on running etch. All of my initial testing was done on gentoo but its now running etch and its in production. What file system are you using. I have seen ext3 run with better performance but fyi 3ware told me thats all you can expect to see .... its a limitation of the ext3 file system. There has to be something a bit wonky. However I did end up switching to xfs and I got 250MB/s write. I spent about 3 months dealing with this issue and tried a lethora of kernels, and drivers, and different kernel tweaks. XFS got my performance where I needed it. Also just the existance of a BBU increased my performance by about 7%. You should also make sure that storesave is set to perform. the blockdev --setra 16384 really only affects read speeds and 16384 is not the reccomended value its just an example so when you get to tweaking your read speeds you should try raising that. I think I have a script somewhere that will try and determine the optmal ra value. (there really is no optimal value) it really depends on how people access your system. My script only does single writes at a time so its an extremely poor representation of real world activity.

At any rate I dont check this thread a whole lot but you can look me up at http://anders0n.net i may be able to do some comparison benchmarks if you like although I can no longer change around my configuration with disks && raid level etc ...

----------

## snIP3r

hi!

i think the main problem is the amount of disks you have in your raid (also including my config with 3 disks in raid5)

i have also done some tests with iozone with these results (my machine has 2gb of ram):

```

area52 opt # iozone -s 5120m -r 64 -i 0 -i 1 -t 1 -b raid0_xfs_tuned.xls

        Iozone: Performance Test of File I/O

                Version $Revision: 3.242 $

                Compiled for 64 bit mode.

                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby

Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain

CYR,

                     Randy Dunlap, Mark Montague, Dan Million,

                     Jean-Marc Zucconi, Jeff Blomberg,

                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Thu Jul 12 12:06:28 2007

        File size set to 5242880 KB

        Record Size 64 KB

        Command line used: iozone -s 5120m -r 64 -i 0 -i 1 -t 1 -b

raid0_xfs_tuned.xls

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

        Throughput test with 1 process

        Each process writes a 5242880 Kbyte file in 64 Kbyte records

        Children see throughput for  1 initial writers  =   82154.59

KB/sec

        Parent sees throughput for  1 initial writers   =   69686.96

KB/sec

        Min throughput per process                      =   82154.59

KB/sec

        Max throughput per process                      =   82154.59

KB/sec

        Avg throughput per process                      =   82154.59

KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 rewriters        =  102218.50

KB/sec

        Parent sees throughput for  1 rewriters         =   98250.48

KB/sec

        Min throughput per process                      =  102218.50

KB/sec

        Max throughput per process                      =  102218.50

KB/sec

        Avg throughput per process                      =  102218.50

KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 readers          =  122396.06

KB/sec

        Parent sees throughput for  1 readers           =  122383.64

KB/sec

        Min throughput per process                      =  122396.06

KB/sec

        Max throughput per process                      =  122396.06

KB/sec

        Avg throughput per process                      =  122396.06

KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for 1 re-readers        =  120712.80

KB/sec

        Parent sees throughput for 1 re-readers         =  120702.87

KB/sec

        Min throughput per process                      =  120712.80

KB/sec

        Max throughput per process                      =  120712.80

KB/sec

        Avg throughput per process                      =  120712.80

KB/sec

        Min xfer                                        = 5242880.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"

"Record size = 64 Kbytes "

"Output is in Kbytes/sec"

"  Initial write "   82154.59

"        Rewrite "  102218.50

"           Read "  122396.06

"        Re-read "  120712.80

iozone test complete.
```

i think about 80mb/s write and 120mb/s for read is ok (with performance storsave policy), but i encountered less performance while accessing the raid with 2 ore more parallel threads (e.g. doing an emerge -uD world and copying some large files via samba). so i dont know if i should try to upgrade the raid setup from 3 to 4 disks - and hope  to improve the overall performance of the raid...

3ware support told me to update to the latest bios - but i read here in the forum that this decreases performance, so i will not do this.

sorry, but i am not able to do tests with various kernels cause i have the machine in production use. the only thing i can to is to change some settings (e.g. max_sectors_kb or nr_requests) and then make some tests...

i will post the results if the tests were done.

greets

snIP3r

----------

## nianderson

Indeed the new firmware and driver do decrease performance according to my tests. However they fix a bug in raid 6 which causes lost data when only loosing one disk. In general more disks means better performance so adding another disk should increase your speeds. I use this calculation that I pulled out of the ether but as of yet no one has corrected me so I keep using it, passing it on and asking somone to correct me  :Smile:  So here it is again

Ive got 5 drives in raid 5. If you assume 80% efficiency of their advertised max thats 60MB/s sustained. So 4*60MB/s is 240MB/s assuming no slow down due to raid. However if you consider that you might get 80% efficiency due to raid thats around 192MB/s.

 disk performance 

----------

## gohmdoree

the one thing that i don't reply is that while in the chroot install environment, hdparm -tT /dev/sda shows numbers that are higher outside of the chroot, than inside.  wouldn't it be the same?  is it cus i'm missing a module or something?

----------

## bexamous2

I'm not using the hardware raid instead software md, just using the controller for bandwidth and writecacheing--- right now set to protect or whatever, the safest.  Don't want use the others unless I get a BBU.... tho really how critical is the BBU on a 'home' system?  I might lose power twice a year.  Few years ago we had power problems with blackouts in the summer, last year I had my UPS fail and turn the system off twice before I realized it was going bad... anyways really how much damage can be done by 256MB at most sitting in cache?  The only data I'd lose is what I was currently writing to the disk, correct?  Any chance of filesystem corruption on data thats been there?

Anyways heres my results I'm not understanding from bonnie++ on 8 drive RAID6 array, 128K chunk, XFS fs.

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

bex              8G 80815  81 62696   9 25866   4 77196  76 189297  15 214.9   0

Why is per-char faster than per-block????

Same controller different set of disks in 6 drive RAID5, still software raid 128k chunk xfs fs...

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

bex              8G 75278  75 62144   8 27684   4 72154  71 169198  14 353.4   0

Again per-char is faster than per-block?

----------

## drescherjm

 *Quote:*   

> right now set to protect or whatever, the safest. Don't want use the others unless I get a BBU.... tho really how critical is the BBU on a 'home' system?

 

I would get a UPS instead. Get a 1000VA or 800VA Powercomm King Pro for less than $110US. I have had my King Pro 1000VA for many years and it works great. About every 3 years I need to spend $35 for two new batteries though.

----------

## snIP3r

hi all!

i bought a 4th drive for my raid and added it into the raid.

now i get around 175 mb/s with hdparm:

```

area52 public # hdparm -Tt /dev/sda

/dev/sda:

 Timing cached reads:   2302 MB in  2.00 seconds = 1151.52 MB/sec

 Timing buffered disk reads:  528 MB in  3.00 seconds = 175.92 MB/sec

```

thats about 43mb/s for each drive. so thats the same performance with the 3 drives (~125mb/s all and ~41mb/s each drive). hmmmm, i hoped to get better performance instead of only adding single perfomance of one drive to the overall performance of the raid.

iozone now gives me this:

```

area52 opt # iozone -s 5120m -r 64 -i 0 -i 1 -t 1 -b raid0_xfs_tuned.xls

        Iozone: Performance Test of File I/O

                Version $Revision: 3.242 $

                Compiled for 64 bit mode.

                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million,

                     Jean-Marc Zucconi, Jeff Blomberg,

                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Sun Aug 19 14:30:39 2007

        File size set to 5242880 KB

        Record Size 64 KB

        Command line used: iozone -s 5120m -r 64 -i 0 -i 1 -t 1 -b raid0_xfs_tuned.xls

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

        Throughput test with 1 process

        Each process writes a 5242880 Kbyte file in 64 Kbyte records

        Children see throughput for  1 initial writers  =  150777.25 KB/sec

        Parent sees throughput for  1 initial writers   =  125126.46 KB/sec

        Min throughput per process                      =  150777.25 KB/sec

        Max throughput per process                      =  150777.25 KB/sec

        Avg throughput per process                      =  150777.25 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 rewriters        =  122249.62 KB/sec

        Parent sees throughput for  1 rewriters         =  120502.20 KB/sec

        Min throughput per process                      =  122249.62 KB/sec

        Max throughput per process                      =  122249.62 KB/sec

        Avg throughput per process                      =  122249.62 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 readers          =  188414.77 KB/sec

        Parent sees throughput for  1 readers           =  188385.18 KB/sec

        Min throughput per process                      =  188414.77 KB/sec

        Max throughput per process                      =  188414.77 KB/sec

        Avg throughput per process                      =  188414.77 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for 1 re-readers        =  187022.05 KB/sec

        Parent sees throughput for 1 re-readers         =  186997.38 KB/sec

        Min throughput per process                      =  187022.05 KB/sec

        Max throughput per process                      =  187022.05 KB/sec

        Avg throughput per process                      =  187022.05 KB/sec

        Min xfer                                        = 5242880.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"

"Record size = 64 Kbytes "

"Output is in Kbytes/sec"

"  Initial write "  150777.25

"        Rewrite "  122249.62

"           Read "  188414.77

"        Re-read "  187022.05

iozone test complete.

```

so with iozone the performance benefit will be better than with hdparm. ok, the overall performance benefit is satisfying me. but i still hope to get more performance at all.

@bexamous2:

i dont think you have to buy a BBU for your home server. better by a UPS as drescherjm also suggested. IMHO this will be a better choice.

greets

snIP3r

----------

## drescherjm

 *Quote:*   

> thats about 43mb/s for each drive. so thats the same performance with the 3 drives (~125mb/s all and ~41mb/s each drive). hmmmm, i hoped to get better performance instead of only adding single perfomance of one drive to the overall performance of the raid. 

 

You are using raid 5 correct? In my calculations (even though I know the parity is striped) I subtract one drive for parity with raid 5 and then add up the speed of the remaining drives as in reality at any one time you are only reading from the data disks. So by that calculation your hdparm number is 58MB/s for the 3 data drives. I am not sure how fast your WD drives are as my 330 GB Seagate 7200.10 disks net 75MB/s hdparm so if I had that I would consider that not so good but if the WD drives net around 65MB/s this is good.

----------

## overkll

Had a 3ware-7x00 series controller.  Ever since the 2.6 series kernel became stable, I had to dik with it to get performance out of it.  mostly with the -setra option.  I've since switched to linux softraid and all is good.

I've heard good stories re Areca raid controllers.  Maybe get one of those instead of 3ware if you need a hardware raid controller.

EDIT:  Here's a link to an old 32 page comparion of 9 different controllers.  This is one of the server performance pages.  The table of contents is at the bottom of the page.

----------

## snIP3r

 *drescherjm wrote:*   

>  *Quote:*   thats about 43mb/s for each drive. so thats the same performance with the 3 drives (~125mb/s all and ~41mb/s each drive). hmmmm, i hoped to get better performance instead of only adding single perfomance of one drive to the overall performance of the raid.  
> 
> You are using raid 5 correct? In my calculations (even though I know the parity is striped) I subtract one drive for parity with raid 5 and then add up the speed of the remaining drives as in reality at any one time you are only reading from the data disks. So by that calculation your hdparm number is 58MB/s for the 3 data drives. I am not sure how fast your WD drives are as my 330 GB Seagate 7200.10 disks net 75MB/s hdparm so if I had that I would consider that not so good but if the WD drives net around 65MB/s this is good.

 

hi!

with this calculation i think its not so bad getting about 58mb/s per drive. thats ok for me.

greets

snIP3r

----------

## mwhitlock

I have 8 Seagate Barracuda ES (nearline 7200.10) 320GB drives in RAID6 on a 3ware 9650SE.  The system is an Intel Core 2 Quad Q6600 2.4GHz with 8 GB DDR2 SDRAM.

$ cat /proc/version

Linux version 2.6.22-gentoo-r5 (root@Crushinator) (gcc version 4.2.0 (Gentoo 4.2.0 p1.4)) #1 SMP Fri Aug 24 19:25:34 EDT 2007

$ fgrep /dev/root /proc/mounts

/dev/root / xfs rw,noatime,sunit=128,swidth=768 0 0

$ cat /sys/block/sda/queue/{max_sectors_kb,nr_requests,read_ahead_kb,scheduler}

64

512

384

[noop] anticipatory deadline cfq

/c0 Driver Version = 2.26.02.008

/c0 Model = 9650SE-8LPML

/c0 Memory Installed  = 224MB

/c0 Firmware Version = FE9X 3.08.02.005

/c0 Bios Version = BE9X 3.08.00.002

/c0 Monitor Version = BL9X 3.05.00.002

/c0 Serial Number = L326021A7080022

/c0 PCB Version = Rev 032

/c0 PCHIP Version = 2.00

/c0 ACHIP Version = 1.90

/c0 Number of Ports = 8

/c0 Number of Units = 1

/c0 Number of Drives = 8

/c0 Total Optimal Units = 1

/c0 Not Optimal Units = 0

/c0 JBOD Export Policy = off

/c0 Disk Spinup Policy = 1

/c0 Spinup Stagger Time Policy (sec) = 2

/c0 Auto-Carving Policy = off

/c0 Auto-Carving Size = 2048 GB

/c0 Auto-Rebuild Policy = on

/c0 Controller Bus Type = PCIe

/c0 Controller Bus Width = 4 lanes

/c0 Controller Bus Speed = 2.5 Gbps/lane

/c0/u0 Cache State = on

/c0/u0 volume(s) = 1

/c0/u0 name = RAID6

/c0/u0 serial number = 9QF23ED7CCD6AF000928

/c0/u0 Ignore ECC policy = off

/c0/u0 Auto Verify Policy = off

/c0/u0 Storsave Policy = performance

/c0/u0 Command Queuing Policy = on

/c0/u0 Parity Number = 2

bonnie++ results:

```
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

Crushinator     16G   869  99 257277  38 68305  13  1599  97 407293  41 498.2   9

Latency             18595us     315ms     226ms   40009us   50127us   80715us

Version 1.93c       ------Sequential Create------ --------Random Create--------

Crushinator         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16 11244  59 +++++ +++ 12661  49  8206  44 +++++ +++  3672  18

Latency             92011us    1033us   57648us     471ms      46us     627ms
```

# /sbin/hdparm -Tt /dev/sda

/dev/sda:

 Timing cached reads:   7206 MB in  2.00 seconds = 3606.77 MB/sec

 Timing buffered disk reads:  1124 MB in  3.00 seconds = 374.40 MB/sec

One thing to note is that when I created my partition using fdisk, I overrode the default starting sector and put in sector 768 as the first sector of the partition.  The stripe size is 64KB and the width is 6 (2 of the 8 drives are used for parity, leaving 6 for data), so that means the total amount of data in each full stripe is 384 KB, which is 768 sectors.  Also, when I created my XFS file system, I specified the sunit and swidth as 128 and 768, respectively, to correspond to my 64KB stripe size and 6 data drives.

----------

## snIP3r

hi mwhitlock!

so let me assume this calculation:

you use 8 drives - 6 for the data an get around 375mb/s reading - thats about 62,5mb/s per drive

i use 4 drives - 3 for data and get around 175mb/s reading - thats about 58mb/s per drive

so i think this your performance is similar to mine  :Wink: 

have you used the blockdev command like this

```

blockdev --setra 16384 /dev/sda

```

to possibly enhance overall performance?

@home i will install bonnie++ and run a test to compare the results because the above printed results were made with a 3-drive raid5 and i have added a 4th drive but no bonnie++ testings yet...

greets

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> have you used the blockdev command like this
> 
> ```
> blockdev --setra 16384 /dev/sda
> ```
> ...

 

I have been using a read-ahead of 768 sectors, which is 384 KB (at 512 bytes per sector), which is 6 times the stripe size, corresponding to my six drives' worth of data.

Here's the output from some benchmarks I performed a few minutes ago:

```
# hdparm -Tt /dev/sda 

 

 /dev/sda: 

  Timing cached reads:   9538 MB in  2.00 seconds = 4775.46 MB/sec 

  Timing buffered disk reads:  1212 MB in  3.00 seconds = 403.73 MB/sec

# dd if=/dev/zero of=11G bs=1M count=11K

11264+0 records in

11264+0 records out

11811160064 bytes (12 GB) copied, 45.5972 s, 259 MB/s
```

Here's bonnie++ with read-ahead set to 768 sectors:

```
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

Crushinator     16G   980  99 191234  28 51144   9  1449  78 288346  27 443.1   9

Latency             17786us     870ms     427ms     457ms     163ms     150ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

Crushinator         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  6928  37 +++++ +++  8874  41  8745  49 +++++ +++  2975  13

Latency               699ms      50us     195ms     348ms      53us     907ms
```

And here's bonnie++ with read-ahead set to 0 sectors:

```
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

Crushinator     16G  1084  99 219808  34 18681   6  1546  81 24089   6 412.9   8

Latency             12310us     685ms     303ms     138ms     220ms   85877us

Version 1.93c       ------Sequential Create------ --------Random Create--------

Crushinator         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  5115  30 +++++ +++  8978  37  6888  40 +++++ +++  2210  10

Latency              1003ms      95us     412ms     546ms     385us     736ms
```

And finally, bonnie++ with read-ahead set to 16384 sectors:

```
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

Crushinator     16G  1035  99 212313  30 100206  18  1877  95 444901  44 481.1  10

Latency             11701us     668ms     314ms   71187us   51347us     204ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

Crushinator         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  5685  31 +++++ +++  9052  34  4632  25 +++++ +++  2738  13

Latency               430ms     507us     300ms     710ms    1034us    1087ms
```

Over twice the rewrite performance and about 50% better read performance with the larger read-ahead size!  Looks like my theory about the optimal read-ahead size was dead wrong.  I will be using 16384 sectors from now on.  Awesome!

----------

## mwhitlock

By the way, if anyone else is reading this, you can use 

```
echo 8192 > /sys/block/sda/queue/read_ahead_kb
```

 instead of 

```
blockdev --setra 16384 /dev/sda
```

 in your startup scripts.

Then you don't need blockdev and you don't have to read the binary and execute it.

----------

## overkll

If you dig around on the 3ware website, you can find the syntax to add it to /etc/sysctl.conf instead of a startup script.

----------

## mwhitlock

 *overkll wrote:*   

> If you dig around on the 3ware website, you can find the syntax to add it to /etc/sysctl.conf instead of a startup script.

 

```
# find /proc/sys -type d

/proc/sys

/proc/sys/kernel

/proc/sys/kernel/random

/proc/sys/kernel/pty

/proc/sys/vm

/proc/sys/net

/proc/sys/net/core

/proc/sys/net/ipv4

/proc/sys/net/ipv4/route

/proc/sys/net/ipv4/neigh

/proc/sys/net/ipv4/neigh/default

/proc/sys/net/ipv4/neigh/eth0

/proc/sys/net/ipv4/neigh/lo

/proc/sys/net/ipv4/conf

/proc/sys/net/ipv4/conf/all

/proc/sys/net/ipv4/conf/default

/proc/sys/net/ipv4/conf/eth0

/proc/sys/net/ipv4/conf/lo

/proc/sys/net/unix

/proc/sys/fs

/proc/sys/fs/inotify

/proc/sys/fs/nfs

/proc/sys/fs/xfs

/proc/sys/fs/mqueue

/proc/sys/debug

/proc/sys/dev

/proc/sys/dev/scsi

/proc/sys/dev/rtc

/proc/sys/dev/hpet

/proc/sys/dev/cdrom

/proc/sys/abi

/proc/sys/sunrpc

# /bin/ls /proc/sys/dev/scsi

logging_level
```

Nothing in /proc/sys looks like it would apply to the 3ware controller.

I have all the deprecated crap turned off in my kernel.  The preferred way to configure devices these days is through sysfs.  Of course, /etc/sysctl.conf only tweaks stuff in /proc/sys, not in /sys.  I just put some echos in /etc/conf.d/local.start.  Works fine.

----------

## snIP3r

hi all!

after some mount options optimization i get new results with iozone.

```

 mount

/dev/sda3 on / type xfs (rw,noatime,nobarrier,logbufs=8)

```

with new options nobarrier and logbufs=8

here are the results:

```

area52 opt # iozone -s 5120m -r 64 -i 0 -i 1 -t 1

        Iozone: Performance Test of File I/O

                Version $Revision: 3.242 $

                Compiled for 64 bit mode.

                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million,

                     Jean-Marc Zucconi, Jeff Blomberg,

                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Mon Mar 10 21:13:11 2008

        File size set to 5242880 KB

        Record Size 64 KB

        Command line used: iozone -s 5120m -r 64 -i 0 -i 1 -t 1

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

        Throughput test with 1 process

        Each process writes a 5242880 Kbyte file in 64 Kbyte records

        Children see throughput for  1 initial writers  =  176441.88 KB/sec

        Parent sees throughput for  1 initial writers   =  159852.78 KB/sec

        Min throughput per process                      =  176441.88 KB/sec

        Max throughput per process                      =  176441.88 KB/sec

        Avg throughput per process                      =  176441.88 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 rewriters        =  148128.45 KB/sec

        Parent sees throughput for  1 rewriters         =  147003.01 KB/sec

        Min throughput per process                      =  148128.45 KB/sec

        Max throughput per process                      =  148128.45 KB/sec

        Avg throughput per process                      =  148128.45 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for  1 readers          =  189536.80 KB/sec

        Parent sees throughput for  1 readers           =  189506.96 KB/sec

        Min throughput per process                      =  189536.80 KB/sec

        Max throughput per process                      =  189536.80 KB/sec

        Avg throughput per process                      =  189536.80 KB/sec

        Min xfer                                        = 5242880.00 KB

        Children see throughput for 1 re-readers        =  176014.23 KB/sec

        Parent sees throughput for 1 re-readers         =  175991.95 KB/sec

        Min throughput per process                      =  176014.23 KB/sec

        Max throughput per process                      =  176014.23 KB/sec

        Avg throughput per process                      =  176014.23 KB/sec

        Min xfer                                        = 5242880.00 KB

"Throughput report Y-axis is type of test X-axis is number of processes"

"Record size = 64 Kbytes "

"Output is in Kbytes/sec"

"  Initial write "  176441.88

"        Rewrite "  148128.45

"           Read "  189536.80

"        Re-read "  176014.23

iozone test complete.

```

that 17 -21 % increase of write performance. i thought this was worth posting  :Wink: 

HTH

snIPer

----------

## mwhitlock

Be careful turning off the write barrier.  Only do it if you are using a RAID controller that has an on-board battery backup for its write cache.

----------

## snIP3r

 *mwhitlock wrote:*   

> Be careful turning off the write barrier.  Only do it if you are using a RAID controller that has an on-board battery backup for its write cache.

 

hi!

thx for the tip. this was only a test. i do not use a bbu on my 9650, but i use an apc  Back-UPS BR 800 for my server so i think this is ok.

thx

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> i do not use a bbu on my 9650, but i use an apc  Back-UPS BR 800 for my server so i think this is ok.

 

I have my machine on a Smart-UPS 1500 and I'm running apcupsd, but I still won't be turning off the write barrier, because my 9650SE has locked up on very rare occasion (it has happened twice), and I want the drives of my array to be in sync if and when that occurs.  (I know it was the 9650 freezing up because after I hit the system Reset button and the system BIOS POST'd, the 3ware BIOS hung and eventually said, "No response from controller firmware."  I had to kill power entirely to get the 3ware card to fully reset.)

----------

## snIP3r

 *mwhitlock wrote:*   

>  *snIP3r wrote:*   i do not use a bbu on my 9650, but i use an apc  Back-UPS BR 800 for my server so i think this is ok. 
> 
> I have my machine on a Smart-UPS 1500 and I'm running apcupsd, but I still won't be turning off the write barrier, because my 9650SE has locked up on very rare occasion (it has happened twice), and I want the drives of my array to be in sync if and when that occurs.  (I know it was the 9650 freezing up because after I hit the system Reset button and the system BIOS POST'd, the 3ware BIOS hung and eventually said, "No response from controller firmware."  I had to kill power entirely to get the 3ware card to fully reset.)

 

ok, thats an argument. what you told never happend to me yet - but it can surely occur. i think i will test the increase a little and if i see no enhancements, i activate barriers again.

snIP3r

----------

## snIP3r

hi all!

3ware has released a new firmware 9.5.2 with supposedly improved performance. has anybody installed and tested this firmware successfully?

greets

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> 3ware has released a new firmware 9.5.2 with supposedly improved performance. has anybody installed and tested this firmware successfully?

 

I installed it on my 9650SE and have no complaints.  I haven't done any tests of performance, objective or otherwise, but it doesn't seem any worse at least.  Note, when you upgrade, you do have to manually turn on the read cache, if you want it.

----------

## snIP3r

 *mwhitlock wrote:*   

> I installed it on my 9650SE and have no complaints.  I haven't done any tests of performance, objective or otherwise, but it doesn't seem any worse at least.  Note, when you upgrade, you do have to manually turn on the read cache, if you want it.

 

thx for the info. how can i enable read cache? i havent found any info about this yet.

greets

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> how can i enable read cache? i havent found any info about this yet.

 

I did it with tw_cli:

```
tw_cli /c0/u0 set rdcache=intelligent
```

I don't know the difference between "basic" and "intelligent" read cache.  When I looked, 3ware still hadn't posted the user manual addendum for the 9.5.2 codeset.

----------

## snIP3r

 *mwhitlock wrote:*   

> I did it with tw_cli:
> 
> ```
> tw_cli /c0/u0 set rdcache=intelligent
> ```
> ...

 

ah, ok, that explains it to me. the rdcache setting is available since 9.5.2. if you download the tw_cli sources from 3ware homepage you can find a html page with the new instruction set.

sorry, but i have another question: did you upgrade from firmware 9.5.1. to 9.5.2?

thx

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> sorry, but i have another question: did you upgrade from firmware 9.5.1. to 9.5.2?

 

Technically, I upgraded firmware from 4.06.00.004 to 4.08.00.006.  But yes, it was the firmware included in the 9.5.1 codeset, now upgraded to the firmware included in the 9.5.2 codeset.

----------

## snIP3r

 *mwhitlock wrote:*   

>  *snIP3r wrote:*   sorry, but i have another question: did you upgrade from firmware 9.5.1. to 9.5.2? 
> 
> Technically, I upgraded firmware from 4.06.00.004 to 4.08.00.006.  But yes, it was the firmware included in the 9.5.1 codeset, now upgraded to the firmware included in the 9.5.2 codeset.

 

sorry for asking so many questions, but i had very bad experience while updating from firmware 3.xx to 4.06.00.004. i got error messages like this:

```

3w-9xxx: scsi0: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.

sd 0:0:0:0: WARNING: (0x06:0x002C): Command (0x2a) timed out, resetting card.

```

and the system freezes a little later.

what kernel do you use, can you please post me your kernel config and did you have any kernel options at startup (e.g to enable MSI).

thx in advance

snIP3r

----------

## mwhitlock

 *snIP3r wrote:*   

> what kernel do you use, can you please post me your kernel config and did you have any kernel options at startup (e.g to enable MSI).

 

I'm using sys-kernel/gentoo-sources-2.6.29-r5.

Here's my kernel config:

http://www.mattwhitlock.com/config-2.6.29-gentoo-r5

And my kernel command line:

```
$ cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz root=/dev/sda1
```

----------

## snIP3r

hi!

thx for your kernel config. i successfully upgraded to 2.6.29-gentoo-r5 but after applying the newest firmware my harddisk subsystem got stuck after about three days running 24hrs a day without any problems. i think i cannot install any firmware >= 4.xx (means codeset >=9.5.1). so i wonder if my controller has some sort of damage...

greets

snIP3r

----------

## oleo

Hi all!

I also have some problems with a 9650 controller ad a 4 disks raid5 array.

Reading data is ok, with a transfer rate higher than 200Mb/s.

```
gandalf2 # time dd if=mybigfile of=/dev/null bs=1M count=3K

2303+0 records in

2302+0 records out

2413821952 bytes (2.4 GB) copied, 9.36377 s, 258 MB/s

```

Writing is quite different:   :Evil or Very Mad: 

```
gandalf2 # dd if=/dev/zero of=11G_file bs=65536 count=163840

2691+0 records in

2691+0 records out

176357376 bytes (176 MB) copied, 28.61 s, 6.2 MB/s

```

It's quite strange...

Does anyone has the same problem?

Tomorrow I will upgrade firmware to see if something goes better.

----------

## mwhitlock

 *oleo wrote:*   

> It's quite strange...
> 
> Does anyone has the same problem?

 

The Linux kernel has had a severe I/O scheduling bug at least since 2.6.17.

As a workaround, see if this helps your write throughput:

```
# echo 1 > /proc/sys/vm/dirty_background_ratio

# echo 2 > /proc/sys/vm/dirty_ratio

# echo deadline > /sys/block/sda/queue/scheduler
```

----------

## snIP3r

hi oleo!

there are also several suggestions to increase performance. i use theses settings:

```

#not neccessary for codeset >= 9.4.2

#echo "64" > /sys/block/sda/queue/max_sectors_kb 

echo "512" > /sys/block/sda/queue/nr_requests

echo "deadline" > /sys/block/sda/queue/scheduler

blockdev --setra 16384 /dev/sda

```

the first line should only be uses for fimrware prior 9.4.2 codeset. 3ware has also some information about increasing performance:

http://3ware.com/kb/article.aspx?id=11050.

http://3ware.com/kb/article.aspx?id=10889.

http://3ware.com/kb/article.aspx?id=10036.

http://3ware.com/kb/article.aspx?id=15244.

do you have enabled write cache?

HTH

snIP3r

----------

## oleo

Hi!

With write cache enabled I have reached approx. 150Mb/s:

```
gandalf2 # dd if=/dev/zero of=11G_file bs=65536 count=163840

16503+0 records in

16503+0 records out

1081540608 bytes (1.1 GB) copied, 6.34697 s, 170 MB/s

gandalf2 # dd if=/dev/zero of=11G_file bs=65536 count=163840 && rm 11G_file

163840+0 records in

163840+0 records out

10737418240 bytes (11 GB) copied, 82.8089 s, 130 MB/s

```

Reading some linux benchmark results from 3ware website, I think this is a good starting point to make some further optimizations.

I don't have a BBU so there's the risk of data loss with write cache enabled. Without write cache enabled, is possible to have only 7Mb/s ow write speed? Or it's a symptom of other problems or misconfigurations that persist now, and are only weakened by "write cache" enabled?

Moreover: I don't have the last released firmware. Do you suggest to update? Is it safe or I can have data loss?

Now I'm going to read thoroughly the links you have posted.

Thanks a lot,

Oleo.

----------

## snIP3r

hi oleo!

i've read other posts from 9650 users having similar experience. i think its safe to activate write cache to get appropriate performance . i do also not have a BBU but i use a UPS (ok, that's not really the same but better than nothing). without activating write cache i think you will not get any better write performance - no matter what config you try.

upgrading the firmware to the latest codeset increased my read/write speed (temporary, see above posts why temporary) about 15% (activating newly introduced read cache and having activated write cache, storsafe profile set to performance).

HTH

snIP3r

----------

## drescherjm

 *Quote:*   

> UPS (ok, that's not really the same but better than nothing)

 

The question to me is how often do you have a kernel panic? I mean I consider a UPS a much better value if you never have kernel panics.

----------

## mwhitlock

 *drescherjm wrote:*   

> The question to me is how often do you have a kernel panic? I mean I consider a UPS a much better value if you never have kernel panics.

 

*blink*  I've never had a kernel panic.  I've had a file system crash once, which was not pretty, but the kernel was still running.

----------

## drescherjm

 *Quote:*   

> *blink* I've never had a kernel panic.

 

I have but in all cases it was either bad hardware or a very experimental kernel that I was using. 

The point that I wanted to make is a BBU protects you against kernel freezes, spontaneous reboots, kernel panics, suddenly dead power supply... But with a normal linux system these should not happen or should be very rare. And on top of that if it did happen you could have data loss anyways at the filesystem level. I do admit the risk is higher though.

----------

## snIP3r

hi!

i trust my hardware and kernel config, and i never use any exerimental stuff. personally i do not have any kernel panics (ok, after trying the latest codeset from 3ware i did, bat that was due to hardware defect). i agree with drescherjm that on normal linux systems freezes and so on are very rare. i use xfs as a filesystem so i think even if a kernel freeze happens the filesystem will not crash. ok, a bbu was supposed to be the best way to protect the data and i think i will buy one later on.  so actually i can live with the situation.

HTH

snIP3r

----------

## oleo

Hi all!

I've read all the articles from 3ware KB site and I've applied some other configurations without any gain in write performance but reaching 337 Mb/s in reading.

```
gandalf2 ~ # time dd if=11G_file of=/dev/null bs=1M count=3K

3221225472 bytes (3.2 GB) copied, 9.57094 s, 337 MB/s

```

I've turned on statistics as explained here. These are the results and after some read/write operations:

```
gandalf2 ~ # tw_cli /c0 show dpmstat

Drive Performance Monitor Configuration for /c0 ...

Performance Monitor: ON

Version: 1

Max commands for averaging: 100

Max latency commands to save: 10

Requested data: Instantaneous Drive Statistics

                               Queue           Xfer         Resp

Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)

------------------------------------------------------------------------

p0     OK               u0     12      11      0.196        9

p1     OK               u0     13      11      0.210        9

p2     OK               u0     6       11      0.231        5

p3     OK               u0     5       11      0.203        6

```

I'm not able to interpret these results: are they reasonable? What about Xfer Rate?

Thanks,

Oleo.

PS: Wait a moment! If I launch tw_cli /c0 show dpmstat during write op. I've these results:

```
gandalf2 ~ # tw_cli /c0 show dpmstat

Drive Performance Monitor Configuration for /c0 ...

Performance Monitor: ON

Version: 1

Max commands for averaging: 100

Max latency commands to save: 10

Requested data: Instantaneous Drive Statistics

                               Queue           Xfer         Resp

Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)

------------------------------------------------------------------------

p0     OK               u0     25      531     32.877       32

p1     OK               u0     27      532     32.930       22

p2     OK               u0     25      527     32.669       31

p3     OK               u0     25      535     33.105       32

```

And during read operations:

```
gandalf2 ~ # tw_cli /c0 show dpmstat

Drive Performance Monitor Configuration for /c0 ...

Performance Monitor: ON

Version: 1

Max commands for averaging: 100

Max latency commands to save: 10

Requested data: Instantaneous Drive Statistics

                               Queue           Xfer         Resp

Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)

------------------------------------------------------------------------

p0     OK               u0     17      2487    76.120       6

p1     OK               u0     22      2617    82.112       7

p2     OK               u0     27      2352    73.713       8

p3     OK               u0     31      2320    71.905       10

```

----------

## mwhitlock

Oleo, you might want 

```
tw_cli /c0 show dpmstat type=ra
```

 to see running-average statistics, rather than instantaneous statistics.

Also, you should specify conv=fdatasync in your dd commands so you get results that aren't "helped" by the write-behind cache.

----------

## oleo

 *mwhitlock wrote:*   

> Also, you should specify conv=fdatasync in your dd commands so you get results that aren't "helped" by the write-behind cache.

 

I've the same write performance with or without conf=fdatasync.

Thanks a lot for your help!

Oleo.

----------

