# Hardware vs. Software RAID

## Soda Ant

What kind of performance differences can I expect for a RAID-5 setup with a true hardware RAID controller (3ware, Netcell, etc.) versus doing it in software using the kernel's software RAID support?

----------

## adaptr

Impossible to tell without actually comparing it on your hardware.

Generally, the more drives you have in a software RAID-5 set the more CPU power it takes.

A hardware solution can obviously reach its maximum speed with the maximum amount of drives attached.

If you mean in terms of raw disk I/O, that depends on the interface you run on, and the kind of drives.

An 8-channel Areca SATA RAID card with 8 WD Raptors running at 10K rpm is benchmarked at around 400MB/sec sustained throughput - something I very much doubt you will ever get from a software solution...

But in normal cases, Linux softraid will perform very well; with 3-5 drives and a decent CPU you will not run slower than on a dedicated card.

There is CPU overhead, obviously, but no performance hit as long as the CPU is fast enough.

----------

## Soda Ant

Thanks.

This will be on a dual Opteron motherboard (eventually with dual cores), so CPU horsepower shouldn't be an issue.

Is the sata_nv driver robust enough at this point to support stable software RAID?

----------

## adaptr

If it's good enough to run one drive, it's good enough to run N drives.

Software RAID is accomplished through the md and raid* kernel modules, not the chipset driver.

----------

## Vietor

I can not currently provide any date for software, but I can provide some numbers on hardware for you.

I'm running a 3ware 7506-8 with a 7 (+1 spare) disk RAID 5 array on a Tyan TigerMPX (@x AthlonMP 2800+, 1.75GB RAM). The disks are a combination of Western Digital WD2500SB Drives and some crappy 250GB Maxtor drives that have no purpose being in the array, but I'm stuck with until I can find the capital to replace them.

These numbers are from in use hardware, though during a very quiet period traffic wise. Note though that the array is serving data to Squid, Mysql, Apache, and a few NFS Root clients. So I would expect numbers to be slightly higher in an acutally idle environment.

I am using the deadline scheduler with:

```

fifo_batch      16

front_merges   1

read_expire     300

write_expire    5000

writes_starved 4

```

First set of numbers is from a bonnie++ run, no options were given to bonnie, this is a vanilla run.

```

Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

uber          3536M   551  96 71804  27 22188  10   760  99 255965  83 374.9   8

Latency             18317us    7652ms     343ms   34993us   48720us    2921ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

uber                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                16  2111  49 +++++ +++  2306  13  2159  21 +++++ +++  3083  24

Latency               747ms      68us     887ms     625ms     934us     594ms

```

The second set of numbers is fron an IOZone run, and should give you some idea of parallel performance.

```

        Iozone: Performance Test of File I/O

                Version $Revision: 3.242 $

                Compiled for 32 bit mode.

                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million,

                     Jean-Marc Zucconi, Jeff Blomberg,

                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Sat Sep 24 18:26:36 2005

        Record Size 512 KB

        Excel chart generation enabled

        File size set to 524288 KB

        Command line used: iozone -i 0 -i 1 -i 2 -l8 -u8 -r512k -R -s 512M

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

        Min process = 8

        Max process = 8

        Throughput test with 8 processes

        Each process writes a 524288 Kbyte file in 512 Kbyte records

        Children see throughput for  8 initial writers  =   51214.25 KB/sec

        Parent sees throughput for  8 initial writers   =   17286.00 KB/sec

        Min throughput per process                      =    1424.68 KB/sec

        Max throughput per process                      =   14114.26 KB/sec

        Avg throughput per process                      =    6401.78 KB/sec

        Min xfer                                        =   53248.00 KB

        Children see throughput for  8 rewriters        =   39618.62 KB/sec

        Parent sees throughput for  8 rewriters         =   31734.50 KB/sec

        Min throughput per process                      =    2394.02 KB/sec

        Max throughput per process                      =   13606.48 KB/sec

        Avg throughput per process                      =    4952.33 KB/sec

        Min xfer                                        =  110592.00 KB

        Children see throughput for  8 readers          =   83668.89 KB/sec

        Parent sees throughput for  8 readers           =   83627.29 KB/sec

        Min throughput per process                      =    8685.70 KB/sec

        Max throughput per process                      =   12124.75 KB/sec

        Avg throughput per process                      =   10458.61 KB/sec

        Min xfer                                        =  375808.00 KB

        Children see throughput for 8 re-readers        =   74837.25 KB/sec

        Parent sees throughput for 8 re-readers         =   74774.16 KB/sec

        Min throughput per process                      =    7748.56 KB/sec

        Max throughput per process                      =   10580.14 KB/sec

        Avg throughput per process                      =    9354.66 KB/sec

        Min xfer                                        =  384000.00 KB

        Children see throughput for 8 random readers    =   66062.66 KB/sec

        Parent sees throughput for 8 random readers     =   65981.90 KB/sec

        Min throughput per process                      =    6050.48 KB/sec

        Max throughput per process                      =    9761.92 KB/sec

        Avg throughput per process                      =    8257.83 KB/sec

        Min xfer                                        =  325632.00 KB

        Children see throughput for 8 random writers    =   21346.51 KB/sec

        Parent sees throughput for 8 random writers     =   10109.23 KB/sec

        Min throughput per process                      =    1122.20 KB/sec

        Max throughput per process                      =    5094.63 KB/sec

        Avg throughput per process                      =    2668.31 KB/sec

        Min xfer                                        =  117760.00 KB

```

Another set with smaller record size.

```

Iozone: Performance Test of File I/O

                Version $Revision: 3.242 $

                Compiled for 32 bit mode.

                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million,

                     Jean-Marc Zucconi, Jeff Blomberg,

                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Sat Sep 24 18:50:50 2005

        Record Size 32 KB

        Excel chart generation enabled

        File size set to 524288 KB

        Command line used: iozone -i 0 -i 1 -i 2 -l8 -u8 -r32k -R -s 512M

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

        Min process = 8

        Max process = 8

        Throughput test with 8 processes

        Each process writes a 524288 Kbyte file in 32 Kbyte records

        Children see throughput for  8 initial writers  =   39900.10 KB/sec

        Parent sees throughput for  8 initial writers   =   21559.02 KB/sec

        Min throughput per process                      =    2031.67 KB/sec

        Max throughput per process                      =    8196.20 KB/sec

        Avg throughput per process                      =    4987.51 KB/sec

        Min xfer                                        =  134464.00 KB

        Children see throughput for  8 rewriters        =   33571.62 KB/sec

        Parent sees throughput for  8 rewriters         =   28753.34 KB/sec

        Min throughput per process                      =    1413.65 KB/sec

        Max throughput per process                      =    7100.69 KB/sec

        Avg throughput per process                      =    4196.45 KB/sec

        Min xfer                                        =  120672.00 KB

        Children see throughput for  8 readers          =   74826.65 KB/sec

        Parent sees throughput for  8 readers           =   74793.74 KB/sec

        Min throughput per process                      =    8590.69 KB/sec

        Max throughput per process                      =   10396.37 KB/sec

        Avg throughput per process                      =    9353.33 KB/sec

        Min xfer                                        =  433536.00 KB

        Children see throughput for 8 re-readers        =   69910.06 KB/sec

        Parent sees throughput for 8 re-readers         =   69828.71 KB/sec

        Min throughput per process                      =    7959.95 KB/sec

        Max throughput per process                      =    9925.44 KB/sec

        Avg throughput per process                      =    8738.76 KB/sec

        Min xfer                                        =  420736.00 KB

        Children see throughput for 8 random readers    =   15773.16 KB/sec

        Parent sees throughput for 8 random readers     =   15772.05 KB/sec

        Min throughput per process                      =    1831.48 KB/sec

        Max throughput per process                      =    2138.23 KB/sec

        Avg throughput per process                      =    1971.65 KB/sec

        Min xfer                                        =  449088.00 KB

        Children see throughput for 8 random writers    =   13089.78 KB/sec

        Parent sees throughput for 8 random writers     =    7389.74 KB/sec

        Min throughput per process                      =     842.49 KB/sec

        Max throughput per process                      =    2587.57 KB/sec

        Avg throughput per process                      =    1636.22 KB/sec

        Min xfer                                        =  172544.00 KB

```

If there is any further information you would like about the conditions fo the test run let me know and I'll try to get back to you.

----------

## elestedt

This isn't really performance, but something one should consider anyway:

If your hardware controller is smart enough - which most are - it will only present one disk to your operatingsystem.

This makes installing an OS with HW raid just as easy as installing it on a single disk... i.e. your OS doesn't even know you're running a raid system, which is fine as long as the controller can handle the loss of a disk - which is has to.

This should mean that the time it takes to install HW raid, and probably maintain it, is lower than SW raid. I've only worked with HW raid and I wouldn't change to SW.

----------

## brain salad surgery

software raid reconstruction is completely transparent...

so that you can use the raid partition as soon as you've

booted. 

you must have the appropriated driver in your kernel

and a device mdx will be created at boot if you

enable autodetection. you can adjust

each drive performances using hdparm

using smartmontools will allow you to check

the health of each drive of the raid partition

and monitor them

it won't be longer to format at installation since

you don't have to wait for reconstruction and

can perform the whole installation without waiting.

i think software raid offers much more flexibility

(for troobleshooting, monitoring, administration, adding

drives, etc.). Give a look at mdadm...

performances are good too (i'll post some bonnie++

results) and no more than 30% of my single athlon XP 2800

is used (given i run a software raid5 composed of

8 160Gb drives).

----------

## elestedt

 *brain salad surgery wrote:*   

> software raid reconstruction is completely transparent...
> 
> so that you can use the raid partition as soon as you've
> 
> booted. 

 

So is hardware raid...

 *brain salad surgery wrote:*   

> you must have the appropriated driver in your kernel
> 
> and a device mdx will be created at boot if you
> 
> enable autodetection. you can adjust
> ...

 

That might be hard to do with hw raid - havn't tried

 *brain salad surgery wrote:*   

> it won't be longer to format at installation since
> 
> you don't have to wait for reconstruction and
> 
> can perform the whole installation without waiting.

 

Same with hw raid.

 *brain salad surgery wrote:*   

> i think software raid offers much more flexibility
> 
> (for troobleshooting, monitoring, administration, adding
> 
> drives, etc.). Give a look at mdadm...

 

Depends on controller/manufacturer. There are probably tools which as just as good available for hw raid controllers.

 *brain salad surgery wrote:*   

> performances are good too (i'll post some bonnie++
> 
> results) and no more than 30% of my single athlon XP 2800
> 
> is used (given i run a software raid5 composed of
> ...

 

Using hw raid, if the controller is good, the performance would probably be even better.

----------

## adaptr

 *Quote:*   

> performances are good too (i'll post some bonnie++
> 
> results) and no more than 30% of my single athlon XP 2800
> 
> is used (given i run a software raid5 composed of
> ...

 

Yes, well - if you ran 8 SATA drives on an Areca card, you would incur no CPU overhead.

And a software array definitely will not be able to sustain the 300MB+ throughput of a dedicated raid card.

----------

## nevynxxx

One thing to consider.

With software raid, you have to know the setup to rebuild it, and you have to stick to the same operating system, you can play with a BSD dual boot for example.

With hardware raid, you can.

I would go for hardware raid everytime, battery backed with a big cache. I believe in dedicated hardware for a dedicated task.

----------

