# RAID Hardware vs. Software

## mgbowman

So I'm building a new Gentoo server (yay!  :Very Happy: ) and I'm looking at various RAID configurations. Before now, I was unaware of implementing RAID in software and would like to get some feedback from the community.

Firstly, I would think it would be must faster using a hardware controller (reducing the load on the CPU) but I am not sure which controller I should go with (if I do choose this solution). Could anyone in the community comment on the performance increase (if any) of choosing a hardware over software implementation?

Secondly, my initial thoughts were to use 3 SATA HDDs (the size I'm not sure of) and my configuration was planned as follows: 1 HDD for the OS, kernel, sources, etc. and the other 2 HDDs would be configured as RAID-0 ("stripe") to provide storage for the local LAN as well as other services requiring a good bit of space (large web applications, database storage, etc). I would like to keep the OS separate from the storage as I've been told by colleagues in the past this is a good way to implement RAID. Is this contrary to belief and/or even possible with Gentoo? (I couldn't imagine it not being so) Anyone have any type of experience with this type of configuration?

Other than that, I'm still looking around on the net for resources and figured I submit a quick post to help speed up the process. I hope someone can give some advice/tips and that in the future this post could help someone else in my same situation.  

Thanks a million,

--mgb

----------

## mattmatteh

i dont hava raid but would like to set it up and was reading up on it.   if i were to do raid, i would do 1 for mirring if a hard drive fails, if it were to be for network storage.

also, for software or hardware, if you get or have a hardware card, i read that alot say hardware but they are not really hardware.  they need a special driver or kernel module for them to work.  i think its like they are mostly software but can do some of it in hard ware.  kinda misleading.

also, if its on the network, is this 100 Mb or gigabit?  100 Mb is 12MB and i dont think you will see a speed decrease wiith software raid, and most certainly no speed up with raid 0.

these are mostly comments and guesses, i have not set up either and have no factual info

matt

----------

## Jearil

If you're only going for RAID 0 (which isn't really redundant as it provides no failsafe at all) to increase storage space using two drives, you might want to look into LVM.  There's also an installation guide on how to do such on Gentoo available that works quite well.

Again, it won't provide any redundancy that a RAID 1 or greater would provide, but it's handy for managing large amounts of space between multiple drives (I would even use it on top of RAID 5 just as allocating volumns is easy with LVM).

Good luck in your endevours.

----------

## lbrtuk

Worse than that, raid0 will decrease your reliability over a single drive. You will double the liklihood of losing data because either one of the drives failing will cause everything to be lost.

And yes, hardware raid is pretty useless except for big installations. IMHO at least.

----------

## mgbowman

The RAID-5 configuration is what I originally had in mind (the 0 was a typo  :Shocked:  in my original post). However, after some more research, I read that you can'y do any type of stripping on the /boot partition so now I'm in a bit of a pickle. I have 3x250GB HDDs and I'm not quite sure how to partition them - I was thinking of stripping all 3 together. Now, if I set one partition aside for /boot, my HDDs won't be the same size and this is going to leave me with some unused space on the other 2 - correct? (they all need to be the same size no?) Another thought was to get a smaller HDD for the /boot and swap and then stripe the 3x250 for /. Would this be a viable solution?

Thanks for the input and I hope to resolve this issue relatively quickly - all my new H/W arrived today!  :Very Happy: 

----------

## NeddySeagoon

mgbowman,

There are three sorts of raid Linux supports.

Software raid in the BIOS (called fakeraid). The BIOS makes the drives appear as a single block device, as they would in real hardware raid, hence the nickname. This form of software raid is Windows compatible.

Hardware raid - you buy a raid card, thats a computer on a card. It does all the raid work.

Consider the cost and the PCI bus bandwidth before you go down this route.

Kernel software raid. You partition the drives as normal then contribute partitions to raid sets. Thus you can have a mix of raid levels on the same drive. This solves the /boot problem at a stroke. /boot can be a single partion, or a raid1 and grub will still be happy.

On three drives I would make a small partition at the front of each for use as /boot <swap> and <swap> or as /boot /boot (raid 1) and <swap>.  Swap need not be raided, the kernel can manage up to 16 swap spaces, however, loosing a used swap will bring your system down.

Aftet the small partitions, I would break up the install int at least root and /home, with whatever raid levels you like.

----------

## mgbowman

Okay, so from what you're saying maybe partition like this:

sda

1: /boot (2GB)

2: /

sdb

1: /boot (2GB)

2: /

sdc

1: <swap> (2GB) - this box will start with 1GB of RAM

2: /

Now I can setup the first 2 /boot partitions in RAID-1 using

```
# mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
```

And then stripe the second partitions of each drive in RAID-5 using

```
# mdadm --create --verbose /dev/md2 --level=5 --raid-devices=3 /dev/sda2 /dev/sdb2 /dev/sdc2
```

Then I would effectively have

/dev/sdc1 as swap

/dev/md1 as /boot (in RAID-1) ... and

/dev/md2 as / (in RAID-5) ... without having any unwasted space (minus the excessive size of /boot)

However, if /dev/sdc fails, my system will crap due to swap going down? If so, is there anyway to get around this as the whole point of me implementing RAID-5 is to prevent my system from going down on a single HDD failure?

----------

## troymc

I would suggest a layout more like this:

sda:

1) 64MB

2) 2GB

3) 7GB

4) 240GB

sdb:

1) 64MB

2) 2GB

3) 7GB

4) 240GB

sdc:

1) 64MB

2) 2GB

3) 7GB

4) 240GB

Then setup your RAID:

md0 - RAID 1 - sda1 & sdb1 (64MB /boot mirror, use sdc1 as offline hot-backup)

md1 - RAID 1 - sda2 & sdb2 (2GB mirrored swap)

md2 - RAID 5 - sda3 & sdb3 & sdc3 (14GB root - LVM? - everything except /usr/portage/distfiles)

md3 - RAID 5 - sda4 & sdb4 & sdc4 (480GB - data? LVM?)

sdc3 - 2GB /usr/portage/distfiles (doesn't matter if you lose this)

cron a job to dd /dev/md0 to sdc1 once a week or so. If you /boot mirrors does get corrupted you can point to a kernel here to get up & running.

troymc

----------

## NeddySeagoon

mgbowman,

/boot is only needed to get started - so raid1 swap and make /boot sda1 alone.

Everything in /boot except grub.conf is scattered arouund the system anyway. If sda failed, you could steal part of swap to reform /boot to get going with the raid in degraded mode, while you got a replacement drive. This is assuming that you needed to reboot after sda had failed but was still not fixed.

----------

## neysx

http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml might help you.

----------

## tgh

For almost all cases, I prefer Linux Software RAID over proprietary hardware RAID cards.  Especially if you're using mdadm without a configuration file.  The big advantages are:

- you're not tied to a specific vendor's hardware implementation

- if the interface card goes bad, you can move the drives to a different make/model card

- it's easy to move disks between machines and get the RAID back up and running on a new box

- you can pop a Gentoo LiveCD in and mount the software RAID using mdadm

So mdadm is very flexible in my mind over hardware RAID.  

...

So what are your goals for this server?  Massive space?  Database work?  RAID5 over 3 disks isn't going to be terribly quick (unless you tune your stripe size correctly).  Do you want the machine to keep operating if one of the O/S disks fails?  

Since you're going with (3) drives and want to use RAID5 to maximize space...

sda

sda1 128MB /boot (member of RAID1 md0)

sda2 2GB swap (member of RAID1 md1)

sda3 8GB / root partition (member of RAID1 md2)

sda4 214GB (member of RAID5 md3)

sdb

sdb1 128MB /boot (member of RAID1 md0)

sdb2 2GB swap (member of RAID1 md1)

sdb3 8GB / root partition (member of RAID1 md2)

sdb4 214GB (member of RAID5 md3)

sdc

sdc1 10GB /backup partition (not a member of any RAID)

sdc2 (member of RAID5 md3)

This gives you the following software RAID devices

md0 - 128MB RAID1 for /boot

md1 - 2GB RAID1 for swap (server will keep running even if a disk dies)

md2 - 8GB RAID1 for / partition

md3 - 428GB RAID5 used for LVM partitions

Then you'll allocate your additional partitions out of LVM area, for instance:

1GB - /home

2GB - /tmp

1GB - /opt

1GB - /usr

4GB - /usr/portage

4GB - /usr/src

2GB - /var

2GB - /var/tmp

8GB - /var/log

4GB - /var/svn

300GB - /media

(leave the rest of the LVM2 area free for future expansion or other partitions)

(Note that these space considerations are what I'm working on for a new server with special needs.  Your space considerations would be different, mostly for /home and /var/log needing to be larger/smaller.  Plus you probably won't have a /var/svn in the mix unless you use SubVersion to track changes to server configuration files.)

It's been a year since I setup my last few servers, but I blogged about it (www.tgharold.com/techblog/).  I've always used mdadm + LVM2 to setup my systems.  Once you do it a few times, it becomes second nature and provides a lot of flexibility (such as resize of partitions down the road).

The 10GB backup area on sdc could be used to write snapshots of / too.  Or you may wish to partition it similar to the first two disks and set those partitions up as hot spares for md0 / md1 / md2.  In which case you might want to make the root partitions (sda3/sdb3) larger so that there are fewer O/S files on the RAID5 area.

----------

## freetown

mgbowman,

What motherboard are you using? Does it have a PCIe 4-lane slot or a PCI-X slot? Why three disks?

Hardware raid controllers were actually far slower than software raid until recently. I hope you enjoy your fancy new machine.

----------

## sf_alpha

If you are trying to use RAID5 over 3 Hdd.

Use RAID0 Instead because 3 HDD RAID5 have bad performance.

LVM on top of Software Raid is good idea. But on my production server, My "/boot" partition is on COMPACT FLASH

----------

## chrismortimore

 *sf_alpha wrote:*   

> Use RAID0 Instead because 3 HDD RAID5 have bad performance.

 In my eyes, RAID0 has one place: impatiant kiddies who want a uber-fast computer with little regard of it snuffing, and generally these kiddies have never had a hard drive fail on them anyway, hence the bad choice of RAID level.

I've used RAID5 over 3 drives in the past, and the performance was fine for me.  Granted, it wasn't exactly 3x[speed of one drive], but there was a definite increase, with the nice safety net of the parity.  One thing though, the parity calculations can strain the CPU if the server gets busy, so you'll want to make sure you have something decent in the box.

What is your budget?  Could you spring for an extra drive and go RAID6?  All the fun of RAID5, but you can lose 2 drives before you're in trouble  :Wink: 

----------

## NeddySeagoon

chrismortimore,

Raid0 is for those who go into it with their eyes open. You get the reliability of Drives/n where n is the number of drives in the partition. However, when you need the speed, the alternative is to splin the platter n times faster.

15,000 RPM drives fail more oftern than 7,200RPM drives.

I think I would rather have two slower drives in raid0 than one faster drive. At least I can use avaialbale hardware to the the box back up if part of the raid set fails. I don't need to wait for a new drive to arrive.

Of couse - both options need backups.

----------

## sf_alpha

I MEAN Software Raid 5 ... only. HW Raid is fine for 3 drivers  :Smile: 

Or use RAID 1 or RAID 10 if want redundancy.

BTW: I ever got trouble on 2 drives fail with 3 drives RAID 5. ... Anyway BACKUP is more important.

IF you have money. Use Software raid and use /boot from COMPACT Flash is real good, connect to IDE or USB.

My 256MB CF have /boot and whole small gentoo on it

----------

## chrismortimore

 *NeddySeagoon wrote:*   

> chrismortimore,
> 
> Raid0 is for those who go into it with their eyes open. You get the reliability of Drives/n where n is the number of drives in the partition. However, when you need the speed, the alternative is to splin the platter n times faster.
> 
> 15,000 RPM drives fail more oftern than 7,200RPM drives.
> ...

 I agree, but I've personally yet to come across a situation where speed is more important than reliability.

And I also agree with backups, hence I have two of every drive in my system and rsync from the main to the backup on a weekly basis.  I considered RAID1, but thought that I mostly only use backups for when I accidently delete a file or it becomes corrupt for whatever (currently unknown) reason, so rsync proved to be a better solution.

----------

## Ast0r

 *chrismortimore wrote:*   

> I've used RAID5 over 3 drives in the past, and the performance was fine for me.  Granted, it wasn't exactly 3x[speed of one drive]

 

RAID5 with 3 drives is often slower than a single drive.

What exactly is your server going to be doing? If you need redundancy, RAID5 is decent for redundancy and better-than-single disk performance between 4 and 15 disks. It's good for building decent-sized arrays and having a pretty good chance that 2 disks will not fail simultaneously, but it's not good for small arrays with 3 disks. You're going to take a huge performance/capacity hit (you lose a whole disk to parity data).Last edited by Ast0r on Wed Aug 16, 2006 5:00 pm; edited 1 time in total

----------

## mgbowman

The server is going to be a basic enterprise level server for a software/security startup. It's going to handle routing of our network traffic, provide NAS, apache, postfix, imap, etc. type apps. The reason for RAID-5 was for the redundancy in being able to loose one drive and still have a working system. I must admit that I am new to RAID (this is my first attempt whatsoever) but am open to suggestions from experienced users.  The box is in a very pre-mature state and I can redo everything in a night - it's a Pentium D 3.2 so it's a beast when it comes time to emerge  :Smile: 

Thanks for all the input, I really do appreciate it as I like doing everything right the first time.

--mgb

----------

## Ast0r

 *mgbowman wrote:*   

> The server is going to be a basic enterprise level server for a software/security startup. It's going to handle routing of our network traffic, provide NAS, apache, postfix, imap, etc. type apps. The reason for RAID-5 was for the redundancy in being able to loose one drive and still have a working system. I must admit that I am new to RAID (this is my first attempt whatsoever) but am open to suggestions from experienced users.  The box is in a very pre-mature state and I can redo everything in a night - it's a Pentium D 3.2 so it's a beast when it comes time to emerge 
> 
> Thanks for all the input, I really do appreciate it as I like doing everything right the first time.
> 
> --mgb

 

I can highly recommend the 3ware 9000 series of RAID cards. Most come with 128MB of RAM, do full hardware RAID, and have very good I/O at a very low transaction cost to the CPU.

I have two servers with 9000 series cards (the 9000 series are all SATA, btw).

1.) File server/devel server at our corporate office in San Antonio. It has 4x300GB Seagate 7200rpm disks in RAID5 (around 900GB usable) on a 3ware 9500S card. I am using 64k stripe size and have done the recommended 3ware kernel setting optimizations. Here is the output of hdparm -tT /dev/sda

```
esgfiles ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   3860 MB in  2.00 seconds = 1932.03 MB/sec

 Timing buffered disk reads:  518 MB in  3.01 seconds = 172.11 MB/sec

```

Not bad.

2.) Our first production server. Currently this machine runs PostgreSQL, Apache, PHP, and a couple other services. Now that I have the disk tuned properly it's going to be loosened from the burden of Apache soon and left to be a dedicated DB server. This server has 5x300GB Seagate 7200 RPM drives in RAID5 (about 1.2TB usable) on the 3ware 9550SX-8. I am using 64k stripe size and have done the recommended 3ware kernel setting optimizations on this machine too. Here is hdparm -tT /dev/sda

```
yang ~ # hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   3556 MB in  2.00 seconds = 1777.89 MB/sec

 Timing buffered disk reads:  648 MB in  3.01 seconds = 215.13 MB/sec
```

> 200MB/s? Sweet right? Yes, it is sweet, but it also cost a lot of money. I mean, for a good production server, it was cheap, but for some people it's too much. The 5-disk configuration cost us about $1600 and the 4-disk configuration cost us about $1400. Keep in mind, this is JUST for the hard drives and RAID controller. If you're on a budget (under $1500 for the server) then this is not an option for you.

However, if you have the money, it's really the way to go.

----------

## tgh

If you're new to RAID, I'd recommend RAID1 over RAID5.  Why?  Because you can boot with a single disk, you can even access the boot and root partitions directly.  There's also the trick of removing the secondary drive, putting it on a shelf, putting a fresh drive in and keeping the removed drive as a backup image.  It's also easier at a later point to upgrade one disk in the RAID to a larger size, let the array re-sync, then drop in a larger disk for the other one and re-sync again.

(It's probably possible to grow a RAID5 array, but I've done the RAID1 array growth and know for sure that it's fairly simple and it works.)

If I have 4 disks, I prefer 2 RAID1 sets, because it allows me to have two independent sets of spindles to spread the disk load across.  PostgreSQL in particular becomes extremely happy when it has a 2nd set of spindles for its log files.

...

Also, when you talk about "routing of our network traffic", are you including firewall duties?  I'd strongly suggest putting those duties on a separate box.  (You can build a dual-core firewall box for around $550 for hardware.  Or $840 if you want something *really* tiny, say 13" x 17" x 4".)

----------

## Ast0r

 *tgh wrote:*   

> If you're new to RAID, I'd recommend RAID1 over RAID5.  Why?  Because you can boot with a single disk, you can even access the boot and root partitions directly.  There's also the trick of removing the secondary drive, putting it on a shelf, putting a fresh drive in and keeping the removed drive as a backup image.  It's also easier at a later point to upgrade one disk in the RAID to a larger size, let the array re-sync, then drop in a larger disk for the other one and re-sync again.
> 
> (It's probably possible to grow a RAID5 array, but I've done the RAID1 array growth and know for sure that it's fairly simple and it works.)
> 
> If I have 4 disks, I prefer 2 RAID1 sets, because it allows me to have two independent sets of spindles to spread the disk load across.  PostgreSQL in particular becomes extremely happy when it has a 2nd set of spindles for its log files.
> ...

 

You can build a firewall box for less than this, depending on the traffic that you are expecting it to handle. Just get some cheap Celeron D or Sempron with integrated network, video, etc ... put in two small hard drives in RAID1 and go. You might even be able to scroung up the hardware for it; it doesn't need much.

----------

## tgh

Alternately, if you have 4 spindles, you may want to try RAID 1+0.  I'm not 100% sure how it gets setup in mdadm yet, but it's a RAID0 spread across two RAID1 sets.  It could double your performance over a separate set of RAID1s.

It's a bit more complex then two RAID1 sets and trickier to partition.  And performance is usually best if the four disks in the set aren't competing for the disk with other volumes on the same disks.  (So you might need 6 disks, 2 for the O/S RAID1 and 4 for the RAID 1+0.)

----------

