# lvm on sw raid5, yee or nee? [answered]

## DaggyStyle

I'm about to redeploy my rigg on a sw raid 5 and I've wanted to know, does deploying lvm on the raid is a good idea performance wise?

Thanks.

----------

## eccerr0r

It's not that great but acceptable in my opinion:

Disk hardware: 4x120GB PATA (Seagate/Maxtor)

CPU hardware: AthlonXP 2200+

Disk Controllers: Onboard SiS735 and a Promise Ultra66, one disk per channel (yes, I have 4 80-pin PATA cables flying around.)

Software: mdadm raid5 0.9

Peak I get around 130MB/sec to /dev/md1 or so on reads but through lvm and the filesystem it's a lot lower.  I suppose it's in the "acceptable" range but not "optimal" 50-60MB/sec I think - this mostly is for redundancy over speed...

I have another system that is 4x500G SATA using onboard SATA ports, it doesn't bench much faster...

----------

## DaggyStyle

 *eccerr0r wrote:*   

> It's not that great but acceptable in my opinion:
> 
> Disk hardware: 4x120GB PATA (Seagate/Maxtor)
> 
> CPU hardware: AthlonXP 2200+
> ...

 

so you lose between 54% to 62%?

not quite acceptable on my part, my setup will be 1TBx4 on 3.0 sata

----------

## frostschutz

LVM does not have any negative/noticable impact on performance. All that's required to make it work is simple address translation. This is the most common operation that exists as it has to be done everywhere and all the time; partitions do it, RAID does it, filesystems do, even processes which each get their own virtual address space in memory; every single operation they do has to be translated to actual physical addresses first. The system will be quite bored if it has to do an extra translation for the LVM layer really.

So yes, use LVM if you like. Probably not necessary if all you want is one whopping big data single filesystem on that RAID anyways. But if you want to split it up in smaller pieces, LVM is the way to go. I use LVM everywhere, can't do without.  :Smile: 

----------

## eccerr0r

Probably need to drop LVM, LVM seems to be a lot of overhead.  But it makes volume management more manageable...

/dev/md0:

 Timing buffered disk reads: 150 MB in  3.04 seconds =  49.41 MB/sec

/dev/md1:

 Timing buffered disk reads: 366 MB in  3.02 seconds = 121.39 MB/sec

/dev/md2:

 Timing buffered disk reads: 338 MB in  3.01 seconds = 112.27 MB/sec

/dev/vg0/l0:

 Timing buffered disk reads: 208 MB in  3.03 seconds =  68.65 MB/sec

/dev/vg0/l1:

 Timing buffered disk reads: 190 MB in  3.03 seconds =  62.61 MB/sec

md0: Boot RAID.  md1: Root RAID. md2: storage RAID.  vg0/l0 and vg0/l1 are within md2.  Kills eh?

Hmm.. one thing I have not considerred is ... argh... need to worry about block sizes.  Oh well, need to redo this I guess...

----------

## DaggyStyle

 *frostschutz wrote:*   

> LVM does not have any negative/noticable impact on performance. All that's required to make it work is simple address translation. This is the most common operation that exists as it has to be done everywhere and all the time; partitions do it, RAID does it, filesystems do, even processes which each get their own virtual address space in memory; every single operation they do has to be translated to actual physical addresses first. The system will be quite bored if it has to do an extra translation for the LVM layer really.
> 
> So yes, use LVM if you like. Probably not necessary if all you want is one whopping big data single filesystem on that RAID anyways. But if you want to split it up in smaller pieces, LVM is the way to go. I use LVM everywhere, can't do without. 

 

I do need smaller partitions but not in expense of over 50% performance impact.

 *eccerr0r wrote:*   

> Probably need to drop LVM, LVM seems to be a lot of overhead.  But it makes volume management more manageable...
> 
> /dev/md0:
> 
>  Timing buffered disk reads: 150 MB in  3.04 seconds =  49.41 MB/sec
> ...

 

you're very much convinced me, I'd rather have excellent performance than the ability to easily expand and shrink.

just need to partition the raid5 smartly.

Thanks.

----------

## frostschutz

 *DaggyStyle wrote:*   

> I do need smaller partitions but not in expense of over 50% performance impact.

 

In that case I suggest you run your own benchmarks.

There's virtually no overhead with LVM. Loads of people use it without any issues whatsoever.

----------

## Deathwing00

Moved from Installing Gentoo to Kernel & Hardware.

----------

## eccerr0r

Anyone else have this configuration (LVM over MD RAID5)? According to some people there shouldn't be this much degradation... but on my particular computer it's very clear...

I've heard people with raid0 and raid1 setups that maintain speeds but haven't heard any raid5...

----------

## DaggyStyle

I've decided to go with raid5 without lvm, although it would be nice to have the option, till now I didn't had to use it.

thanks for the help and eccerr0r, I hope you'll be able to solve your performance issues.

----------

## gentoo_ram

I'm not seeing any degradation in performance with 'hdparm'.  Notice though that these tests are only a couple seconds, so I'm not sure how useful they are.

My RAID5 is in an external enclosure with 5 disks going through a PCIe eSATA connector.  

```

04:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

md20 : active raid5 sdg1[4] sdf1[3] sde1[2] sdd1[1] sdc1[0]

      7814047744 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]

/dev/sdf1:

 Timing buffered disk reads:  336 MB in  3.00 seconds = 111.89 MB/sec

/dev/md20:

 Timing buffered disk reads:  384 MB in  3.01 seconds = 127.54 MB/sec

/dev/mapper/vg-media:

 Timing buffered disk reads:  376 MB in  3.01 seconds = 124.74 MB/sec

```

I also have a second physical volume in this volume group that's RAID1.  But the LVM volume vg-media is on the md20 physical volume which is the RAID5.

```

Linux 3.5.0-gentoo #1 SMP Tue Jul 24 14:48:37 PDT 2012 x86_64 Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz GenuineIntel GNU/Linux

  VG   #PV #LV #SN Attr   VSize VFree

  vg     2  11   0 wz--n- 8.18t 1.47t

  --- Physical volume ---

  PV Name               /dev/md15

  VG Name               vg

  PV Size               923.71 GiB / not usable 25.69 MiB

  Allocatable           yes 

  PE Size               64.00 MiB

  Total PE              14779

  Free PE               8059

  Allocated PE          6720

  PV UUID               DmUyz2-leCp-fp8V-KCvM-950N-SNx8-GWtQQR

   

  --- Physical volume ---

  PV Name               /dev/md20

  VG Name               vg

  PV Size               7.28 TiB / not usable 58.00 MiB

  Allocatable           yes 

  PE Size               64.00 MiB

  Total PE              119232

  Free PE               15952

  Allocated PE          103280

  PV UUID               5fsrfe-neyF-SayP-Ya9V-DKKx-W7WX-I07ZRE

```

----------

## eccerr0r

This is my other RAID5 (also 4-disk), all SATA with NCQ enabled, my Core2 Quad with P43 chipset (PCIe.  I hope I'm not running too close to the bandwidth limit of the SATA controllers):

```
/dev/sdb:

 Timing buffered disk reads: 268 MB in  3.01 seconds =  89.10 MB/sec

/dev/md0:

 Timing buffered disk reads: 132 MB in  1.71 seconds =  76.97 MB/sec

/dev/md1:

 Timing buffered disk reads: 674 MB in  3.01 seconds = 224.17 MB/sec

/dev/v0/root:

 Timing buffered disk reads: 532 MB in  3.01 seconds = 176.56 MB/sec

```

Still a significant drop in speed over the raw RAID5. (sdb=single disk, md0=raid1, md1=raid5, and/dev/v0/root is one logical volume within md1)

Honestly I don't know why the raid5 array itself is pretty slow compared to the speed bump my other RAID got.  Well, maybe it's not as slow as I thought... 3*singledisk minus a little...  I guess both machines are around 3*singledisk times some factor less than 1...

The other machine is an AthlonXP machine with SiS735 (PCI) chipset.  I'm likely running close to the PCI bandwidth limit.

In terms of speed the raid gets blown away by my i7's SSD...  so I guess I don't care way too much about it now...

----------

## 1clue

From a purely practical standpoint, LVM2 makes all kinds of sense but RAID is not a cure-all.

IMO, if you NEED to stay up in event of a drive failure, or you NEED to combine multiple disks for some reason, then RAID is worth a look, otherwise it's irrelevant.

RAID can't be a complete backup strategy.  You are not protected from lightning strikes or fire or flood or whatever other disaster that might completely destroy your data center, and it can't protect from software corruption or damage by intrusion or loss of data to theft.  Those things reduce its value quite a bit, to darn near 'marginal' IMO.

I've been playing with RAID just to learn more about how it works, but in the end my backups are onto a removable slot-loaded ESATA drive. I just back up my databases to whatever file is appropriate in my archives folder, straight copy everything else (including home directories), then plug my drive in and copy everything I want to keep onto that drive.  No compression, no funky tools, just copy with permissions preserved.  That way there's no maximum archive size other than the size of the disks.

That strategy gives me the latest backup of whatever it was on the 'permanent' filesystem tree, and a disk that travels with me just in case which holds several copies staggered with the other drive stored off site.  One in my backpack, one at another site and the latest on disk covers just about any disaster I can think of.

----------

## DaggyStyle

 *1clue wrote:*   

> From a purely practical standpoint, LVM2 makes all kinds of sense but RAID is not a cure-all.
> 
> IMO, if you NEED to stay up in event of a drive failure, or you NEED to combine multiple disks for some reason, then RAID is worth a look, otherwise it's irrelevant.
> 
> RAID can't be a complete backup strategy.  You are not protected from lightning strikes or fire or flood or whatever other disaster that might completely destroy your data center, and it can't protect from software corruption or damage by intrusion or loss of data to theft.  Those things reduce its value quite a bit, to darn near 'marginal' IMO.
> ...

 

there is no such thing as fullproof, even your current way, all backups can easily be stolen, infact it is easier to steal the removable drives than the original.

also nature disasters can affect all sites, not to mention hd failures.

being cautious is ok, being paranoid is not so much.

----------

## 1clue

It's true that theft of your data is easier if it exists in multiple places.

It's also true that if your only backup strategy is RAID, then theft of your data means that YOU no longer have your data, only the thieves do.

IMO the theft of my data is very bad, but complete loss of that data to the hands of somebody else makes theft of a copy look like a nice dinner in a fancy restaurant.

I'm definitely paranoid.  The question is, am I paranoid enough?

----------

## 1clue

Sorry for the double-post.

You need to scale your backup strategy to the value of your data.  For the case of my data, a disaster which puts all my backups out of reach will almost certainly cause my data to be irrelevant.  My backups are far enough apart that a natural disaster is unlikely to destroy everything.  All I can imagine would be a tornado  which stays down for more than 50 miles, or two tornadoes (the statistical probability of even one tornado hitting me is similar to winning big on the lottery) or an earthquake of epic proportions, or a hurricane.  Hurricanes don't get to the Chicago area too often.   :Smile: 

In any case, in my personal situation I don't really need to increase the size of my drives, and I my world won't end if the system crashes due to a drive failure.  There are 2 reasons I've ever installed RAID:

My employer wanted a high availability system, and I was installing it.

Curiosity/entertainment/education, yes all wrapped in the same bundle.

I just scrolled back to see the original post.  I think I may have misread the purpose of the thread.  I thought the RAID5 was optional, and it seems on another glance that it's the LVM which is optional.  My apologies if this isn't really relevant.

----------

## eccerr0r

In my past history, unexpected drive failure was the #1 cause of data loss for whatever reason.  Hackers, rm -rf botching, and bad software were very minor compared to disk failure.

In the past couple of times I lost data, it was the time between backups that was lost.  If it had waited out till the next scheduled backup I'd not be so upset, but it didn't.

RAID turned out to be the solution to help against loss of the last few hours of work lost.  I've never liked wasting disk space for redundancy so I picked RAID5 over RAID1 -- and hoping that I'd also get a speed boost which will offset the loss of disk space.  Now I do see a large bump from single disk to RAID5, it looks like a lot of people, maybe everyone including myself are not seeing near theoretical speed bump from single disk -> RAID5 -> LVM.

My PATA array: (two SiS735 channels (MASTER, no slaves) and two Promise Ultra66 channels (MASTER, no slaves), 4 120GB PATA disks)

-> gain of about 5% over single disk (4 disk RAID5)... horrible (59MB/sec to 62MB/sec)

My SATA array: (ICH9 non-R on-board, 4 500GB SATA disks)

-> gain of about 90% over single disk (4 disk RAID5)... tolerable but it should be higher. (80MB/sec to 150MB/sec)

gentoo_ram's array: (SiI 3132 RAID controller)

-> gain of about 10% over single disk (5 disk RAID5)... horrible in my opinion. (112MB/sec to 125MB/sec)

Anyway, I still back up to another media, so it'd take three disk failures to lose data (two to break the RAID, and the third is the backup disk.)

----------

## DaggyStyle

where I live, hw costs are quite high, not everyone can buy all needed, my previous purchase was 2 1tb internal hds to get the raid up and running, the next one is a real ups.

then I may consider getting an external hd for backup but that is all I'm willing to go.

I'm not paranoid.

----------

## drescherjm

 *eccerr0r wrote:*   

> Anyone else have this configuration (LVM over MD RAID5)? According to some people there shouldn't be this much degradation... but on my particular computer it's very clear...
> 
> I've heard people with raid0 and raid1 setups that maintain speeds but haven't heard any raid5...

 

At work I have used this lvm over linux software raid6 for over a decade on all of my servers. I am not sure if lvm causes any noticeable performance loss. Although my raid6 arrays have STR of 500 to 900 MB/s depending on the system and I only use a single gigabit link gigabit to the clients. When thinking about this one thing that my effect performance is read ahead that lvm has.

----------

## eccerr0r

Once again, in theory it doesn't but apparently in practice it does.  Both of my arrays exhibit the degradation.  I did do some tuning to get my md arrays as fast as possible but once lvm comes into the picture, the bonus is lost.

Read ahead definitely can have an issue.  With the stacking, each layer may dump many unneeded reads and writes to the mechanical disks, which will affect performance especially if the heads are involved.  I suspect SSDs will not take as much hit as they don't have heads to move...

----------

## 1clue

Maybe I missed the hardware, but I suspect that if you are using the onboard disk controller for a fairly mainstream desktop box, then (pure speculation on my part) maybe the controller isn't so good as a controller card you buy, or so good as a controller on real server hardware.

I'm using an onboard SATA controller on an Asus P6T and I never was impressed by my RAID performance, it was noticeably slower than a regular disk but I never bothered to measure it.

----------

## eccerr0r

However, both of my cases RAID5 is faster than single disk by itself, approaching theoretical read speeds.  It's once LVM got added things went down the toilet.

There must be some tuning for LVM that could bring speeds back up... or something that can be done to MD that would speed up LVM...

----------

## energyman76b

never had the need for lvm. And with bind mounting there is even less reason to use it.

----------

## drescherjm

At work and home I use lvm to have separate filesystems (which I can easily do online grow/shrink) for very different types of data. For example at work public, private, lxc containers, data analysis, temp data, medical images, databases, software and distfiles all get their own filesystems. Although as I begin to use btrfs for important data I am reducing the need for lvm.

----------

