# software raid1 + lvm2 = performance drop - is this true ?

## noobstate

http://gentoo-wiki.com/HOWTO_Gentoo_Install_on_Software_RAID_mirror_and_LVM2_on_top_of_RAID

the above states that there is a performance drop when using LVM2 ontop of software raid(mdadm) is that true ? 

can anyone verify this ?Last edited by noobstate on Mon Jan 28, 2008 7:08 pm; edited 1 time in total

----------

## schachti

Apart from the fact that the posted link does not work: Yes, of course, software RAID needs more CPU power than no RAID at all.

----------

## noobstate

 *Quote:*   

> link fixed - ma bad

 

 *schachti wrote:*   

> Apart from the fact that the posted link does not work: Yes, of course, software RAID needs more CPU power than no RAID at all.

 

i can understand a little processing power of course, but what about hard drive seek time ? isnt the entire point of RAID1 for it to be performance enhancing (i would assume reading at the very least since there are two and i can do synchronized reading )?

also im wondering about specific LVM2 ontop of RAID1 30-50% as the wiki describes it in losses ... is that really what it accounts to if u use it ... ?

maybe someone who has both can check

----------

## andreas_st

 *noobstate wrote:*   

> isnt the entire point of RAID1 for it to be performance enhancing

 

No, absolutely not. RAID1 is for data redundancy. Reading from RAID1 can be faster but writing is definitely much slower.

----------

## noobstate

 *andreas_st wrote:*   

>  *noobstate wrote:*   isnt the entire point of RAID1 for it to be performance enhancing 
> 
> No, absolutely not. RAID1 is for data redundancy. Reading from RAID1 can be faster but writing is definitely much slower.

 

seems logical  enough i believe it 

now if anyone can just verify the LVM2 theory it would be much appreciated.

----------

## HeissFuss

My ad hoc tests show about 10-13% speed decrease using LVM2 vs a normal partition for a single disk with various file systems (no raid) for large file writes (4GB.)  I used both 1MB(as in the article) and 4096K blocksize with a similar percentage spread.  The 30-50% stated in that wiki article is plain wrong, unless testing was done on a 200Mhz celeron.  My tests were done on a 3Ghz P4, and keep in mind that this was with a single large file.  I didn't test numerous small files or simultaneous writes.

It really depends on what you'll be using the storage for.  LVM2 offers a lot of flexibility over plain partitions such as snapshots and dynamic allocation from a pool.  For example, say you have a 500GB VG and 4 100GB LV file systems.  One file system fills up.  You can add more space to that file system from the remaining 100GB pool.Last edited by HeissFuss on Wed Jan 30, 2008 7:26 pm; edited 1 time in total

----------

## gentoo-dev

 *andreas_st wrote:*   

>  *noobstate wrote:*   isnt the entire point of RAID1 for it to be performance enhancing 
> 
> No, absolutely not. RAID1 is for data redundancy. Reading from RAID1 can be faster but writing is definitely much slower.

 Not much slower, a bit slower, writes happen in parallel unless both drives are on a single IDE channel.

----------

## noobstate

 *HeissFuss wrote:*   

> My ad hoc tests show about 10-13% speed decrease using LVM2 vs a normal partition for a single disk with various file systems (no raid) for large file writes (4GB.)  I used both 1MB(as in the article) and 4096K blocksize with a similar percentage spread.  The 30-50% stated in that wiki article is plain wrong, unless testing was done on a 200Mhz celeron.  My tests were done on a 3Ghz P4, and keep in mind that this was with a single large file.  I didn't test numerous small files or simultaneous writes.
> 
> It really depends on what you'll be using the storage for.  LVM2 offers a lot of flexibility over plain partitions such as snapshots and dynamic allocation from a pool.  For example, say you have a 500GB VG and 4 100GB LV file systems.  One file system fills up.  You can add more space to that file system from the remaining 100GB pool.

 

hmm on my desktop iv been using lvm2 for a while and when i ran out of space (because i keep low lvm2 size to start as i read its better on the seek time then keeping large drives with unused space) i tried to expand it and no go. so i didnt wana put it on my server for this specific reason. i tried live (aka while it was mounted) and i tried with livecd with no partition mount then it finally let me expand it. but with such a small drive and odd cylinder setup it had left overs it woulnd let me mount, leaving a little space therefor unusable 

as for snapshots where have i been LOL it would be much easier and simpler for me to use this method to backup then to rsync manually onto a spare dvd every week. i have to look into that never even know that.

----------

## noobstate

 *gentoo-dev wrote:*   

>  *andreas_st wrote:*    *noobstate wrote:*   isnt the entire point of RAID1 for it to be performance enhancing 
> 
> No, absolutely not. RAID1 is for data redundancy. Reading from RAID1 can be faster but writing is definitely much slower. Not much slower, a bit slower, writes happen in parallel unless both drives are on a single IDE channel.

 

i assumed that the advantage of information seek time to write time is irrelevant making it more productive to use it

----------

## the.ant

If you are not looking for redundancy but speed improvement, omit the raid and set up a sliced LVM. 

What I would be curious about is if some filesystems work better with lvm than others.

----------

## noobstate

 *the.ant wrote:*   

> If you are not looking for redundancy but speed improvement, omit the raid and set up a sliced LVM. 
> 
> What I would be curious about is if some filesystems work better with lvm than others.

 

i did speed tests a while ago, it turned out reiserfs was the quickest (because of all the small files) however SELinux was not compatible with it. so i then moved onto xfs (as it had realtime) first kernel lock , and bam lost all information (not reliable) so i went to ext3 which i have had no problems with as of yet

ill look into sliced LVM  when i get home i have yet to read about it or know what the difference between simpe lvm is

and now that u mentioned it i too wouldnt mind seeing lvm tests with filesystems

unfortunately i wish to go with raid to be on the safe side as its a server running this 24/7 under a staircase u never know

----------

## the.ant

Sorry, I meant striped LVM. 

Basically like a Raid 0, designed for performance boost (provided that all PVs of the LVM are separate disks and not on the same ide channel). A benefit to Raid0 is that the disks afaik don't have to be of the same sizes. 

Also, the striping is done per logical volume, which is actually pretty cool. So, you bundle all your disks into a volume group and then create the LVs to your liking. You can choose per volume how many disks should be used and how large the stripes should be. I haven't tested it extensively, but I guess that you can get a good performance increase if you choose small stripes for partitions with small files. I could imagine there is quite some room for finetuning stripe-size of the volume and block-size of the filesystem. The partitions where you have the really important data, you just don't stripe, then the data is only on one disk (provided it's large enough). 

Also, Raid1 is afaik 'just' a complete mirror of the disk, meaning you can use only half of the hd-space you actually have. With LVM you can make snapshots of you system, which is more flexible since the backup only has to be the size you actually need, if you don't need to backup the whole system, it's quite an option. 

So, with three disks you stripe two and use the third for snapshots. With two disks you can create one striped volume for performance and two non-striped for redundancy mirroring each other. If you have data you don't care about at all you can even safe some diskspace. 

I only very recently discovered this topic for myself and I am by no means an expert but from my understanding I don't see much of a benefit in using raid instead of lvm (in a non-professional environment, where you don't have raid-hardware).

----------

## HeissFuss

Snapshots are not a replacement for backups.  Yes, they'll allow you to recover from an accidental delete, but since they're running on the same disks as the rest of your VG they're vulnerable to the same hardware failure issues.  You should make a backup from the snap when you take it and then remove the snap.  That will give you a point in time copy without worrying about files which are currently being written.  Also note that for snapshots you're effectively doubling the write operations you're doing because in order to keep the snap in sync you need to read the block you're going to overwrite, then write the original to the snap and write the new block to the file system.  If you have a lot of write activity, snapshots can slow your access times if you do a lot of writes, especially with a small disk count.

If you're thinking of just striping or mirroring and not any higher level RAID, you might as well use the LVM level tools for this rather than putting it on top of mdraid so that you don't need to pay twice for software physical disk emulation.

For the filesystem tests I did, ext2 was fastest on all writes followed by reiserfs for small blocks and XFS for large.  There wasn't a noticeable difference between the speed on physical disk vs loss of speed on LVM for any filesystem (none seemed above the others for LVM speed.)  The filesystems I tested were ext2, ext3, reiserfs, and XFS (I didn't have JFS built in my kernel so didn't test it.)  I have been using XFS myself for about 6 months since my files are large without any issues.  XFS had a smaller difference between small and large writes than reiserfs between small an large (reiser had a larger speed loss for larger blocks than XFS did for smaller blocks.)  By small I'm talking 4k.  If most of your files are in MB, XFS will be faster.

----------

## the.ant

 *HeissFuss wrote:*   

> Snapshots are not a replacement for backups.  Yes, they'll allow you to recover from an accidental delete, but since they're running on the same disks as the rest of your VG they're vulnerable to the same hardware failure issues.

 

Is that the case? I always thought that a damaged drive in a VG would not destroy the whole group, just the LVs on that drive (as opposed to software raids). Since you can assign LVs to drives I thought they would be safe. 

 *Quote:*   

>  If you have a lot of write activity, snapshots can slow your access times if you do a lot of writes, especially with a small disk count.

 

Yes, but isn't that the case for a raid-1 as well? You are right, it's not wise to use snapshots as replacement for backups, but to make a snapshot once a day, I think they would be good. But to mirror every operation (be it wirh raid or lvm) is IMHO overkill in a private setting. But I did not yet come up with a good backup scheme for myself as well, just thinking through different possibilities.  

 *Quote:*   

> XFS had a smaller difference between small and large writes than reiserfs between small an large (reiser had a larger speed loss for larger blocks than XFS did for smaller blocks.)  By small I'm talking 4k.  If most of your files are in MB, XFS will be faster.

 

What I don't like about ext is that it's doing far worse capacitywise than xfs, reiser or jfs. My first impression of reiserfs was good, but I did a lot of experimentation and recompiling on that machine and by now it's so heavily fragmented that I can't wait to get rid of it. JFS get good benchmarking results from what I read about it, but when I was setting up my server I was shocked how slow it was. No measurement though, just personal impression. I switched to reiser4 now (it's my experimental machine anyways), I am curious how that will do. First impression is really good, files are just flying by... 

But if I encounter any problems I guess I will switch to XFS as well, it seems to be the safest option and has a decent performance as well.

----------

## HeissFuss

 *Quote:*   

> HeissFuss wrote:
> 
> Snapshots are not a replacement for backups. Yes, they'll allow you to recover from an accidental delete, but since they're running on the same disks as the rest of your VG they're vulnerable to the same hardware failure issues.
> 
> Is that the case? I always thought that a damaged drive in a VG would not destroy the whole group, just the LVs on that drive (as opposed to software raids). Since you can assign LVs to drives I thought they would be safe. 

 

It really depends on how you lay it out.  Default LV will be laid out in serial on your VG, so theoretically you'll fill the last disk last (think JBOD.)  If you use LV striping though everything will be on all disks, so any single failure will hose all striped LVs.  Even if you aren't striping though, there's no guarantee that a drive that you lose won't be critical to LVs unless you're using mirroring.

 *Quote:*   

> Quote:
> 
> If you have a lot of write activity, snapshots can slow your access times if you do a lot of writes, especially with a small disk count.
> 
> Yes, but isn't that the case for a raid-1 as well? You are right, it's not wise to use snapshots as replacement for backups, but to make a snapshot once a day, I think they would be good. But to mirror every operation (be it wirh raid or lvm) is IMHO overkill in a private setting. But I did not yet come up with a good backup scheme for myself as well, just thinking through different possibilities. 

 

A snapshot is effectively mirroring every operation at the reduced cost of requiring only the amount of space for the changes rather than needing space for the entire disk.  The more changes you are making, the larger the size of the snapshot you'll need.  Once the snapshot fills it becomes useless.  The cost of snapshots (at least on a software level) is that the writes are more costly than mirroring since you need to do the additional read before write.  Also, the snapshot may be on the same physical disk as the LV it's a snapshot of, so performance may be further hindered there (raid 1 would at least be on a seperate physical disk.)

All of this said, for a personal system I'd suggest using RAID-5.  It's the most economical solution, and the performance isn't that bad.  You'll see very fast reads and slightly slower writes (slower than raid-0, faster than raid-1 and faster than single disk.)  Try out the daily keep-around snapshot and see if you notice performance loss.  If it's acceptable to you, that's all that matters.

Good luck   :Smile: 

----------

## roki942

 *HeissFuss wrote:*   

> 
> 
> If you're thinking of just striping or mirroring and not any higher level RAID, you might as well use the LVM level tools for this rather than putting it on top of mdraid so that you don't need to pay twice for software physical disk emulation.

 

If done this way and one drive crashes, will the system still boot without losing anything like a raid 1 set up will?

----------

## RoundsToZero

I think snapshots were meant to be used to get consistent backups from rather than being backups themselves.  That is, you take a snapshot, mount it, and then rsync files from there instead of from your real filesystem.  That way, no files in the source directory change during the backup, and you're guaranteed that your files are in a consistent state (at least as good as if the machine had simply lost power at the time you took the snapshot).

----------

