# Performance problems with mdadm (Raid6) + Crypto Luks + LVM2

## rralf

Hi!

I have terrible performance problems with my hard disk setup.

This is my Configuration:

4 x 3TB Sata hard drives (/dev/sd[abcd]) formatted with a GPT Partition Table.

/dev/sd[abcd]1 : 100 MiB Boot Drive (RAID1 without encryption/LVM) <- works fine

/dev/sd[abcd]2:  2.73TiB Data Storage (Raid6) <- problem child

According to gparted, both partitions are aligned.

This is what I did for formatting the data storage:

Raid Setup:

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sd[abcd]2

(15 hours of initialization...  :Wink:  )

Speed of initialization: ~30-50Mib/s <- In my opinion, even this is slow

If I do "dd if=/dev/zero of=/dev/sda2 bs=512K count=1024" I have round about 100MB/s write speed.

Well...

Crypto Setup:

cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --key-size 256 /dev/md0

cryptsetup luksOpen /dev/md0 Storage

LVM2 Setup:

According to /proc/mdstat, chunk size of my Raid6 is 512KiB

pvcreate --dataalignment 512K -M2 /dev/mapper/Storage

vgcreate vg /dev/mapper/Storage

lvcreate -L100G -nStore1 vg

mkfs.ext4 -j -LStore1 /dev/vg/Store1

mount /dev/vg/Store1 /mnt/foobar

Everything works fine except of the performance  :Smile: 

Read speed is about ~100MiB/s. That's okay for me.

But write speed is ~20MiB/s, that's really bad.

What am I doing wrong?

I'm sure write speed could be faster than 20Mib/s....

Thanks for help!

Best regards,

Ralf

----------

## chithanh

Do you have the same problem when you create the filesystem directly on /dev/md0 (instead of using lvm)?

----------

## rralf

No, I tried it out.

When i directly create a filesystem on /dev/md0, performance is fine.

----------

## chithanh

So given that read performance is fine and write performance without LVM is fine too, the problem looks like an LVM alignment issue.

----------

## rralf

Yes, probably in combination with cryptsetup, but i don't know what I'm doing wrong...

I tried to align everything correctly...

----------

## chithanh

Ah sorry, I misread what you wrote above.

What if you create the filesystem on /dev/mapper/Storage? How is the performance then?

EDIT: cryptsetup does have an --align-payload option, did you try that too?

----------

## rralf

Ok, i read some good articles:

https://support.mayfirst.org/wiki/disk_alignment

http://wiki.drewhess.com/wiki/Creating_an_encrypted_filesystem_on_a_partition

http://blog.kanojo.de/2011/10/11/raid-system-8tb-home-storage-on-budget/

After tweaking and trying i got it a bit better:

echo 32768 > /sys/block/md127/md/stripe_cache_size

This brought an amazing speedup!

Read speed: ~200MB/s, tested with:

dd if=/dev/md127 of=/dev/null bs=512K

Write speed: ~120MB/s

dd if=/dev/zero of=/dev/md127 bs=512K

That's quite good.

Cryptsetup tweaking

cryptsetup luksFormat --argsfoobar --align-payload 4096 /dev/md127

cryptsetup luksOpen /dev/md127 vault

...

Crypto Read speed: ~100MB/s, tested with:

dd if=/dev/mapper/vault of=/dev/null bs=512K

Crypto Write speed: ~90MB/s

dd if=/dev/zero of=/dev/mapper/vault bs=512K

That's not the truth... Read speed is half than before, write speed decreased also by 30MB/s....

But I suppose, that this is the crypto overhead.

LVM Creation:

pvcreate --metadatasize 2048K --dataalignment 4096 -M2 /dev/mapper/vault

vgcreate vg /dev/mapper/vault

lvcreate -nGentoo -L30G vg

LVM Read speed: ~100MB/s, tested with:

dd if=/dev/vg/Gentoo of=/dev/null bs=512K

LVM Write speed: ~90MB/s

dd if=/dev/zero of=/dev/vg/Gentoo bs=512K

So speed doesn't decrease when using LVM. (This is an important information for me....)

After creating the ext4fs using

mkfs.ext4 -j -LGentoo -b 4096 -E stride=128,stripe-width=256 /dev/vg/Gentoo

mount /dev/vg/Gentoo /mnt/gentoo

Ext4 Read speed: ~100MB/s, calculated with:

sync;date;cp /mnt/gentoo/3gbfile /mnt/ramfs/foobar;sync;date

Ext4 Write speed: ~85MB/s, calculated with:

sync;date;cp /mnt/ramfs/3gbfile /mnt/gentoo/3gbfile;sync;date

So the bottleneck seems to be the cryptolayer....

Anyone further ideas?

Thanks!

----------

## chithanh

What is the read/write speed if you create a luks volume on a single disk?

Also if your CPU does not support AES acceleration, better use 128-bit encryption (aes-xts-plain with 256 bits, half of which are the AES key) and not 256-bit (aes-xts-plain 512).

----------

## rralf

R/W on a single disk: ~70MB/s

Urgh, according to /proc/cpuinfo my CPU doesn't support AES instruction set.

Damn....

----------

## chithanh

```
# openssl speed aes-128-cbc
```

can tell you how fast you CPU does AES-128. If the CPU does AES-NI hardware acceleration,

```
# openssl speed -evp aes-128-cbc
```

will show you the accelerated speed (requires >=openssl-1.0.1).

----------

