# Encryption Speed

## weingbz

Hi all,

I have an aes256 dm-crypt encrypted root partition on a raid (mirror). I'm wondering how to speed things up. In the moment I'm getting about 42MB/s read speed of a possible 190MB/s. So here are my questions:

In the kernel I can activate AES and AES (i586). Which one is faster on an Athlon X2? Which gets selected if I activate both?

I have read that some things are faster and some slower when I go from a 32bit system to a 64bit system. What about encryption speed?

In the moment decryption seems to run on one processor only, can I parallelize it?

That's all for now, thanks for the help.

----------

## jrtayloriv

I do not know the answers to all of your questions, but I can help you with a few of them:

You want AES(i586), which is for i586 and above, while the other is a generic version for all x86s -- so the i586 one will have some optimizations that the other won't. You don't need both.

I know that some algorithms, such as Tiger, are optimized for 64-bit processors, and will run less efficiently on 32-bit CPUs. You'll have to do research on your own to find out exactly which other ones are out there...

Hope this is of some help,

--jrtayloriv

----------

## Sadako

With amd64 there are x86_64 optimized AES cipher algorithms, as opposed to the i586 ones on x86 archs, so even if you're stuck using a 32 bit userland you could try a 64 bit kernel, even just for testing to see if there's any performance difference.

----------

## weingbz

 *jrtayloriv wrote:*   

> You want AES(i586), which is for i586 and above, while the other is a generic version for all x86s -- so the i586 one will have some optimizations that the other won't. You don't need both.
> 
> 

 

OK, the weekend is here and I tried some measurements. It seems that if you select both the faster one gets picked. So I already had the faster algorithm activated. The generic algorithm is about 15MB/s slower.

I also noticed that there is quite a bit of variance in my measurements. I get values between 40 and 65 MB/s. I use

```

hdparm -tT /dev/mapper/...

```

Hdparm was not exactly written for this usecase, is there a better way to test the speed?

----------

## flash49

 *weingbz wrote:*   

> 
> 
> Hdparm was not exactly written for this usecase, is there a better way to test the speed?

 

bonnie++, zcav(part of bonie++ package)

or use 

```
dd if=/dev/mapper/data_vg-bigspace_lv of=/dev/null bs=1073741824 count=1
```

 Count is the number ofGB to read.

My values , Athlon 2300BE 2x1900Mhz, Perc5/i raid5,lvm,luks(aes256,amd64 optimized on 64Bit kernel)   :Cool: 

```

dd if=/dev/mapper/data_vg of=/dev/null bs=1073741824 count=10

10+0 records in

10+0 records out

10737418240 bytes (11 GB) copied, 44.9293 s, 239 MB/s

dd if=/dev/mapper/data_crypt of=/dev/null bs=1073741824 count=1

1+0 records in

1+0 records out

1073741824 bytes (1.1 GB) copied, 13.5736 s, 79.1 MB/s

```

 *Quote:*   

> OK, the weekend is here and I tried some measurements. It seems that if you select both the faster one gets picked.

 

You can check the used crypto driver in "/proc/crypto":

```

name         : cbc(aes)

driver       : cbc(aes-x86_64)

module       : kernel

priority     : 200

refcnt       : 3

type         : blkcipher

blocksize    : 16

min keysize  : 16

max keysize  : 32

ivsize       : 16

[...]

name         : aes

driver       : aes-x86_64

module       : kernel

priority     : 200

refcnt       : 3

type         : cipher

blocksize    : 16

min keysize  : 16

max keysize  : 32

```

----------

## weingbz

 *flash49 wrote:*   

> 
> 
>  *weingbz wrote:*   
> 
> Hdparm was not exactly written for this usecase, is there a better way to test the speed?
> ...

 

Oh yeah, dd, could have thought about that myself. OK, let's see

 *flash49 wrote:*   

> 
> 
> My values , Athlon 2300BE 2x1900Mhz, Perc5/i raid5,lvm,luks(aes256,amd64 optimized on 64Bit kernel)  
> 
> 

 

Hmm, I only get about half of your value although I have almost the same hardware (Athlon 2400BE 2x2300Mhz). And I don't use luks.

 *flash49 wrote:*   

> 
> 
> You can check the used crypto driver in "/proc/crypto":
> 
> ```
> ...

 

Yes that seems to be the problem, I have:

```

name         : cbc(aes)

driver       : cbc(aes-i586)

module       : kernel

priority     : 200

refcnt       : 2

type         : blkcipher

blocksize    : 16

min keysize  : 16

max keysize  : 32

ivsize       : 16

...

name         : aes

driver       : aes-i586

module       : kernel

priority     : 200

refcnt       : 2

type         : cipher

blocksize    : 16

min keysize  : 16

max keysize  : 32

```

Strange, where do I activate aes-x86_64? I have Processor type AMD K8/Hammer activated 

```

CONFIG_MK8=y

```

----------

## likewhoa

 *weingbz wrote:*   

>  *flash49 wrote:*   
> 
>  *weingbz wrote:*   
> 
> Hdparm was not exactly written for this usecase, is there a better way to test the speed?
> ...

 

inside the kernel crypto section. just select aes-x86-64.

----------

## weingbz

 *likewhoa wrote:*   

> 
> 
> inside the kernel crypto section. just select aes-x86-64.
> 
> 

 

There is no such option in the config file. I have kernel linux-2.6.23-gentoo-r8 and I can only select

CONFIG_CRYPTO_AES or CONFIG_CRYPTO_AES_586.

----------

## Sadako

 *weingbz wrote:*   

>  *likewhoa wrote:*   
> 
> inside the kernel crypto section. just select aes-x86-64.
> 
>  
> ...

 That would be because you have a 32-bit userland...

Try `ARCH=x86_64 make menuconfig`, and you should see the x86_64 option in place of i586 (for all the good it'll do you...).

----------

## weingbz

 *Hopeless wrote:*   

> 
> 
> Try `ARCH=x86_64 make menuconfig`, and you should see the x86_64 option in place of i586 (for all the good it'll do you...).

 

Your're right with the "for all the good..." part  :Smile: . If I try that and start make bzImage make starts to configure the kernel from scratch (more or less). It seems to me that the way to compile the kernel to 64 bit is horribly clunky and indirect. I would have wished for a simple kernel parameter. Well no time left now, I will try again tomorrow.

----------

## Sadako

 *weingbz wrote:*   

>  *Hopeless wrote:*   
> 
> Try `ARCH=x86_64 make menuconfig`, and you should see the x86_64 option in place of i586 (for all the good it'll do you...). 
> 
> Your're right with the "for all the good..." part . If I try that and start make bzImage make starts to configure the kernel from scratch (more or less). It seems to me that the way to compile the kernel to 64 bit is horribly clunky and indirect. I would have wished for a simple kernel parameter. Well no time left now, I will try again tomorrow.

 You could take a look at kgcc64, which is specifically for compiling 64-bit kernels on 32-bit userlands.

----------

## neuron

What io scheduler do you use?  I've seen quite large performance differences between them on encrypted storage, anticipatory scheduler for me it WAY faster than CFQ is.

----------

## weingbz

 *neuron wrote:*   

> What io scheduler do you use?  I've seen quite large performance differences between them on encrypted storage, anticipatory scheduler for me it WAY faster than CFQ is.

 

When I read the documentation about the schedulers it's clear that if you copy one big file the anticipatory scheduler is faster than CFQ. It's made for that usecase. CFQ is optimized for lots of small simultaneous transfers. The question now is, which one fits my use pattern better? I tried setting anticipatory as default scheduler but I couldn't boot with the new kernel and with the old my 2nd harddisk was kicked from the raid. Now I'm rebuilding the array and then I will try again.

----------

## weingbz

OK, done with rebuilding and I set the anticipatory IO scheduler but now I'm confused. 

Copying from /dev/mapper/... is faster by about 10MB/s  :Smile: . OK that was expected.

Copying from /dev/md0 is slower by about 10 MB/s.   :Question:   Really strange. Copying one big file should have gotten faster.

Also copying from /dev/sda1 or /dev/sdb1 which together make /dev/md0 is faster than copying from /dev/md0.   :Question:   When reading from a mirroring raid I thought I would get almost the combined speed of the drives but I get less than from one drive  :Sad: 

Also even if reading directly from the drives (so without decryption) one core goes to 100%. As if I didn't have DMA enabled (which should be impossible on SATA drives).

OK, it seems I have to read a couple of guides and howtos over the weekend.

----------

## Sadako

AFAIK, when reading one large file, a software raid mirror won't be any faster than reading directly from just one disk.

Supposedly the advantages come in when reading multiple files at the same time, however I've seen even this being disputed...

----------

## kernelOfTruth

you guys might find the following interesting:

http://andi.neuriem.de/wordpress/?p=4 (windows only ???)

turbo truecrypt, wohoo !   :Cool: 

----------

## snIP3r

hi all!

this is a very interesting thread and i also do have a question about sppeding up my encrypted partition:

i have an amd64 cpu and i use a 64bit kernel. a partition is encrypted with dm_crypt. my kernel config looks like this:

```

area52 linux # less .config |grep CRYPT

CONFIG_BLK_DEV_CRYPTOLOOP=y

CONFIG_DM_CRYPT=m

CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y

CONFIG_CRYPTO_BLKCIPHER=y

CONFIG_CRYPTO_HASH=y

CONFIG_CRYPTO_MANAGER=y

CONFIG_CRYPTO_HMAC=y

CONFIG_CRYPTO_XCBC=y

CONFIG_CRYPTO_NULL=y

CONFIG_CRYPTO_MD4=y

CONFIG_CRYPTO_MD5=y

CONFIG_CRYPTO_SHA1=y

CONFIG_CRYPTO_SHA256=y

CONFIG_CRYPTO_SHA512=y

CONFIG_CRYPTO_WP512=y

CONFIG_CRYPTO_TGR192=y

CONFIG_CRYPTO_GF128MUL=y

CONFIG_CRYPTO_ECB=y

CONFIG_CRYPTO_CBC=y

CONFIG_CRYPTO_PCBC=m

CONFIG_CRYPTO_LRW=y

# CONFIG_CRYPTO_CRYPTD is not set

CONFIG_CRYPTO_DES=y

# CONFIG_CRYPTO_FCRYPT is not set

CONFIG_CRYPTO_BLOWFISH=y

CONFIG_CRYPTO_TWOFISH=y

CONFIG_CRYPTO_TWOFISH_COMMON=y

CONFIG_CRYPTO_TWOFISH_X86_64=y

CONFIG_CRYPTO_SERPENT=y

CONFIG_CRYPTO_AES=y

CONFIG_CRYPTO_AES_X86_64=m

CONFIG_CRYPTO_CAST5=y

CONFIG_CRYPTO_CAST6=y

CONFIG_CRYPTO_TEA=y

CONFIG_CRYPTO_ARC4=y

CONFIG_CRYPTO_KHAZAD=y

CONFIG_CRYPTO_ANUBIS=y

CONFIG_CRYPTO_DEFLATE=y

CONFIG_CRYPTO_MICHAEL_MIC=y

CONFIG_CRYPTO_CRC32C=y

CONFIG_CRYPTO_CAMELLIA=y

# CONFIG_CRYPTO_TEST is not set

CONFIG_CRYPTO_HW=y

```

the encrypted partition uses this cypher:

```

cipher:  aes-lrw-benbi

```

and i found out that the kernel built-in aes cypher is used. the optimized aes_x86_64 module is not used (because it is not loaded). is there a way to use this optimized module instead?

i thought about this and i would suggest the following procedure:

- set the 'CONFIG_CRYPTO_AES=y' option to '# CONFIG_CRYPTO_AES is not set' (so only the optimized version is used furthermore)

- recompile the kernel and install it

- unmount the encrypted partition

- reboot the new kernel

- re-emerge the package 'sys-fs/cryptsetup-luks'

- remount the encrypted partition

will this be ok?

so there is another question: if i deactivate the generic aes cypher in kernel, do other packages that require the aes cypher use the optimized (x86_64) version? or do i have to re-emerge all packages that used the generic aes cypher?

i hope someone can give me advise...

greets

snIP3r

----------

## Sadako

snIP3r; it should be enough to unmount and cryptsetup remove your encrypted partition(s), load (modprobe) the x86_64 optimized cipher module, then re-mount.

Otherwise, what you laid out should work, although re-emerging cryptsetup is pointless.

Edit; as a side note, it might be worth noting that sys-fs/cryptsetup-1.0.5 (and up) is cryptsetup-luks, and sys-fs/cryptsetup-luks is somewhat depreciated.

----------

## snIP3r

 *Hopeless wrote:*   

> snIP3r; it should be enough to unmount and cryptsetup remove your encrypted partition(s), load (modprobe) the x86_64 optimized cipher module, then re-mount.
> 
> Otherwise, what you laid out should work, although re-emerging cryptsetup is pointless.
> 
> Edit; as a side note, it might be worth noting that sys-fs/cryptsetup-1.0.5 (and up) is cryptsetup-luks, and sys-fs/cryptsetup-luks is somewhat depreciated.

 

hi hopeless!

thx for the tips. i'll try it asap.

greets

snIP3r

----------

## mmoufid

Has anyone done a bench test of AES vs Twofish (that is, their implementations in the Linux kernel)?

Currently, I'm using Twofish for swap and /tmp, because from what I've read the Twofish implementation, although slower in theory, is more optimized in the Linux kernel than AES, and therefore faster.

----------

## neuron

 *mmoufid wrote:*   

> Has anyone done a bench test of AES vs Twofish (that is, their implementations in the Linux kernel)?
> 
> Currently, I'm using Twofish for swap and /tmp, because from what I've read the Twofish implementation, although slower in theory, is more optimized in the Linux kernel than AES, and therefore faster.

 

in my experience (and I tested this on amd64 a while back) AES is faster, but that was before any twofish_64 implementations (and I haven't checked if they've added any).

----------

## odessit

https://forums.gentoo.org/viewtopic-p-3103988.html

Somewhat outdated, I will run latest 64b benches sometime this week. I do not have access to 32b Gentoo setup anymore on equivalent hardware platform.

----------

## mmoufid

 *odessit wrote:*   

> https://forums.gentoo.org/viewtopic-p-3103988.html
> 
> Somewhat outdated, I will run latest 64b benches sometime this week. I do not have access to 32b Gentoo setup anymore on equivalent hardware platform.

 I might try on 32bit if I have time this weekend.

Could anyone provide me a good way of doing a benchmark?

Would something like this

```
time dd if=/dev/urandom bs=1024k count=10 > /tmp/test
```

where /tmp would be mounted with a different cipher each time, be any good?

----------

## odessit

Quick and dirty

Edit encrypted/scripts/BackupPass to be some plaint text wile with long password.

xaa was  512MB file

xbb was 4096MB file

```

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo twofish --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes192 --passphrase-fd 0 --output xaa.gpg xaa 

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes256 --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg 

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo blowfish --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo cast5 --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo 3des --passphrase-fd 0 --output xaa.gpg xaa

rm xaa.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo twofish --passphrase-fd 0 --output xbb.gpg xbb

rm xbb.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes --passphrase-fd 0 --output xbb.gpg xbb

rm xbb.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes192 --passphrase-fd 0 --output xbb.gpg xbb 

rm xbb.gpg

time cat encrypted/scripts/BackupPass | gpg --batch -z 0 -c --cipher-algo aes256 --passphrase-fd 0 --output xbb.gpg xbb

rm xbb.gpg 

time openssl enc -aes-128-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -aes-192-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -aes-256-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -bf -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -des3 -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -cast5-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xaa -out xaa.enc

rm xaa.enc

time openssl enc -aes-128-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xbb -out xbb.enc

rm xbb.enc

time openssl enc -aes-192-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xbb -out xbb.enc

rm xbb.enc

time openssl enc -aes-256-cbc -k bIaClAbriewrlUP5a8rlA3LutIakOeTo -salt -in xbb -out xbb.enc

rm xbb.enc

```

----------

## Sadako

 *mmoufid wrote:*   

> Would something like this
> 
> ```
> time dd if=/dev/urandom bs=1024k count=10 > /tmp/test
> ```
> ...

 You'll need a count far higher than 10 to get any decent results (think in the 1,000's if /tmp is large enough), and you should really use /dev/zero rather than /dev/urandom.

Also, if you could use a device rather than a filesystem it would remove any filesystem overhead, for example;

```
time dd if=/dev/zero of=/dev/mapper/tmpcrypt bs=1024k count=1000
```

You might also want to lower the bs a bit (and raise the count in proportion).

It would be even better if you performed the tests on a ramdisk, as that would eliminate the hard disk IO completely...

----------

## Sadako

odessit: does openssl use the kernel's algorithms/ciphers?

----------

## odessit

Hmmm, now that you are asking..., I think openssl is using its own library. Just like GnuPG.

Kernel vs library did not come to my mind.

So the question is - what file encryption software utilizes Kernel for encryption? Something that we can script easily.

----------

## RoundsToZero

I don't know if anything userspace does (or can).  I know wireless network drivers use it for WEP/WPA/WPA2 encryption.  If you're in userspace it's better to user a userspace library to do encryption because then you don't have to do userspace/kernelspace context switches when calling into the encryption routines.

----------

## mmoufid

 *RoundsToZero wrote:*   

> I don't know if anything userspace does (or can).

 

It should be enough to do something like 

```
#include <stdio.h>

#include <string.h>

#include <sys/types.h>

#include <rmd160.h>

int main( int argc, char ** argv )

{

           RMD160_CTX rmd;

           u_int8_t output[RMD160_DIGEST_STRING_LENGTH];

           char *buf = "abc";

           printf("0x%s0, RMD160Data(buf, strlen(buf), output));

}
```

I used the OpenBSD RIPEMD example here, but in general anything in /usr/src/linux/include/crypto/*.h can be used similarly.

 *odessit wrote:*   

> So the question is - what file encryption software utilizes Kernel for encryption? Something that we can script easily.

 

I haven't been around long enough but from what I see Linux is singularly lacking in this respect. For example, alot of the Linux kernel crypto is based off code in GnuPG...

----------

## RoundsToZero

I'm not sure if it's a good idea to include kernel headers in userspace programs.

----------

