# Many problems with a dmraid, LUKS, and LVM system with boot

## KintaroBC

I am trying to make a system which uses a kernel made by genkernel to boot and do everything it needs. I have had many issues and I can't get it to work as expected. The guides and advice out there, including the messages at the end of genkernel are only relevant to a system which uses partitions for root and swap on the LUKS drive. I however am using an LVM on top of the encrypted LUKS partition. I must use an LVM that is encrypted because the system's purpose is being a virsh KVM/QEMU cloud with guests using logical volumes.

I will explain my configuration and current problem, but I will update this thread until I have worked through all issues and have it working.

I have done the kernel and initrd with the following:

```

genkernel --luks --dmraid --lvm --disklabel --mdadm all

```

On my setup there are for disks /dev/sd[a-d] all formatted and configured with RAID, and are assembled as a RAID 5 volume. This volume appears to the admincd as /dev/md127 but shows as a long and ugly /dev/md/localhost.localdomain\:0 from the recovery shell. I could from there do cryptsetup and unlock the LUKS drives, but then I had trouble, the pvscan command was not fine and seems missing from the initrd.

I have updated the kernel boot parameters /etc/grub/default with this, in the past I used the md device I had on the admincd of /dev/md127 and have changed that to what I saw in the recovery shell trying to boot.

```

GRUB_CMDLINE_LINUX="dodmraid cryptdevice=/dev/md/localhost.localdomain\:0 dolvm root=/dev/mapper/master--hv-root"

```

I am not sure what is wrong with this setup that the commands for LVM are not in the initrd despite giving the --lvm argument to genkernel. I have everything required installed...

```

# emerge -pqv lvm2

[ebuild   R   ] sys-fs/lvm2-2.02.184-r5  USE="readline (selinux) thin udev -device-mapper-only -lvm2create_initrd -sanlock -static -static-libs (-systemd)"                                                                           

```

```

# emerge -pqv mdadm

[ebuild   R   ] sys-fs/mdadm-4.1  USE="-static" 

```

```

# emerge -pqv cryptsetup

[ebuild   R   ] sys-fs/cryptsetup-2.2.2  USE="argon2 luks1_default nls openssl udev -gcrypt -kernel -libressl -nettle -pwquality -reencrypt -static -static-libs -urandom"                                                            

 * IMPORTANT: 7 news items need reading for repository 'gentoo'.

 * Use eselect news read to view new items.

```

Edited: I realised that I forgot to enable the static use flag for the packages above, but I have now done this and rebuilt these packages and their static dependencies.

At this point my issue is that for some reason genkernel has not installed the tools for LVM, and then I may still need help with other issues. Though it would be nice if I could at least use the shell I can drop to if there is no root filesystem to get the system running.

After someone can help me get it so I can do an LVM pvscan and vgscan after doing cryptsetup, I then also want it to unlock the LUKS encrypted raid with the name /etc/secured in mapper. I cannot find much of a definitive reference to kernel arguments passed by grub and how they work with the initramfs. At this point I am just tinkering and seeing what happens when I reboot.

I got this working and also discovered that from the initramfs that I must use 'lvm' in front of commands like 'pvscan' - I still need support on what to do to get the kernel parameters to boot automatically, but at least I can use the system now.

I need to get the initrd to do the following:

* Assemble the raid which I can do from the rescue shell with: mdadm --assemble --scan

* Decrypt the RAID and name the LUKS device 'secured' which I can do from rescue with: cryptsetup luksOpen /dev/md/localhost.localdomain\:0 secured

* Get the LVM running automatically, which unlike the admincd I installed with requires a series of commands, that all from the rescue shell must start with lvm and are...

```

lvm pvscan

lvm vgscan

lvm vgchange -a y master

```

After all this one can exit the rescue shell, and specify the logical volume with the root filesystem: /dev/master/hv-root (and hv stands for hypervisor).

Finally and for now I can get my system to boot, but I would still like assistance in getting the system to do it automatically.

----------

## NeddySeagoon

KintaroBC,

Are you confusing dmraid with mdraid?

dmraid is for BIOS created fakeraid. It looks like hardware raid in use but under the skin, is software in the BIOS.

The only excuse for using fakeraid is that Windows and Linux must both access the raw raid.

If you don't have real hardware raid and Windows is not involved, use mdraid (that's linux kernel raid) or since you will use LVM anyway, LVM can do its own raid without any help from other sources.

Why do you need LUKS? 

Its only useful to protect the drive content when the LUKS volume is not unlocked. That restricts its use to portable devices.

I run several virsh KVM/QEMU systems but I don't see a use case for LUKS as none of them are portable. 

A trap for the unwary is that Hosts and Guests may not share the same PV. I didn't find that out until my first bare metal install was complete.

Here's how I did it. There is no LUKS there.

Hmm, that's from 2011. Don't follow it line by line. Extract the intent, and add LUKS.

Also, building things statically for the initrd is no longer a hard requirement.

Point lddtree at the dynamic binary and include all the bits.

----------

## Goverp

 *NeddySeagoon wrote:*   

> ...
> 
> Why do you need LUKS? 
> 
> Its only useful to protect the drive content when the LUKS volume is not unlocked. That restricts its use to portable devices.
> ...

 

<Aside>Neddy, IIUC you're saying you only need to encrypt portable devices.  Is there because non-portable devices are harder to steal (in which case there's still a use case for LUKS), or because there's a better solution for non-portable devices (other than big locks)?</Aside>

----------

## NeddySeagoon

Goverp,

If someone gains unauthorised access to my PC, I have much bigger problems than what they might find there.

If they break into a data centre and access my server, which runs 24/7 then any LUKS container will be unlocked anyway.

Anyone who targets you for data theft, will either do a USB reboot and image your RAM, so that they can recover its contents or if they have more time, pull the live RAM and image that.

DRAM can retain data for several minutes after power off.

There are several papers on both attacks and plain crypto key recovery has been demonstrated using both methods.

To my knowledge, nobody flushes RAM on power off yet, so it takes the DRAM data decay time for encryption to become effective.

Security is always about evaluating your threats then deploying countermeasures against the perceived threats.

----------

## Hu

I agree that encryption is far more interesting on devices that may be stolen, but I see a use case for encrypted drives on desktops: peace of mind for RMA.  If I have a drive fail under warranty, and I want to exercise that warranty, I'm usually required to send the failed drive back to the manufacturer as a condition for obtaining the new drive.  Some manufacturers will let you receive the new drive and then mail back the old one, but I cannot recall dealing with one that would let me just keep the failed drive for free.  Depending on how exactly the drive failed, I may not be able to wipe it as fully as I would want before sending it back.  If the drive is encrypted via LUKS, and I can get the drive functional long enough to clobber the LUKS header (or I'm confident enough in the LUKS setup + my chosen key that I don't feel the need to wipe the LUKS header), then I can RMA the drive and not worry about someone exploring it.  The other option, of course, is to accept that you can never RMA a drive that has had sensitive information (financial records, tax documents, etc.) on it, even if the warranty is clearly still valid.

----------

## Jaglover

How much can you recover from a single drive from RAID-5?

----------

## NeddySeagoon

Jaglover,

Not a lot. 

I'm aware of a classified raid set that had its off site backups performed by cloning the raid onto a new set of drives then sending drives individually by road by to as many destinations an there were drives in the raid set.

No two drives were in transit at the same time.

IT security, who are more paranoid that most, must have been happy with the risk of losing a single drive to someone who really wanted it.

You would get the partition table, unless its fake raid, then you might not.

----------

## Jaglover

Yes, that's what I thought. I remember reading somewhere everyone who encrypts their drive will regret it at some point.   :Razz: 

----------

## rufnut

I have been looking at super-encryption (just crypto on crypto) for some of the above reasons. 

I am sorry I cannot bring myself to trust wallets and third party add-ons.

For an example my new phone has "Android 10" of which they claimed was so secure companies like "Cellebrite" can not get in, yet.

However I noticed browsing the pictures I had taken with the phone and they were auto-tagged with appropriate labels!!

( I later learnt to switch this off in "app control" but the default I am sure was on!!)

The problem I visualize with multiple crypto layers is the mt-bf of drives with just "1" bit out could see a massive data block error with little or nothing recovered. (yes, test and scan those backups.)

 :Smile: Last edited by rufnut on Mon Feb 10, 2020 9:16 am; edited 1 time in total

----------

## rufnut

KintaroBC,

Sorry to hijack your thread.   :Sad: 

don't forget genkernel has "/etc/genkernel.conf" for configuration.

 *Quote:*   

> # Add in LVM support from static binaries if they exist on the system, or
> 
> # compile static LVM binaries if static ones do not exist.
> 
> LVM="yes"
> ...

 

https://forums.gentoo.org/viewtopic-t-1108168-highlight-.html

might also help even though it is for dracut.

https://wiki.gentoo.org/wiki/Genkernel#Genkernel.3F_Genkernel-next.3F_Dracut.3F

keep this in mind as I seem to run an older version of genkernel-next.

----------

## NeddySeagoon

rufnut,

For lots of reasons, the probability of single bit errors is "infinitely improbable".

For a long time now magnetic HDD have guessed what was written.

If the drive can't recover the data, you loose a whole block. That's 4kB on magnetic media, 32kB on optical media and on SSD, it may be a whole erase block, depending on the failure mode.

Some research suggest that crypto on crypto is actually weaker than either single crypto but the theory made my head hurt.

I don't trust third party add-ons either.

----------

## rufnut

NeddySeagoon,

Just trying to work it out to stop catastrophic failures. ( I am new to this.)

When I say crypto on crypto I should say LUKS on LUKS and different types at that, as say someone accidentally gains access to my unlocked LUKS of which I am currently on, there is another stage for more secure information they must also unlock.

i.e. I only unlock that when needed. Sort of stateful_LUKS I imagine wallets/vaults are similar.

I guess that's why I bought the android 10 thing up, is the AI on camera app that good or is the app browsing my photo's to see what they are. Perhaps I should ask Google?

 :Smile: 

----------

## hkmaly

 *rufnut wrote:*   

> 
> 
> Just trying to work it out to stop catastrophic failures. ( I am new to this.)
> 
> When I say crypto on crypto I should say LUKS on LUKS and different types at that, as say someone accidentally gains access to my unlocked LUKS of which I am currently on, there is another stage for more secure information they must also unlock.
> ...

 

Just because you need two different LUKS doesn't mean you must have one on the other. I think it would be just as safe, if not safer, to have two separate partitions, LUKS on both of them, and only mount each when needed.

 *NeddySeagoon wrote:*   

> 
> 
> use mdraid (that's linux kernel raid) or since you will use LVM anyway, LVM can do its own raid without any help from other sources. 
> 
> 

 

Does it? To be more specific, can LVM do raid with spreading reads fairly between source devices? Because last I looked, LVM's "mirror" was only reading from single drive. Granted, it's few years back so I'm seriously asking if that changed.

(I am currently running LUKS on mdraid for this reason.)

----------

## NeddySeagoon

hkmaly,

I don't know the detail ot LVM raid. When I set this system up about 11 years ago, I didn't know about it.

My only raid1 is /boot, everything else is LVM on top of mdadm raid5.

Even then, boot is raid1 to make like easy for me. My bootloader is grub-static which is not raid aware anyway.

----------

