# initramfs & busybox won't mount cryptsetup created mapping

## Aries97

I'm using these guides: Booting_encrypted_system_from_USB_stick & Initramfs to encrypt my root partition.

    The system is an Atom motherboard, I'm booting from a usb stick, and my root partition is on an sd card.  Everything seems to be working fine, grub loads, kernel & initramfs load, the init script starts. When cryptsetup runs it produces some error messages, but reports success.  However the mount command that follows it fails. 

The problem, I think, is that cryptsetup is not working correctly in this environment.

```
/ # cryptsetup luksOpen /dev/sde1 keyring

Enter LUKS passphrase:

sh: /sbin/udevadm: not found

sh: /sbin/udevadm: not found

key slot 1 unlocked.

sh: /sbin/udevadm: not found

Command successful.

/ # mount -t ext2 /dev/mapper/keyring /mnt/keyring

mount: mounting /dev/mapper/keyring on /mnt/keying failed: No such device

/ # 
```

When I cryptsetup luksOpen this device on any other machine there is no problem finding the ext2 partition on it.  If I don't specify a file system type mount seems to guess that it's a UDF file system and, of course, fails to mount it.  I am able to mount a file system that isn't encrypted, so my guess is there is a problem with cryptsetup, or the busybox shell that it is running in. - I've exhausted my skills, I need some help.

  Thank you!

Two possibly relevant side problems: Nano won't run in busybox:

```
/ # nano

/bin/sh: nano: Too many levels of symbolic links

/ #
```

Also the telnet daemon started by the remote rescue shell allows me to login but it tries to start bash which fails; how do I tell it to use the busybox shell?

My Init:

```
#!/bin/busybox sh

#Rescue Shell

rescue_shell() {

    echo "Something went wrong. Dropping you to a shell."

    busybox --install -s

    exec /bin/sh

}

#Remote Rescue Shell

remote_rescue_shell() {

    # Bring up network interface

    ifconfig eth0 192.168.1.99 up

    route add default gw 192.168.1.1

    # telnetd requires devpts

    mkdir -p /dev/pts

    mount -t devpts none /dev/pts

    # Start the telnet server

    telnetd

    # Continue with the local rescue shell

    rescue_shell

}

# Mount the /proc and /sys filesystems.

mount -t proc none /proc || remote_rescue_shell

echo "Mounted proc."

mount -t sysfs none /sys || remote_rescue_shell

echo "Mounted sys."

mount -t devtmpfs none /dev || remote_rescue_shell

echo "Mounting dev, sleeping for 20 seconds."

sleep 20

#Memorex (boot) = findfs UUID=43df7482-ba29-4593-8798-d00b88b69ed1

#Keyring = findfs UUID=5e27bd30-bec0-47ab-bd49-9085be33ea5d

#Root = findfs UUID=fccf5c8f-9801-4931-ab22-678b7f50af1b

# Do your stuff here.

echo "Opening encrypted keyring."

cryptsetup -T 2 luksOpen $(findfs UUID=5e27bd30-bec0-47ab-bd49-9085be33ea5d) keyring || remote_rescue_shell

echo "Opened keyring."

mount -t ext2 /dev/mapper/keyring /mnt/keyring || remote_rescue_shell

echo "Mounted keyring."

echo "Opening encrypted root."

cryptsetup -d /mnt/keyring/root.key luksOpen $(findfs UUID=fccf5c8f-9801-4931-ab22-678b7f50af1b) root || remote_rescue_shell

echo "Opened root, scanning logical volume groups."

lvm vgscan --mknodes || remote_rescue_shell

lvm lvchange -aly vg0/root || remote_rescue_shell

mount -t reiserfs /dev/vg0/root /mnt/root || remote_rescue_shell

echo "Mounted root."

# Clean up.

umount /proc

umount /sys

umount /dev

umount /mnt/usb/keyring

cryptsetup luksClose /dev/mapper/keyring

# Boot the real thing.

exec switch_root /mnt/root /sbin/init

```

[/url]

----------

## cach0rr0

I had the same problem. This is ultimately what moved me to Arch for my systems where rootvol encryption is a must. 

I don't know if this is of any help, but i extracted the contents of the gzipped CPIO they use which has all the 'crypt hooks' and whatnot that they use to get everything set up. 

If you care to have a look - http://whitehathouston.com/initramfs/

(it's 17MB uncompressed, so compressing was a must - in case anyone wonders why I redid this)

The device nodes are not in that tgz, so check the pastebin link below. I have /home mounted nodev, so...yeah. 

```

$ cpio -i -d -H newc --no-absolute-filenames < kernel26.img 

cpio: Removing leading `/' from member names

cpio: dev/null: Cannot mknod: Operation not permitted

cpio: dev/zero: Cannot mknod: Operation not permitted

cpio: dev/console: Cannot mknod: Operation not permitted

cpio: dev/mem: Cannot mknod: Operation not permitted

```

Maybe you can "reverse engineer" something from that? I am going to whenever I finally have the time, and when I do so move this box back to Gentoo - but today is not the day I have the time!

If you'd like the exact initramfs I use, I am happy to upload that also. EDIT: uploaded it. It is the kernel26.img file. Hell, try it out, it should realistically work with any recent 2.6 kernel

NB: this is the contents of that .tgz - http://paste.pocoo.org/show/227852/

If I had the know-how, I would try and port their mkinitcpio to gentoo (doing this with genkernel --menuconfig was epic fail for me)

----------

## Hu

Since the mount fails with No such device, I would start investigating in the kernel.  Can the failing kernel mount generic ext2 volumes while in initramfs mode?  Can it read the underlying block device?  I would expect that it can read the device, else the LUKS open should have failed.  Are there any interesting lines in dmesg after the mount fails?

 *cach0rr0 wrote:*   

> it's 17MB uncompressed

 Wow.  My initramfs source files take less than 40K.  :Very Happy:   That is using the kernel's built-in support for building an initramfs, so that 40K figure does not count the binaries pulled from the live system at build time.  Once you factor that in, it rises to 7M.

 *cach0rr0 wrote:*   

> The device nodes are not in that tgz, so check the pastebin link below. I have /home mounted nodev, so...yeah.

 The mount option is not the reason for the failure to mknod.  Rather, the lack of permission prevented the mknod.  Unprivileged users cannot be trusted to create arbitrary block devices.

 *cach0rr0 wrote:*   

> If I had the know-how, I would try and port their mkinitcpio to gentoo (doing this with genkernel --menuconfig was epic fail for me)

 Making the cpio is trivial with the kernel's support for building an inline initramfs.  Writing the /init that the initramfs should run to make the system functional is not nearly so easy, though it can be done with a bit of patience and knowledge.

----------

## cach0rr0

 *Hu wrote:*   

> 
> 
> Wow.  My initramfs source files take less than 40K.   That is using the kernel's built-in support for building an initramfs, so that 40K figure does not count the binaries pulled from the live system at build time.  Once you factor that in, it rises to 7M.

 

yeah 7MB of that is under /lib, 4.7MB of which is /lib/modules

500kB of /bin 350kB of /sbin

no idea where the rest of it comes from, haven't looked. 

 *Hu wrote:*   

> 
> 
> The mount option is not the reason for the failure to mknod.  Rather, the lack of permission prevented the mknod.  Unprivileged users cannot be trusted to create arbitrary block devices.
> 
> 

 

cool, noted. 

 *Hu wrote:*   

> 
> 
> Making the cpio is trivial with the kernel's support for building an inline initramfs.  Writing the /init that the initramfs should run to make the system functional is not nearly so easy, though it can be done with a bit of patience and knowledge.

 

yeah I unfortunately lack the knowledge - and while I'm patient enough, I couldn't afford the downtime. I have loads of reading to do before I'm ready to attempt this again. I'm already finding myself a touch crippled on Arch - nice enough distro, no huge complaints, but not quite as flexible as I'd like. The mkinitcpio automation (as part of their installer!) is the only reason I'm on it. Of course, it's running a load of Gentoo guests, my laptops are both Gentoo, the 24 servers at work that run our product are Gentoo...just to make it clear, I haven't left the distro  :Smile:  I just need to figure out how to "port" Arch's way of doing the initramfs stuff to Gentoo. 

Requires some learning, because as of right now I understand maybe half of what pieces together with what. I would have gutted through the rest if I had the time, but it would have taken more downtime than I can afford to do the usual forum routine. 

Once work calms down again and I have some time to breathe, I will be backing everything up (or well, copying over /home, copying over my kvm guest HDD images), and having another go with Gentoo on this box. I do not feel secure on a system where, through my own lack of knowledge and not the distro's fault, I'm a bit lost. So the move must be made at some point. The sooner the better.

EDIT: actually. why don't I hop on my laptop and experiment? I could build a kernel for this box, and try building an initramfs for this box. I boot from an external thumb drive, this is all easy. Why did I not think of this before?

----------

## cach0rr0

actually, doing some thinking out loud that hopefully helps both the original poster and myself

i plucked out proc dev and sys, since that's superfluous for our purposes here

this is all that's involved

http://paste.pocoo.org/raw/228301/

Right, so most of those lines are kernel modules. These are unnecessary if you already have these items configured as built-ins within your kernel, correct? 

Without those kernel modules, this is what it looks like:

http://paste.pocoo.org/raw/228302/

So they don't build things like cryptsetup statically. If we were to build those as a static binary, it looks like so:

http://paste.pocoo.org/raw/228307/

That is of course fudged a bit. I am assuming everything in ./bin can be built as a static binary. 

If so, I could realistically use that as-is without further understanding what's going on. 

Going one further...if I were to move everything from ./hooks into 'init' itself, as well move 'init_functions' content into 'init' itself, I could conceivably have a "one size fits all" init. Or well, a "one size fits most". I suppose this is not technically required, but makes it less confusing. 

hmmm...now to find out what all can be built static, and how difficult it will be to modify 'init' so that it doesnt look for the hooks and init_functions stuff externally.

ANOTHER EDIT: so at least blkid cannot be done static directly through portage. It is part of util-linux and util-linux doesn't seem to have a 'static' USE - I don't know of another way to build static without doing so by hand. I'm betting some others can't either, but that is not a show-stopper.

----------

## cach0rr0

EDIT: I do not think this is functional yet, noticed an issue with the invocation of poll_device in the ./hooks/encrypt - but I'm leaving it up in case someone else wants to take this and run with it. I think it only affects people who want to store their key on an external device, and that passphrase should not be affected. But from what I can tell doing a key on a separate device requires all of that udev nonsense

EDIT 2: ok, so this was my attempt at nuking references to poll_device. Far as I can tell key file may still work, but would do so without aide of udev(???). http://pastebin.com/cQ3SbvMD - hopefully original poster has something more conclusive/useful than I do, I am still BSing my way through this

ok, so I've pared this down as much as I'm comfortable with, and I *think* this should work. 

It is completely untested, but it wont kill your cat nor wipe out your data. If it fails, it, well, just fails. 

http://omploader.org/vNHBuOQ

Turns out blkid was only used to determine FS type. So that binary can be omitted. 

This assumes things are built-in rather than modules (e.g. dm-mod dm-crypt, all fs drivers, all HDD controller drivers, and all of the crypto modules e.g. AES, Serpent, that sort of thing)

I did leave this line in ./hooks/encrypt in case someone has built dm-crypt as a module for some reason:

```

/sbin/modprobe -a -q dm-crypt >/dev/null 2>&1

```

but from what I can tell init will not fail if this fails, so people with dm-crypt as a built-in should be fine. 

If you've already followed the other guides and already have the binaries in place where they should be, this should work. I think. 

The kernel command-line used with this is as such (pulled from /proc/cmdline, I can snag grub.conf if needed):

```

root=/dev/mapper/root cryptdevice=/dev/sda2:root ro

```

I *think* I can prune it down a bit more, but we're reaching the limits of my comfort. 

For anyone uncomfortable downloading an archive from the web, These are the files in question (not including binaries), all of which are needed:

./init

./init_functions (the #include type deal where this is used is on line #5 of 'init')

./hooks/encrypt (this file is read starting on line #48 of 'init', and its 'run_hook' function is fired off on line #57 )

The directory structure is as follows

```

# find .

.

./init

./hooks

./hooks/encrypt

./init_functions

```

There was some stuff I pruned out of init_functions that used blkid, and I did not think this was needed. 

If for some reason it turns out it *is* needed, have a look here - http://pastebin.com/frNw42ZP

Hopefully that's helpful and I'm not just wasting people's reading time? 

again, all of this is untested. YMMV, changes/recommendations/commentary welcome.

This is all the result of semi-educated guesses as well. I do not expect this to be perfect. If I find something glaringly wrong or broken I will fix it, but it's midnight and I've not eaten dinner, so...I need to get on that.

Fingers crossed!

----------

## Aries97

I might have found the problem! (Big shout out to the helpful people on irc freenode #gentoo, I was loosing hope and they gave me ideas and helped me carry on.) I should be able to test it tomorrow.  Hopefully it works so I can explain, and mark this solved.

----------

## cach0rr0

 *Aries97 wrote:*   

> I might have found the problem! (Big shout out to the helpful people on irc freenode #gentoo, I was loosing hope and they gave me ideas and helped me carry on.) I should be able to test it tomorrow.  Hopefully it works so I can explain, and mark this solved.

 

you will have solved an ancient hurdle of mine if so. shit, anything you can post I'll put into the brain and remember for when I take this on again.

I'm going to keep plugging away at this until I have something minimalistic as possible, then try booting with it. But shortcuts? Yes, I love shortcuts. This is one of those where I don't care *how* it works, I just care that it works (which, is why the temporary fling with Arch)

ah wait, crap. I'm idling on #gentoo. My scrollback doesn't go far enough, dammit!  :Laughing: 

----------

## Hu

 *cach0rr0 wrote:*   

> yeah 7MB of that is under /lib, 4.7MB of which is /lib/modules

 My initramfs are always tailored to the specific system, so they carry no /lib/modules.  On the other hand, I was lazy and decided to put a full glibc in the initramfs so that I could ignore static linking concerns.

 *cach0rr0 wrote:*   

> yeah I unfortunately lack the knowledge - and while I'm patient enough, I couldn't afford the downtime.

 So you have plenty of Gentoo guests, running under KVM, which means you can run custom Linux kernels under KVM...  Why not test your initramfs in an otherwise throw-away Gentoo guest?  If you make a mistake, exit KVM and revert to snapshot.  When it works, make the minor tweaks to switch between virtual hardware and the real hardware.  All the logic that it takes to get encrypted volumes working should be mostly independent of the hardware.

 *cach0rr0 wrote:*   

> EDIT: actually. why don't I hop on my laptop and experiment? I could build a kernel for this box, and try building an initramfs for this box. I boot from an external thumb drive, this is all easy. Why did I not think of this before?

 This works too, but KVM is even more convenient.  I often do a quick test in KVM before trying the initramfs on real hardware, to spare the hassle of booting back into a working kernel just so I can fix a mistake and recopy the corrected initramfs.  Combine this with the KVM options -kernel and -initrd and you can swap in test initramfs without even booting the guest to a good kernel first.

----------

## cach0rr0

would love to, dont have the memory on this box. 

I just discovered -balloon which is neat, though on .32-hardened it occasionally causes panic. But yeah, only 8GB memory on this box, 2GB for web+mysql, 2GB postfix+cyrus+amavis+clamd, 1GB inspircd+anope. 

I have *barely* enough for production guests. In the very very near future there will be new hardware, inclusive of more memory. Oh, and I will not make the mistake of doing qcow2 again (even with virtio is dog slow). Real partitions methinks. 

ANYWAY, more rambling from me, I'm awful about that. I will have the hardware to run guest VM's soon enough . 

In retrospect there are some things I did glaringly wrong, or well, less than optimal this go-round, but they've been up long enough my beloved uptime must be preserved. 

Learned a lot though. Hell I've learned heaps just over the course of this thread. rambling again. time to hush.

----------

## Hu

For testing an initramfs, a very tiny guest is adequate.  Spare 128MB of memory and it will probably work.  You can probably get that just by closing Firefox for the duration.  :Wink: 

I found qcow2 to be quite usable, though I suppose that varies depending on your read/write workload.  Most of my guests are CPU bound, not disk bound.  Real partitions are definitely faster than qcow2, but qcow2 is usable for at least some types of realistic workloads.

----------

## cach0rr0

 *Hu wrote:*   

> 
> 
> I found qcow2 to be quite usable, though I suppose that varies depending on your read/write workload.  Most of my guests are CPU bound, not disk bound.  Real partitions are definitely faster than qcow2, but qcow2 is usable for at least some types of realistic workloads.

 

I noticed a considerable speedup with preallocate metadata or whatever the opt was. Didn't see one that allowed full preallocation of all space, which would have been fine for me to take the hit as a one-off when creating the image. 

I think it's just the dynamic allocation of space coupled with crypto that makes it crawl. That, plus, they're on a btrfs volume, been less than impressed. 

I reckon you're right. Long as I didn't have to build much of anything (plus, I could distcc)  128M shouldn't be a problem.

----------

## Hu

You might also benefit from tuning cluster_size.  See man qemu-img.

 *cach0rr0 wrote:*   

> I reckon you're right. Long as I didn't have to build much of anything (plus, I could distcc)  128M shouldn't be a problem.

 I was thinking of building a system that consisted of nothing but the kernel, initramfs, and a virtual disk with the cryptographic volume and an empty filesystem inside it.  Such a system would not be useful for any real work, but would be adequate to test that the initramfs can unlock and mount the volume.  By using the -kernel and -initrd options to kvm, you can avoid needing a bootloader in the guest, and avoid needing it to be functional enough to let you install new kernels.

----------

## cach0rr0

 *Hu wrote:*   

> You might also benefit from tuning cluster_size.  See man qemu-img.

 

I knew I was forgetting something - yes, that's the one. I did use that, set cluster_size=2M

 *Hu wrote:*   

> I was thinking of building a system that consisted of nothing but the kernel, initramfs, and a virtual disk with the cryptographic volume and an empty filesystem inside it.  Such a system would not be useful for any real work, but would be adequate to test that the initramfs can unlock and mount the volume.  By using the -kernel and -initrd options to kvm, you can avoid needing a bootloader in the guest, and avoid needing it to be functional enough to let you install new kernels.

 

Never done it that way, but I'll do the requisite reading and have a crack at it. 

I'd like to get something that's usable for me and other folks, so it'll take me a bit of fiddling, reading, testing, and so forth. From whave I've read/learned thus far, a "one-size-fits-most" init portion of it shouldn't  be that difficult. 

Thanks for the info, time to play.

----------

## udeved

 *cach0rr0 wrote:*   

> 
> 
> If I had the know-how, I would try and port their mkinitcpio to gentoo (doing this with genkernel --menuconfig was epic fail for me)

 

mkinitcpio-0.8.8 arch port

----------

## cach0rr0

 *udeved wrote:*   

> 
> 
> mkinitcpio-0.8.8 arch port

 

ha...neat. I started to port, but quickly lost interest, ended up just crafting my own initramfs. Glad somebody took it on, because it really is a slick little tool.

----------

