# How to cleanly mount /usr before init [solved]

## kwesadilo

Back when udev changed so that you needed to mount /usr in the initramfs, I added a line to my initramfs' init script to do it. I don't think I did it quite right. Here is the whole script:

```
#!/bin/busybox sh

# mount procfs and sysfs

mount -t proc none /proc

mount -t sysfs none /sys

# mount root filesystem and devtmpfs

mount /dev/sda3 /mnt/root

# open crypt

cryptsetup luksOpen /dev/sda4 crypt

# make logical volumes available

lvm vgchange -a y

# pre-mount /usr so udev doesn't freak out

mount /dev/vg/usr /mnt/root/usr                    <----- This is new

# clean up

umount /proc

umount /sys

# switch to real root filesystem

exec switch_root /mnt/root /sbin/init
```

Note that I have most of my partitions in LVM on top of an encrypted device.

My computer still boots, but init doesn't particularly like it. Here is a boot log:

```
rc boot logging started at Fri Feb  8 11:09:16 2013

 * Setting system clock using the hardware clock [Local Time] ...

 [ ok ]

 * Loading module vboxdrv ...

 [ ok ]

 * Loading module vboxnetflt ...

 [ ok ]

 * Loading module vboxnetadp ...

 [ ok ]

 * Loading module vboxpci ...

 [ ok ]

 * Autoloaded 4 module(s)

 * Setting up the Logical Volume Manager ...

  The link /dev/vg/usr should had been created by udev but it was not found. Falling back to direct link creation.

  The link /dev/vg/opt should had been created by udev but it was not found. Falling back to direct link creation.

  The link /dev/vg/var should had been created by udev but it was not found. Falling back to direct link creation.

  The link /dev/vg/tmp should had been created by udev but it was not found. Falling back to direct link creation.

  The link /dev/vg/home should had been created by udev but it was not found. Falling back to direct link creation.

  The link /dev/vg/swap should had been created by udev but it was not found. Falling back to direct link creation.

 [ ok ]

 * Checking local filesystems  ...

/dev/mapper/vg-home: clean, 85200/6553600 files, 18860652/26214400 blocks

/dev/mapper/vg-opt: clean, 10343/3121152 files, 9398263/12582912 blocks

/dev/mapper/vg-tmp: clean, 44/655360 files, 106158/2621440 blocks (check in 5 mounts)

/dev/mapper/vg-var: clean, 37440/458752 files, 149194/1835008 blocks

 [ ok ]

 * Remounting filesystems ...

 [ ok ]

 * Updating /etc/mtab ...

 [ ok ]

 * Activating swap devices ...

 [ ok ]

 * Mounting local filesystems ...

mount: /dev/mapper/vg-usr already mounted or /usr busy

mount: according to mtab, /dev/vg/usr is already mounted on /usr

 * Some local filesystem failed to mount

 [ !! ]

 * Configuring kernel parameters ...

 [ ok ]

 * Creating user login records ...

 [ ok ]

 * Cleaning /var/run ...

 [ ok ]

 * Wiping /tmp directory ...

 [ ok ]

 * Restoring Mixer Levels ...

 [ ok ]

 * Setting terminal encoding [UTF-8] ...

 [ ok ]

 * Setting console font [default8x16] ...

 [ ok ]

 * Setting hostname to deimos ...

 [ ok ]

 * Setting keyboard mode [UTF-8] ...

 [ ok ]

 * Loading key mappings [us] ...

 [ ok ]

 * Bringing up interface lo

 *   127.0.0.1/8 ...

 [ ok ]

 *   Adding routes

 *     127.0.0.0/8 via 127.0.0.1 ...

 [ ok ]

 * Mounting misc binary format filesystem ...

 [ ok ]

 * Loading custom binary format handlers ...

 [ ok ]

 * Activating additional swap space ...

 [ ok ]

 * setting up tmpfiles.d entries ...

 [ ok ]

 * Initializing random number generator ...

 [ ok ]

rc boot logging stopped at Fri Feb  8 16:09:22 2013
```

Note that there is a second attempt to mount /usr. The symlink stuff for the logical volumes has been there for a long time without causing a problem. Although I would be interested if someone knew what was causing it, I don't think it's related to the /usr change.

Here is a shutdown. Note again the difficulty with /usr:

```
rc shutdown logging started at Fri Feb  8 14:13:09 2013

 * Stopping local

 [ ok ]

 * Stopping vixie-cron ...

 [ ok ]

 * Saving random seed ...

 [ ok ]

 * Deactivating additional swap space ...

 [ ok ]

 * Stopping firewall ...

 [ ok ]

 * Stopping Hard Drive Active Protection System daemon ...

 [ ok ]

 * Stopping cupsd ...

 [ ok ]

 * Shutting down bluetooth ...

 [ ok ]

 * Storing ALSA Mixer Levels ...

 [ ok ]

 * Stopping kdm ...

 [ ok ]

 * Stopping gpm ...

 [ ok ]

 * Unmounting network filesystems ...

 [ ok ]

 * Stopping wicd daemon and closing connections ...

 [ ok ]

 * Bringing down interface lo

 * Stopping ConsoleKit daemon ...

 [ ok ]

 * Stopping D-BUS system messagebus ...

 [ ok ]

 * Stopping acpid ...

 [ ok ]

 * Stopping metalog ...

 [ ok ]

 * Unmounting loop devices

 * Unmounting filesystems

 *   Unmounting /media/8bdaa999-4d09-4d2a-ad88-754bc3819607 ...

 [ ok ]

 *   Unmounting /var ...

 [ ok ]

 *   Unmounting /tmp ...

 [ ok ]

 *   Unmounting /opt ...

 [ ok ]

 *   Unmounting /home ...

 [ ok ]

 * Deactivating swap devices ...

 [ ok ]

 * Shutting down the Logical Volume Manager

 *   Shutting Down LVs & VGs ...

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  device-mapper: remove ioctl on  failed: Device or resource busy

  Unable to deactivate vg-usr (253:1)

 * Failed

 [ !! ]

 * Finished Shutting down the Logical Volume Manager

 * Stopping udev ...

 [ ok ]

 * Setting hardware clock using the system clock [Local Time] ...

 [ ok ]

rc shutdown logging stopped at Fri Feb  8 14:13:30 2013
```

To round things out, here are my fstab:

```
# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>

/dev/sda3               /               ext3            noatime         0 1

/dev/mapper/vg-home     /home           ext3            noatime,user_xattr         0 2

/dev/mapper/vg-opt      /opt            ext3            noatime         0 2

/dev/mapper/vg-tmp      /tmp            ext3            noatime         0 2

/dev/mapper/vg-usr      /usr            ext3            noatime         0 0

/dev/mapper/vg-var      /var            ext3            noatime         0 2

/dev/mapper/vg-swap     none            swap            sw              0 0

#/dev/cdrom             /mnt/cdrom      auto            noauto,ro,user  0 0

proc                    /proc           proc            defaults        0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for 

# POSIX shared memory (shm_open, shm_unlink).

# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will

#  use almost no memory if not populated with files)

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0
```

and current mtab:

```
rootfs / rootfs rw 0 0

/dev/sda3 / ext3 rw,noatime,commit=0 0 0

/dev/vg/usr /usr ext3 rw,noatime,commit=0 0 0

proc /proc proc rw,relatime 0 0

tmpfs /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0

udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=496047,mode=755 0 0

devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0

shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0

cgroup_root /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0

openrc /sys/fs/cgroup/openrc cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc 0 0

cpuset /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0

cpuacct /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0

freezer /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0

/dev/mapper/vg-home /home ext3 rw,noatime,user_xattr,commit=0 0 0

/dev/mapper/vg-opt /opt ext3 rw,noatime,commit=0 0 0

/dev/mapper/vg-tmp /tmp ext3 rw,noatime,commit=0 0 0

/dev/mapper/vg-var /var ext3 rw,noatime,commit=0 0 0

binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0

/dev/mapper/udisks-luks-uuid-78da1846-296d-49e6-9f60-413221c432fe-uid1000 /media/8bdaa999-4d09-4d2a-ad88-754bc3819607 ext4 rw,nosuid,nodev,uhelper=udisks 0 0
```

Does anybody know how I can make init play nicely with the initramfs? I'd like to get rid of those ugly error messages and their corresponding delays. It would be especially nice if I could periodically fsck the filesystems mounted before boot, but let's take things one step at a time.

I thank you in advance for your assistance.Last edited by kwesadilo on Wed Mar 20, 2013 3:21 am; edited 1 time in total

----------

## hydrapolic

Try to mark your /usr in fstab as noauto. 

As for now, you mount /usr from initramfs, but you also have /usr in your fstab with implicit auto flag and localmount tries to mount it.

----------

## greyspoke

I ran into similar problems, can't quite remember the scenario, but roughly speaking I think it went like this.

You have already created your logical volumes in the initramfs and this is then confusing the lvm initscript, as it finds the volumes already activated yet the device nodes not there in the way it expects.  (You might think they are there, but they weren't put there by udev and that seems to matter).  If you are activating your logical volumes in your initramfs you don't need to run the lvm init script, removing lvm from your runlevels would deal with the first batch of whinges I think.  Alternatively you could activate just the /usr volume in your initramfs and tell your lvm.conf only to deal with the other volumes, then probably the lvm init script wouldn't complain.

The later issues appear to be because you have already mounted /usr but your /etc/fstab is telling your computer to mount it which it can't do because it is already mounted.  You appear to have set fstab not to check it as there are no complaints at the "Checking locak filesystems" stage and it clearly isn't trying to check /dev/mapper/vg-usr.

If you want your /usr to be checked you will have to put the necessary tools in your initramfs and have it checked there.

----------

## The Doctor

This may not exactly solve your problem, but separate /usr is working again with udev for me. I have a similar setup. You might want to make a new initramfs to test if it works for you as well. Clearly, you should keep the old one, just in case.

----------

## kwesadilo

Thanks for answering. I figured it had something to do with fstab. I will try to fix this up and let you know how it turns out.

I had the lvm init script enabled because the elog messages made it sound like I needed it. Is it getting freaked out by the line lvm vgchange -a y? I'm pretty sure localmount or something will take care of mounting my logical volumes without the lvm init script. Right?

I was never quite clear on why I have to pre-mount /usr. Are you saying I might not have to? What determines whether I have to?

----------

## The Doctor

Well, the systemd guys broke it in the name of whatever and the Gentoo devs reversed this quietly. Bascially, there has been a push to move things from /lib to /usr/lib so of course they can't be used before /usr is mounted.

You can probably get by without premounting now. What would fail is a service that starts up and requires access to /usr before its ready. The only example that jumps to mind is alsasound when run in the boot runlevel. It really makes no sense to run it in the boot run level anyway. Default makes more sense and will not throw errors. There may well be others. My system is *kit, pam, and upower/disks free so I don't know if any of those will affect you.

The best advice I can offer is to try it. You will not hurt anything if it fails, and you will know for sure.

If that fails, you might want to try somthing like this: 

```
lvm vgscan --mknodes

lvm lvchange -aly vg/root

lvm lvchange -aly vg/usr
```

It works for me and lvm does not freak out latter. You should add some sort of fscheck too if it is going to be a long term solution.

----------

## greyspoke

 *kwesadilo wrote:*   

> Thanks for answering. I figured it had something to do with fstab. I will try to fix this up and let you know how it turns out.
> 
> I had the lvm init script enabled because the elog messages made it sound like I needed it. Is it getting freaked out by the line lvm vgchange -a y? I'm pretty sure localmount or something will take care of mounting my logical volumes without the lvm init script. Right?
> 
> I was never quite clear on why I have to pre-mount /usr. Are you saying I might not have to? What determines whether I have to?

 

Well I have been carrying on with logical volumes for /usr /tmp /var /and /home all activated in an initramfs (and /usr checked and mounted) since I got caught by the separate /usr udev thing.  It sounds like I could go back to having it done the traditional way from what the Dr. says.  But I might as well go the whole hog and merge / and /usr when I get round to faffing with the system again.

Anyhow, I also ditched the lvm init scipt and nothing untoward has happened.  The script does appear to de-activate the volumes when stopped.  But this does not appear to be necessary in order to shut down the computer nicely.  I guess it might be useful to stop the lvm service in some situations, but not mine (I'd end up with an unusable computer).  It may be different for a dynamic system where the discs and volumes are changing a lot.

I'd be interested to hear if there is anything the lvm init script does that might be important beyond the basic starting the volumes.

----------

## depontius

As a very minor, very simple change, I note that you do not "vgscan" before doing "vgchange -a y".  I have used lvm, but not very intelligently, so I've always followed the rules by rote, which include the 2-command sequence.

----------

## kwesadilo

Wow, sorry I took so long to get back to you guys. It's been over a month. So here's what happened: The week after I first posted, I started shopping for and bought a car and started a job an hour away from my house. Since then, much of my remaining free time has been spent looking for apartments closer to work. So much, in fact, that I didn't reboot for long enough to forget the passphrase to my encrypted volumes a little over a week ago. I could still mostly remember how to touch type it, so I had a go at brute forcing it. That project was concluded successfully last night, and I was finally able to reboot.

I stopped mounting /usr in the initramfs. This got rid of the /usr-related errors during init. I have rebooted several times and observed no ill side effects during init or otherwise. systemd? More like systemFUD.

I could have sworn that I had vgchange in my init script. I must have deleted it at some point, maybe when I added the /usr line. I added 

```
vgchange --mknodes
```

 and that got rid of my LVM-related warnings during init. The LVM warnings predate the /usr change, so I suspect it was the --mknodes that cleared them up. I'm still running the LVM init script, which I assume is responsible for activating my LVs, since I no longer do that for any of them.

I notice that /usr is not getting fsck'd during init, but I'm pretty sure this is because I had turned that off in fstab, probably because it failed after I started pre-mounting /usr. I'll let you know what happens when I change it back. Now that I think about it, the root directory has always been mounted through the initramfs and may never have been fsck'd since I created it. I should probably revisit the idea of fscking in initramfs.

Edit: /usr was fsck'd after I rebooted with my new fstab. I think that addresses everything that I originally posted about, so I'm marking this thread solved. Thanks for all of your help and (I hope) patience.

For posterity's sake, here are my current init script and fstab:

```
#!/bin/busybox sh

# mount procfs and sysfs

mount -t proc none /proc

mount -t sysfs none /sys

# mount root filesystem and devtmpfs

mount /dev/sda3 /mnt/root

# open crypt

cryptsetup luksOpen /dev/sda4 crypt

# make logical volumes available

lvm vgscan --mknodes

#lvm vgchange -a y vg/usr

# clean up

umount /proc

umount /sys

# switch to real root filesystem

exec switch_root /mnt/root /sbin/init
```

```
# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>

/dev/sda3               /               ext3            noatime         0 1

/dev/mapper/vg-home     /home           ext3            noatime,user_xattr         0 2

/dev/mapper/vg-opt      /opt            ext3            noatime         0 2

/dev/mapper/vg-tmp      /tmp            ext3            noatime         0 2

/dev/mapper/vg-usr      /usr            ext3            noatime         0 2

/dev/mapper/vg-var      /var            ext3            noatime         0 2

/dev/mapper/vg-swap     none            swap            sw              0 0

#/dev/cdrom             /mnt/cdrom      auto            noauto,ro,user  0 0

proc                    /proc           proc            defaults        0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for 

# POSIX shared memory (shm_open, shm_unlink).

# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will

#  use almost no memory if not populated with files)

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0
```

----------

