# Separate /usr on Linux requires initramfs... I don't get it

## peterius

I apologize in advance, I'm exhausted.

I don't understand if this will affect me, or why its even an issue.  I read both of the blog entries in the news.

I don't use kernel modules but those are usually in /lib/ as I recall. And if a separate /usr is a problem, why isn't it just mounted earlier?!?

And the other stuff on www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ just seems idiotic.Last edited by peterius on Fri Oct 25, 2013 3:26 pm; edited 1 time in total

----------

## ryao

 *peterius wrote:*   

> I apologize in advance, I'm exhausted.
> 
> I don't understand if this will affect me, or why its even an issue.  I read both of the blog entries in the news.
> 
> I don't use kernel modules but those are usually in /lib/ as I recall.  And if a separate /usr is a problem, why isn't it just mounted earlier?!?
> ...

 

There are people using a separate /usr mount on Gentoo without an initramfs, but there are some that want to see that capability go away. As far as I know, existing systems are still working.

With that said, the only reason to have separate mount points for / and /usr is personal preference. I don't do it on any of my systems.

----------

## peterius

I always use separate /usr mount points, its personal preference but there's definitely some reasons for it.  There's no reason for the capability to go away.  And I don't see why the initramfs would be necessary?

The news note said that after Nov 1, not using an initramfs "upgrading packages will make your system unbootable."

----------

## ddriver

There's a long and emotive thread about this, but I agree, I don't understand either. Apparently this is not because of systemd or udev, so what is it about? What will break after November? What packages do I need to avoid if I want to keep /usr on a separate partition without an initrd?

I have a number of Gentoo systems of various types. Some have software RAID and most have LVM. On some small systems (netbooks and SBCs) I have everything in one partition, in others I have a tiny root partition (100-200MB) and everything else on LVM. I like the flexibility of being able to grow my /usr if needed without major upheaval. I am happy to use an initrd for special cases, like my Alix SBCs with read-only filesystems and aufs overlays, but I don't see why I should need one for something as mundane as a separate /usr. I am still hoping that someone will manage to make this go away.

Rebuilding several of my systems without separate /usr is not an option. It would be too much work and would lose me some flexibility. If I have to end up with initrd I will do it grudgingly, but my instinct tells me that this will be something home-brewed, not from some dubious package called dracut. It will take some time before I am happy with the solution. In the mean time I will try to avoid packages which break my system.

----------

## szczerb

I don't see any sense in it either, but that's not the point.

The point is that besides dracut (from Fedora?) we also have genkernel. And I use to just to create "init-thingies" for my root-and-seperate-usr-on-lvm systems with eudev. Works like a charm, although I'd rather not have to use it.

----------

## ulenrich

 *ryao wrote:*   

> There are people using a separate /usr mount on Gentoo without an initramfs, but there are some that want to see that capability go away.

 

Want? No, but this capability just doesn't matter.

 *ryao wrote:*   

> With that said, the only reason to have separate mount points for / and /usr is personal preference.

 

Only? No, and this personal preference just doesn't matter: Flexibility regarding the disk space of an installation is not an issue any more. 

The important use case of having a read-only seperate /usr is 

an easy and lightweight setup of virtual machines

It is the reason upstream likes to pull as much as possible into /usr.

Why should Gentoo spend any extra effort to maintain a capability, which doesn't matter, provokes bugs and works against the real use case? Weird idea from the start, isn't it?

----------

## croutch

Little info about it.

http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/

http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

----------

## Hu

I tend to agree with the OP.  As far as the technical angle is concerned, there are certain programs that run early during the boot process and must succeed for your system to be in a fully usable state.  The maintainers of some of those packages have decided they want to be able to use content which is traditionally installed in /usr, even if their tool is traditionally run before local filesystems (other than the root filesystem) are mounted.  Obviously, you cannot satisfy both constraints.  Their solution is to ignore users who have a separate /usr and assume that the content is available.  If you install a package that makes this assumption, and you do not make /usr available, the package will not start correctly.  If it is very important to the system, the system may not start at all.

With regard to "why not mount /usr sooner?" - that is exactly what the initramfs does.  Affected users must prepare an initramfs that has enough of a boot environment to do the job of the regular early boot environment (checking filesystems, loading supporting modules, mounting some filesystems), then transition to the on-disk boot environment that assumes the availability of /usr.

----------

## saellaven

In short, one group of people arrogantly presume they (and their program) should force everyone else to do things the way they want to, even if it means breaking existing functionality that exists for valid reasons, just not reasons that the people forcing changes care about. The most important people in this group are employed by the most commercial and enterprise driven Linux distribution, whom have a vested interest in forcing everyone into their sandbox, knowing they are a primary target, particularly in the enterprise arena.

The lesser distros (in terms of funding and manpower) can either decide to stick with the way things traditionally have been, by maintaining patches to keep the traditional behavior, or else can just hop on the bandwagon and just give in, particularly since said distro controls some of the most popular packages, which are being used intentionally to force this turd sandwich on everyone.

RH's motives aside, outside of ideology and arrogance, there's nothing technologically "better" in terms of the new paradigm... and in fact, there are many ways in which the new paradigm is outright inferior and more fragile, but the devs can't or don't want to do the work to differentiate their distros since it's easier to just follow upstream, particularly because their own personal use cases don't see the point in keeping traditional behavior. Trying to reason with them is pointless - they have their opinion and will simply dismiss any concerns anyone else has because it doesn't affect their use case.

So, today, it's /usr you can't mount with and if you want to use GNOME, be prepared to switch full on to systemd in a couple weeks. Tomorrow, maybe they come for /var (which has similar and arguably even more valid concerns for being split from /). If a package breaks your initramfs and your system becomes unbootable despite not telling you to recompile your initramfs because you updated some random tool, well, that's your fault too. It's only a matter of time before we see the usrmerge follow to ensure that you do things the one-true way because your startup scripts need a web server (yes, part of systemd's one tool to rule them all).

Gentoo could actually take a stand against the insanity but, well, it's easier to just go with the flow... even if it ultimately makes things more fragile, takes thousands of dev-hours to implement and gets tossed out anyway because things weren't all that well done despite being forced from upstream (remember HAL and all the fun we had there?)

----------

## The Doctor

Its fairly easy to write a busybox init by hand. It won't break and will do the job.

That aside, its easy to see why this isn't a big deal for most distros. They use an initramfs anyway. Red hat may claim to just be the messenger, but it is mostly their stuff that is breaking. Network Manager, pulsaudio, dbus, etc. Half of the programs they list as "broken" shouldn't even be started that early. For example, why on earth would you start cups before all your partitions are mounted? So in that respect it seems like systemd is at fault for starting things up in a careless way. OpenRC doesn't have this problem.

The http://freedesktop.org articles sound like they where written by a used car salesman. "Its not our fault systemd doesn't handle a separate /usr! Its <x y and z>!" leaving out that they also maintain <x y and z>, so it is their fault.

It took, what, 5 years for HAL to become unmaintainable. I wonder if systemd will beat it.

----------

## NeddySeagoon

 *The Doctor wrote:*   

> It took, what, 5 years for HAL to become unmaintainable. I wonder if systemd will beat it.

 

I'll keep my fingers crossed.

----------

## _______0

 *NeddySeagoon wrote:*   

>  *The Doctor wrote:*   It took, what, 5 years for HAL to become unmaintainable. I wonder if systemd will beat it. 
> 
> I'll keep my fingers crossed.

 

ha ha ha!!

ok I am reading these last couple of days about Gentoo and systemshiteD convergence and is all like bad dream.

In this case what are the options? If it's been decided to willingly slowly brake openrc to force systemd, what are the options?

Can LFS use gentoo ebuilds?

----------

## NeddySeagoon

_______0.

There are no plans in Gentoo to force systemd on anyone.

There are several init systems in the portage tree - openrc is the default now and is likely to remain so.

Gnome has decided that if you want to use Gnome, you need to use systemd too.  Thats Gnomes loss as systemd is Linux only.

Personally, after 14 years of Gnome, I don't want to use it so badly I will tolerate systemd, so bye bye Gnome.

Putting portage onto LFS won't help.  If you build Gnome from scratch, it will still need systemd

----------

## grey_dot

 *NeddySeagoon wrote:*   

> _______0.
> 
> There are no plans in Gentoo to force systemd on anyone.
> 
> There are several init systems in the portage tree - openrc is the default now and is likely to remain so.
> ...

 

As I said in some other thread, the problem is not the systemd forcing. The problem is the kludges implemented to incorporate systemd and other crappy written pieces of code. The requirement of /usr being mounted is a perfect example of that.

----------

## grey_dot

 *croutch wrote:*   

> Little info about it.
> 
> http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/
> 
> http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

 

freedesktop is broken. Can we just drop it?

----------

## miket

 *The Doctor wrote:*   

> but it is mostly their stuff that is breaking. Network Manager, pulsaudio, dbus, etc. Half of the programs they list as "broken" shouldn't even be started that early.

 

Most of those things shouldn't even be installed, let alone started early.

It boggles my mind how much extra effort that Lennart Poettering has to go through to keep up with his ideology of how system initialization should work.  He works very quickly and somehow is very convincing.  He has to work quickly: lots of things break when they get in his way.  What he never seems to recognize that it is his ideology that is broken!  If things were so blindingly simple, why is there so much code in systemd to handle corner cases (not to mention what you cite about certain daemons being started too soon)?

I have made my living as a programmer for many years.  Over this time, I have had lots of sad experience in cleaning up behind programmers who made their implementions very quickly but without nearly enough consideration over whether they were doing things the right way or even cleanly.  Shiny features, yes, but strange failure modes, poor performance under load, and nightmare maintenance.  Also, to shoehorn their new code into the existing system, they often screw up a lot of that.  What you have to have when designing software is an ability to think ahead to how things will work in the future and a willingness to rethink your grand scheme when you sense things are getting too strange.

The biggest sign of one of these programmers at work is when testers (or worse, real users) report big bugs.  The rock-star programmers respond with even more speed and even less stopping to think--and ever deeper crap to wade through when I need to come back and fix it.

So, do I detect this pattern of rock-star hubris in Lennart Poettering?  You bet I do!  His initial intuition is appealing:  services should start when other services request them, so we don't need to do fancy dependency calculations.  Yes, but we'd like to stop services cleanly, so we'll rely on a kernel system (cgroups) that a kernel maintainer reviewing that code (Al Viro) said was so ghastly that it should never have been merged into the kernel (and has since been refactored inconveniently to systemd).  Yes, but it would be so convenient if we didn't have to rely on separate packages that do things well and cleanly, so we'll reimplement the suckers.  Yes, but for everyone to use our system well, they'll have to patch in shims into all of those daemons so they'll talk nicely to systemd.  Yes, but our declaration files are now going to have to have start-after indications anyway, but somehow we'll still not get it right (why doesn't units/sound.target not have entries like After= or Requires= ?)

There are too many rounds of "yes, but" in there.  I browsed through the source tree as I wrote the above paragraph, and grew ever more apalled as I went along.  It does way too much.  If he had approached the job with the proper degree of humility, he would have stopped to rethink before he took so many flights of fancy.

Here is a great blog post about the upheavals in Linux:  http://www.pappp.net/?p=969

 *The Doctor wrote:*   

> Its fairly easy to write a busybox init by hand. It won't break and will do the job.

 

Funny, that's the way I've been making lemonade out of lemons:  I set up a small boot partition with extlinux, kernels, scripts, and staticly linked busybox and LVM2.  I do something with it that we could never do without early userspace:  root directory on an LVM volume.  What I don't have in this early boot are any of udev, kernel-module insertion, or even an initramfs.  Even if they went through with the stoopid notion of --prefix=/usr/ instead of --prefix=/ for packages that ought to be in /bin or /sbin, I'd still boot without a hitch.  Hooray for busybox!

----------

## NeddySeagoon

miket,

 *miket wrote:*   

> 
> 
> Funny, that's the way I've been making lemonade out of lemons: I set up a small boot partition with extlinux, kernels, scripts, and staticly linked busybox and LVM2. I do something with it that we could never do without early userspace: root directory on an LVM volume. 

 

I read that as if you have what would be your initramfs on a real partition but otherwise it works the same.

I like the sound of that - thank you.  I have root on LVM on raid5

----------

## miket

 *NeddySeagoon wrote:*   

> miket,
> 
>  *miket wrote:*   
> 
> Funny, that's the way I've been making lemonade out of lemons: I set up a small boot partition with extlinux, kernels, scripts, and staticly linked busybox and LVM2. I do something with it that we could never do without early userspace: root directory on an LVM volume.  
> ...

 

That's exactly what it is.  I got it to work, but then I wanted to make a repeatable recipe that I could try out a number of times on different machines with different configurations before I put it into a how-to.  I was working up a script to make it easier to set up the LVM volumes, but that's when work got into the way.  Sigh.

If you want to try things yourself, this outlines what I recall you'll need.

 partition the drive for a smallish ext2, ext3, or ext4 boot partition and a bunch of space for the LVM physical volume

 follow the documentation for doing an LVM install (set up a volume group or groups to taste; set up logical volumes as you like) and install Gentoo on your mounted set of volumes.  Leave the boot partition unmounted for now.  I had LV's for /, /usr, /home, /opt, /var, and /tmp, but suit yourself.  Don't do grub, and leave the kernel in /usr/src/linux/arch/x86/boot/bzImage for now.

 emerge lvm2 with USE="static -udev".  If you really want to emerge a static busybox for yourself, go for it.  I just used the one from the SystemRescueCD.  The same goes for extlinux:  I took the lazy way out.

 outside the chroot, mount the boot directory and switch to it.

 make directories you'll need.  I'm not quite certain you really need all of these; this is one area where I wanted to test things.  This ought to get you going, though:

```
mkdir -p bin boot dev etc/lvm extlinux mnt/rootfs proc sbin sys tmp
```

 if you compiled the kernel with devtmpfs, you can leave /boot/dev unpopulated, otherwise populate it from the stage3 tarball.  I did mine with devtmpfs, and since I did not try my setup without that enabled, I don't even know if the static dev would work right or maybe need some tweaking in the scripts.

 install extlinux from the CD

```
extlinux --install /mnt/boot/extlinux
```

 then copy whatever syslinux modules you'll need from the /usr/share/syslinux directory.  I got cmd.c32, lua.c32, menu.c32, reboot.c32, rosh.c32, vesainfo.c32, vesamenu.c32, and background.png, but this is an area for your own tweaking. (I was going to get around to use the Lua, but I never did.)

 install the bootloader from the CD (If you're nervous about this, you could do it from the very first step before you did the partitioning)

```
dd if=/usr/share/syslinux/mbr.bin of=/dev/[your device]
```

.  There are lots of variants here, including GPT support.  I didn't mess with anything but the vanilla mbr.bin.  See the syslinux documentation for more possibilities.

 put busybox in /boot/bin and set up the command symlinks in that directory.

 copy the newly emerged lvm binary to /boot/sbin and make links to it under all the command names (like busybox, lvm is a multicall binary).  I never tried tweaking this to anything minimal; I just made links for all of them.  Who knows what you might want to use when you are booted into the rescue system? (it comes up wonderfully quickly)

 copy the kernel to /mnt/boot/boot.  This is a point of departure from a normal installation:  in the running system, you'd look to /boot/boot rather than /boot for your kernels.

 set up the extlinux.conf.  The following is basically what I have.  Adjust the names of the kernels as necessary (I use the same kernel for both recovery and the main system).  Pay attention to the real_root parameter on the kernel command line.  That's the name of the LV with the root FS.  Observe also that the recovery option does not set an init parameter on the command line: it uses the Busybox init.

```
UI menu.c32

PROMPT 0

MENU TITLE Pick A System To Start

MENU BACKGROUND background.png

TIMEOUT 600

LABEL test1

  MENU LABEL Start system

  MENU default

  KERNEL /boot/kernel-v.v.v-gentoo

  APPEND root=/dev/sda1 real_root=/dev/vg/test1_root video=[whatever] init=/boot/lvminit.sh

LABEL recover

  MENU LABEL Busybox recovery shell

  KERNEL /boot/kernel-v.v.v-gentoo

  APPEND root=/dev/sda1 video=[whatever]
```

 Here's the /boot/lvminit.sh script:

```
#!/bin/ash

fail() {

        echo "$@"

        echo "dropping to shell"

        exec /bin/ash

}

mount -t proc none /proc

mount -t sysfs none /sys

hadDev=1

if [ ! -c /dev/null ]; then

        mount -t devtmpfs none /dev

        hadDev=

fi

rootfs=

for opt in $(cat /proc/cmdline); do

        case $opt in

                real_root=*)

                        targ=$(echo $opt | cut -d= -f2);

                        if [ "$targ" == "LABEL" ] || [ "$targ" == "UUID" ]; then

                                tval=$(echo $opt | cut -d= -f3)

                                rootfs=$(findfs "$targ"="$tval")

                        else

                                rootfs=$(echo $opt | cut -d= -f2)

                        fi

                ;;

        esac

done

if [ -z "$rootfs" ]; then

        fail "Could not find name of root filesystem"

fi

vgchange -a y --sysinit

if ! mount -o ro "$rootfs" /mnt/rootfs; then

        fail "Failed to mount $rootfs"

fi

echo "$rootfs mounted"

#umount /tmp

umount /sys

if [ -z "hadDev" ]; then

        umount /dev

fi

umount /proc

cd /mnt/rootfs

pivot_root . boot

exec chroot . /sbin/init a </dev/console >/dev/console 2>&1
```

 there are a few smaller details you'll need to keep things happy.  Run these commands from the boot directory.

```
echo -n >etc/fstab

ln -s /proc/mounts etc/mtab

cp [path-to-mounted-root-volume]/etc/lvm/lvm.conf etc/lvm

echo -n >proc/mounts  # yes, really
```

 now for the recovery system (very useful to have!):  you can use the busybox base inittab (inittab.bz2 from the Busybox distribution--it may not be on the live CD) or can get by without one.  If you use one, decompress it to /mnt/boot/etc/inittab; if not, Busybox will use its defaults.  Make the following links in the /sbin directory to your busybox binary: init, reboot, swapoff

 you need a little shell script to boot into the recovery system.  If you did not install the busybox inittab, you'll have to put the script at the default location of /etc/init.d/rcS.  If you want to put it somewhere else, install the inittab and edit the line beginning with ::sysinit: to point to the file.  In any event, here is such a file:

```
#!/bin/ash

mount -t proc none /proc

mount -t sysfs none /sys

if [ ! -c /dev/null ]; then

        mount -t devtmpfs none /dev

fi

mount -t tmpfs -o size=200M tmpfs /tmp

echo -n 0 >/proc/sys/kernel/printk

echo

echo "Minimal rescue mode"
```

There.  What you get is an early-userspace setup that is very tweakable and does not require generating an initrd.

It's all nifty and slick, but as I look at it, I notice a problem which may be looming:  I had misremembered what happens in lvminit.sh.  I thought it mounted all of the LVM volumes.  It doesn't.  It mounts only the root volume and leaves it up to OpenRC to mount all the rest.  I'm betting, though, that since LVM is already started and that (I hope) OpenRC, fsck, and mount will continue to live in sane places, there won't be problems.

The /etc/fstab on the main system has all the entries except for /boot.  You get /boot for free because of the pivot_root.  This is my main fstab:

```
/dev/vg/test1_root      /               ext3            noatime,user_xattr      0 1

/dev/vg/test1_opt       /opt            ext3            noatime,user_xattr      0 2

/dev/vg/test1_usr       /usr            ext3            noatime,user_xattr      0 2

/dev/vg/test1_var       /var            ext3            noatime,user_xattr      0 2

/dev/vg/common_home     /home           ext3            noatime,user_xattr      0 2

/dev/vg/common_tmp      /tmp            ext3            noatime,user_xattr      0 2

/dev/sda2               none            swap            sw              0 0

192.168.0.10:/usr/portage     /usr/portage    nfs     rw,soft         0 0
```

I went ahead and re-emerged lvm2 with the default USE flags for use on the main system.  I think I should not have bothered.  Really, there may be some people who would want lvm to get udev notifications, but I don't care about that.  The device mapper is already running by the time OpenRC starts, so I don't even start the lvm service.

So I hope this gives you enough of a start if you want to play with it.  Some day I'll get it ready for a real writeup with actual setup scripts, but in the meantime, have fun.

----------

## NeddySeagoon

miket,

I've pretty much done it but I wrapped it all up in a file and compressed it.

----------

## _______0

May I add a tip to make things less complicated?

LVM is quite an elegant solution but having to destroy hdd geometry for a boot partition is sad.

I love this:

```
pvcreate /dev/sda
```

For anyone having a desktop I advice to use a cheap small usb for the boot partition and fugly uefi stuff.

----------

## 1clue

OK I'm kinda old school on Linux but not really that much of a boot expert.  Well, not a boot expert at all, to be more accurate.  I've followed the init scripts before and I know how it works, but I don't know any magical spells to make it different.

I've waded through a lot of the systemd rants, and looked at the "case for /usr merge" page.  Frankly that page looks a lot like horse droppings, I disagree with every point.

In particular, my main reason for a separate /usr has been for simpler rescue in the event of disk problems.  In other words, "myth 9."

I don't care what megabyte number you have on your root partition.  The core utilities which are traditionally in /bin and /sbin rarely change if ever.  If /var and /tmp are also on different partitions, then root partition is essentially read-only unless you personally change your configuration.  That's bound to have fewer errors than something that's continually updated.  Not only that, but having Libre Office, Firefox and all that junk (like plugins) that come with it on the boot partition scares the crap out of me.

Again, it all comes to what you plan to do with the box.  For a specific purpose machine like a router I'd put it all in the same place.  For a workstation it's entirely different.

Why in the world would you not want to establish a minimum environment and system sanity before starting extraneous services?

This page (the case for usr merge) reads like a first-time-user rant.

Maybe I should have looked this up first:  What does Linus Torvalds say about systemd?  Time to google it.

----------

## 1clue

Wow.

I just googled it.  I always thought it would be cool if Linus recognized something I did, but not if he recognizes it that way.  Whew!

OK I feel much better about my opinion of systemd now.

So the next question is, HOW can it be fixed?  Just take udev back to the point just before it was broken?

I'd dive in, but I simply don't understand the problem domain.  I would hate to have a n00b go make changes to anything boot-critical, and for this sort of thing I'd be a n00b.  I haven't even done C or C++ for over a decade.  I'd even have to start over on that.

Personally I have no use for gnome anymore.  I was thinking to go back to fvwm frankly.  It's a PITA but at least you get what you want.

----------

## saellaven

 *1clue wrote:*   

> Wow.
> 
> I just googled it.  I always thought it would be cool if Linus recognized something I did, but not if he recognizes it that way.  Whew!
> 
> OK I feel much better about my opinion of systemd now.
> ...

 

SteveL has developed patches to keep a separate /usr working and it is working fine for me.

The developers of systemd are hostile to ANYTHING not done their way (and their way includes infecting as much of your system as possible) and some other projects, like GNOME have hitched their ride to systemd, so some people may be unable to avoid it. If people want the baggage of systemd, more power to them, but I do not want their mindset taking over my system.

For better or worse, the Gentoo Council has basically accepted the arguments presented in Poettering's debunked rants and have turned hostile to existing working systems to facilitate systemd, even though systemd is only an option and not mandatory on Gentoo at this time. I happen to believe this paves the way for the Council to eventually force us onto systemd, particularly given that the Council has completely ignored any arguments contrary to Poettering, the patches that SteveL coded TWO YEARS ago, etc. TomWij said that William Hubbs - the Council member that requested the elimination of a separate /usr (whom is also a systemd herd member and the lead on openrc), despite knowing about SteveL's patches and not informing the Council about them, would respond to us here, but he has yet to do so.

----------

## ulenrich

 *saellaven wrote:*   

> SteveL has developed patches to keep a separate /usr working and it is working fine for me.

 

By giving SteveL Your feedback, you pushed him to announce a big step forward:

 *steveL wrote:*   

> Thanks, saellaven, that's great info to have. I've updated the title to reflect that the patches are confirmed to work with eudev.
> 
> The way things are going with upstream "systemd-udev-dbus-syslog-kitchen_sink" it looks like I'll have to switch soon as well: 

 

He finally has promised to use this alternative! This hopefully will foster further development and he will overcome his Stockholm syndrome as he writes about it.

----------

## mv

I posted this question quite a while ago, but it was never answered:

If you use a solution as suggested here and do not replace udev completely by mdev already on the initfs, you actually postpone starting udev.

What is happening with all the signals about connected devices etc. until udev eventually is started? ["signal" is not the technically correct term, but I hope you can understand what I mean: the way how the kernel usually "informs" udev about new devices]

Does the kernel buffer/queue these "signals" for you (and then deliver them all to udev in the moment when udev is finally up?) or do all these signals go unnoticed and are actually lost (which could mean that e.g. some initialization usually done by udev for some hardware actually never happens when you use such solutions)? If the kernel does queue, is the length of the queue dynamic or is there some danger that it may be too short if a lot of devices are attached?

I do not want to find this out by trial-and-error (it might work hundreds of times but eventually fail) but would like to know the answer from somebody who actually knows how these "signals" are implemented in linux...

----------

## gotyaoi

So, from looking around a bit, this is what I have discovered so far (caveat; I may have no idea what I'm talking about):

Under normal operation, when a device is plugged in, the kernel creates a folder for it under /sys somewhere which contains various files who's contents are the attributes of the new device. It also generates a 'uevent', which it both sends out over the netlink socket as well as storing in the new folder under /sys. If udev is currently running, it is listening on the netlink socket, receives the uevent, runs it through the rules and takes appropriate action.

For the case where udev is not yet running, the uevent sent over netlink is ignored. When udev starts is where things happen.

The udev init script calls

```
udevadm trigger --type=subsystems --action=add

udevadm trigger --type=device --action=add
```

looking into the source code here: http://cgit.freedesktop.org/systemd/systemd/tree/src/udev/udevadm-trigger.c

we see that this eventually boils down to a call to udev_enumerate_scan_subsystems or udev_enumerate_scan_devices, which can be found here: http://cgit.freedesktop.org/systemd/systemd/tree/src/libudev/libudev-enumerate.c

As they say, these functions scan /sys for kernel subsystems or devices respectively and adds them. So it seems that there's not really a buffer per-se, except the /sys tree.

----------

## mv

Thanks for your reply.

So there are two possibilities:

1. The loss of the uevent (finally, I know the right term) is severe for some devices which need udev care and which perhaps do not appear in /sys correspondingly. If this is the case this make all early boot techniques (including initrd) very questionable since you have a built-in race for all attached such devices.

2. All kernel drivers follow the rule that the later trigger actually has the same effect as an uevent would have (I did not check the sources carefully, but I assume that at least the standard udev rules will be carried out for the devices started with adevadm trigger).

If the latter is the case, I do not understand the whole arguments for separate /usr: You just need the early-boot dependencies correctly and start udev only afterwards. This should work with any sane init-system and all claims about e.g. bluetooth needing stuff would just be nonsense, since one could just mount /usr before starting to accept uevents.

----------

## Anon-E-moose

 *mv wrote:*   

> If the latter is the case, I do not understand the whole arguments for separate /usr:

 

it's simple really, they want people to only look at things one way. 

I'm not sure why a distro based on building source from scratch thinks that blindly following binary distros is a good idea.

----------

## saellaven

 *mv wrote:*   

> 
> 
> If the latter is the case, I do not understand the whole arguments for separate /usr: You just need the early-boot dependencies correctly and start udev only afterwards. This should work with any sane init-system and all claims about e.g. bluetooth needing stuff would just be nonsense, since one could just mount /usr before starting to accept uevents.

 

Because there is a political agenda. The technical merits have long been debunked multiple times and there are easy solutions which are intentionally being withheld, because zealotry wins the day.

----------

## mv

I am still not convinced that case 2 of the possibilities I mentioned above is really true. You all seem to assume this tacitly.

 *saellaven wrote:*   

> Because there is a political agenda. The technical merits have long been debunked multiple times

 

Not really: It was always claimed that with systemd it is not possible.

But if case 2 of the above possibilities actually is true, it should also be trivial to do this even with systemd: Essentially you just have to depend (with Wants= and After=) on usr.mount and make sure that the latter depends on whatever you need to mount /usr (and all contain DefaultDependencies=No). Only systemd itself would have to be moved to /, but this move was a gentoo decision against upstream...

----------

## saellaven

 *mv wrote:*   

> I am still not convinced that case 2 of the possibilities I mentioned above is really true. You all seem to assume this tacitly.
> 
>  *saellaven wrote:*   Because there is a political agenda. The technical merits have long been debunked multiple times 
> 
> Not really: It was always claimed that with systemd it is not possible.
> ...

 

It's not possible with systemd because the developer of systemd doesn't want it to be possible. systemd (formerly udev) managed a separate /usr just fine until he decided to intentionally remove the functionality. He sees "more purpose" for a separate /var but doesn't promise that won't go away if he desires at some point as well.

He doesn't really care about the technical merits. He just does what he wants and assumes that his way is the one true way that everyone must follow. Outside of some corner cases, none of this stuff was broken until he decided to break it and in doing so, he's breaking the majority of systems that just worked before. Today, it's my pet piece of functionality, tomorrow it's going to be someone else's, yesterday it was network renaming which totally fails his stated purpose of making it persistent when the solution was always there (assigning by MAC address).

----------

## mv

 *saellaven wrote:*   

> It's not possible with systemd because the developer of systemd doesn't want it to be possible.

 

This is not true: Despite giving his post an opposite title, Poettering claims that systemd itself is fine with separate /usr, and only bluetooth etc. are blaimed. With setting the dependencies as I mentioned, things like these should not be a problems. I really see no technical obstacle why a separate /usr shouldn't work with systemd (again: if case 2 is true).

Of course, with systemd installed in /usr as by the gentoo ebuild, there is no chance, but Poettering himself claimed that this is not a good idea.

----------

## steveL

 *ulenrich wrote:*   

> By giving SteveL Your feedback, you pushed him to announce a big step forward:
> 
>  *steveL wrote:*   Thanks, saellaven, that's great info to have. I've updated the title to reflect that the patches are confirmed to work with eudev.
> 
> The way things are going with upstream "systemd-udev-dbus-syslog-kitchen_sink" it looks like I'll have to switch soon as well:  
> ...

 

WTF are you on about? You apparently cannot even read, and want to attribute a symptom I describe, with a quote showing said symptom, to me. You are deliberately misquoting me and twisting my words, and I do not like it.

Don't do that.

Further more I am free to use whatever I want: it's not your place to declaim that "finally" I have promised to use an alternative. I'm the one who's kept upstream udev working without an initramfs for the last 2 years. I do not at all appreciate you injecting your ignorance and attributing your position to me: I have not promised to use anything at all. What I choose to use in the future, is my business. So GTFO with your politically-motivated claptrap.

And learn to think about what you are saying, before you post such blatant nonsense in the future.

----------

## steveL

 *mv wrote:*   

> So there are two possibilities:
> 
> 1. The loss of the uevent (finally, I know the right term) is severe for some devices which need udev care and which perhaps do not appear in /sys correspondingly. If this is the case this make all early boot techniques (including initrd) very questionable since you have a built-in race for all attached such devices.
> 
> 2. All kernel drivers follow the rule that the later trigger actually has the same effect as an uevent would have (I did not check the sources carefully, but I assume that at least the standard udev rules will be carried out for the devices started with adevadm trigger).
> ...

 

I sincerely doubt that the device enumeration scan is not as encompassing as the uevents. In any event as you point out, the same thing happens in both cases: there is always a delay between system boot and udev startup: there has to be, since all local drives must be mounted by that point, as Greg K-H initially outlined on the mailing-list. So it really makes no difference, and as you say, you just need the early boot dependencies available, however you do it.

WRT bluetooth, apparently the device (keyboard) is needed in order to interact with the system during early boot. Not sure if that's grub (if so, how do you get grub to load it, and what does that have to do with udev since the kernel has not started up?) or just being able to interrupt the boot process, eg for "interactive" startup with openrc. In any event, the chicken-and-egg nature of needing udev-started devices in order to start udev has never been explained, but it's not my problem, so I don't really care too much.

----------

## dmpogo

 *steveL wrote:*   

> 
> 
> WRT bluetooth, apparently the device (keyboard) is needed in order to interact with the system during early boot. Not sure if that's grub (if so, how do you get grub to load it, and what does that have to do with udev since the kernel has not started up?) or just being able to interrupt the boot process, 

 

I, for one, can't develop much sympathy to that problem.  I think it is not much to ask user to connect physical keyboard if one is to  play with interactive boot.  Otherwise one may put out an argument that one needs a networking stack in grub to be able to early ssh into the machine while it is booting etc.  There are limits how much of different hardware can be available during early boot, or it is not early at all.

----------

## mv

 *steveL wrote:*   

> [I sincerely doubt that the device enumeration scan is not as encompassing as the uevents. In any event as you point out, the same thing happens in both cases: there is always a delay between system boot and udev startup

 

This was also my thought, but OTOH, it wouldn't be the first time that udev has an obvious race which nobody noticed for years...

(Recall the net naming disaster. And e.g. once or twice a year it still happens that my camera and tv card get exchanged links, despite  they should be setup depending on "DRIVER" which should be unique for these two devices...)

----------

## steveL

 *dmpogo wrote:*   

>  *steveL wrote:*   WRT bluetooth, apparently the device (keyboard) is needed in order to interact with the system during early boot. Not sure if that's grub or just being able to interrupt the boot process,  
> 
> I, for one, can't develop much sympathy to that problem.  I think it is not much to ask user to connect physical keyboard if one is to  play with interactive boot.  Otherwise one may put out an argument that one needs a networking stack in grub to be able to early ssh into the machine while it is booting etc.  There are limits how much of different hardware can be available during early boot, or it is not early at all.

 

Well, ISTR that the device was basically a tablet, or some such, that only did bluetooth for keyboard. As you say, an esoteric setup, and one for which I have no issue with an initrd/ramfs: if you need it, by all means use it.

I just don't accept that we should make the simple cases complex, in order to deal with the complex cases. As we've discussed, the same thing happens in both instances, whether you're booting via initramfs, or directly via rootfs. So we should just focus on those issues (which aren't that difficult, ffs, they've been with us since the beginning if we consider eg encrypted root) instead of arguing about why people should just do things "our way" which is never the right approach. Or we wouldn't be able to work, since one size really would fit all.

I don't see that happening any time soon, and even if it does, Gentoo is predicated on the exact opposite approach to computing.

 *mv wrote:*   

>  *steveL wrote:*   [I sincerely doubt that the device enumeration scan is not as encompassing as the uevents. In any event as you point out, the same thing happens in both cases: there is always a delay between system boot and udev startup 
> 
> This was also my thought, but OTOH, it wouldn't be the first time that udev has an obvious race which nobody noticed for years...
> 
> (Recall the net naming disaster. And e.g. once or twice a year it still happens that my camera and tv card get exchanged links, despite  they should be setup depending on "DRIVER" which should be unique for these two devices...)

 

There may well be a "race" in that uevents may be fired before udev starts up and does a scan. Just from what I recall (for PCI), I'd expect that scan to initialise the drivers, ie call the device driver and tell it we're starting up. Granted some devices may already have started up (some will have to, such as our disk) but isn't that something expected?

The important point is that it's inherent in the whole process, and thus it makes no difference whether you are in an initramfs or not.

Either way you need "local" drives mounted before udev starts, and initramfs is meant to handle the shiny "new" world of /usr over the network, so there is no way anyone can tell me that it does not also imply a delay between bootup and udev startup, which requires /usr mounted before it starts.

If there is an issue, then it needs to be addressed for both setups, and the "traditional" method (with udev delayed after localmount) is no worse than initramfs in this regard. It simply cannot be, if one is looking to fix the "race".

----------

## mv

 *steveL wrote:*   

> it makes no difference whether you are in an initramfs or not

 

True, but since usually initramfs is finished very quickly, this might mask some race which always existed.

----------

