# Failed to boot due to udev

## lowsfer

Hi guys!

Yesterday I occasionally booted my laptop with an external eSATA drive plugged in. Then it stopped starting up with "waiting for uevents to be processed", then give an 60sec timeout messege and a queue of uevents with "!!" at the end.

Fortunately I have another Linux distribution installed so I'm able to use chroot to do something. (I always install another distribution in case of such disaster   :Very Happy:  )

At first I thought It was because I forgot to compile something into the kernel. So I tried genkernel, but the problem remained.

So I thought it was because when udev found my eSATA drive, it wrote something bad. I tried to re-install udev and udisks, but that didn't help. Then I tried to remove all rules in /etc/udev/rules.d and /lib/udev/rules.d, then the system started! But certainly this solution is dirty, it causes other problems.

Does anyone know what happened to udev?

I can remove the udev rules one by one to see which one caused the problem but that would cost lots of time. Maybe I can do it later.

Sorry if my English if not easy to understand. My first language is not English.

----------

## vaxbrat

Did you update and get a new version of udev since the last time you use the external drive?  That might require a visit to the rules.d to tweak some stuff that is now obsolete.  In my experience with udev so far I've only needed to go in and get rid of the xxx-persistent-xxx files to have udev rebuild them.  That's also usually only needed when I clone a disk and bring it to new hardware.   Most common file I get rid of in this case is the persistent net rules since mac addresses have changed for NICs.

----------

## lowsfer

 *vaxbrat wrote:*   

> Did you update and get a new version of udev since the last time you use the external drive?  That might require a visit to the rules.d to tweak some stuff that is now obsolete.  In my experience with udev so far I've only needed to go in and get rid of the xxx-persistent-xxx files to have udev rebuild them.  That's also usually only needed when I clone a disk and bring it to new hardware.   Most common file I get rid of in this case is the persistent net rules since mac addresses have changed for NICs.

 

Not on 5 April. I'm sure the update on 4 Apr was not the reason because I did boot several times during the day on 5 April without problem, the problem occurs in the night. I tried removing all *persistent* files, but no luck. I did completely remove udev with "emerge --unmerge udev udisks", and reinstall it to have the files regenerated. Maybe later I will remove the rules one-by-one to locate the faulty file when I got some time.

The external eSATA had nothing to do with the system, it is just for personal storage. I rebooted from windows to gentoo with it plugged in. I just forgot to unplug it. Didn't know this could cause problems.

```

# genlop -t udev

 * sys-fs/udev

     Thu Mar 31 12:10:59 2011 >>> sys-fs/udev-164-r2

       merge time: 19 seconds.

     Fri Apr  1 15:34:38 2011 >>> sys-fs/udev-167

       merge time: 28 seconds.

     Mon Apr  4 19:46:23 2011 >>> sys-fs/udev-167-r1

       merge time: 38 seconds.

     Wed Apr  6 10:00:23 2011 >>> sys-fs/udev-167-r1

       merge time: 46 seconds.

     Wed Apr  6 10:19:28 2011 >>> sys-fs/udev-167-r1

       merge time: 41 seconds.

     Wed Apr  6 13:26:46 2011 >>> sys-fs/udev-167-r1

       merge time: 45 seconds.

     Wed Apr  6 13:38:17 2011 >>> sys-fs/udev-167-r1

       merge time: 38 seconds.

```

----------

## lowsfer

I have found that commenting out this line in 80-drivers.rules make the system able to boot.

```

 DRIVER!="?*", ENV{MODALIAS}=="?*", RUN+="/sbin/modprobe -bv $env{MODALIAS}"

```

But my openSUSE is with exactly the same 80-drivers.rules file. So I guess udev read some other files which are modified incorrectly when eSATA drive is detected. Is there a file where udev reads hardware info from? Or is there a tool that can help me reset udev settings?

----------

## lowsfer

I found that many sd* nodes exists in my gentoo's /dev directory when I checked from openSUSE installed on my laptop.

Is that normal? I guess nodes like sdb,sdc,sdd should be removed when the devices do not exist, right?

I deleted all non-existing nodes ( sda11-15, sdb*, sdc*, sdd*). But the issue remain unchanged.

----------

## cach0rr0

 *lowsfer wrote:*   

> 
> 
> I deleted all non-existing nodes ( sda11-15, sdb*, sdc*, sdd*). But the issue remain unchanged.

 

these device nodes will be recreated by udev upon every reboot

for what it's worth, the same thing happens to me when i reboot my system with my phone plugged in (via USB), or plug my phone in too early in the boot process.

No idea the cause. I simply mask the issue by unplugging my phone and leaving it that way until things are bootedup  :Smile: 

----------

## lowsfer

 *cach0rr0 wrote:*   

>  *lowsfer wrote:*   
> 
> I deleted all non-existing nodes ( sda11-15, sdb*, sdc*, sdd*). But the issue remain unchanged. 
> 
> these device nodes will be recreated by udev upon every reboot
> ...

 

But it seems it will never boot up for me. How did you get it boot up? Just wait? I waited more than 10 minutes, nothing happened.

----------

## Ant P.

It might be getting confused and thinking the eSATA is the first hard disk. Change your fstab to use /dev/disk/by-uuid/* if it has lines for /dev/sd*, and make sure devtmpfs is turned on in the kernel.

----------

## NeddySeagoon

lowsfer,

It looks like you have a static /dev/ on your hard drive.  This will become hidden when the tmpfs /dev/ is mounted over the top and populated by udev.

What you see in Gentoos /dev when you look from another distro has nothing to do with udev.

----------

## cach0rr0

 *lowsfer wrote:*   

> 
> 
> But it seems it will never boot up for me. How did you get it boot up? Just wait? I waited more than 10 minutes, nothing happened.

 

nope. I took the sledgehammer approach - unplugged my phone, Alt+SysRq+B, reboot without the phone plugged in, plug it back in *after* the system is up.

----------

## lowsfer

 *Ant P. wrote:*   

> It might be getting confused and thinking the eSATA is the first hard disk. Change your fstab to use /dev/disk/by-uuid/* if it has lines for /dev/sd*, and make sure devtmpfs is turned on in the kernel.

 

Yes I was using /dev/sd*. But later I removed the eSATA disk, the issue remain unchanged.

BTW, in grub.conf, I was use the corresponding (hd0,*-1) and grub can load well. In my experience, grub is always with same disk order with the system.(correct me if I'm wrong)

----------

## lowsfer

 *NeddySeagoon wrote:*   

> lowsfer,
> 
> It looks like you have a static /dev/ on your hard drive.  This will become hidden when the tmpfs /dev/ is mounted over the top and populated by udev.
> 
> What you see in Gentoos /dev when you look from another distro has nothing to do with udev.

 

Hmm, thanks, that's interesting. I also check my openSUSE installed in the same laptop from a liveDVD and found only md*, pty, null exist in its /dev when not running. But gentoo has heaps of nodes there. I heard other distros are also like openSUSE, while gentoo is unique here. Why the difference?

----------

## lowsfer

 *cach0rr0 wrote:*   

>  *lowsfer wrote:*   
> 
> But it seems it will never boot up for me. How did you get it boot up? Just wait? I waited more than 10 minutes, nothing happened. 
> 
> nope. I took the sledgehammer approach - unplugged my phone, Alt+SysRq+B, reboot without the phone plugged in, plug it back in *after* the system is up.

 

You are much luckier than me, I have to say.

Anyway I have created all the binary packages with quickpkg and re-installed the system. I also copied the complete /etc directory. Now it's up again!

I will backup my gentoo and then do the test again with eSATA drive plugged in. If the issue comes up again, I guess... it should be reported as some kind of bug?

----------

## lowsfer

I tried several times booting up with eSATA disk plugged in again. But the problem didn't appear. So this problem is not repeatable.

Thanks for you help, guys.

I guess my new laptop just got angry about high load of compiling all the gentoo packages and CPU temperature kept above 80 degree C, and that was an warning.  :Wink: 

It is Intel i7-820QM, which gets really hot at full load. Sometimes I get worried about the temperature and restrict the frequency to 1.2GHz with cpufreq-set when compiling packages with -j9. One of my friend toasted his i7 laptop with a MPI based numerical quantum chemistry software. I don't want to follow him. I guess high compiling load in gentoo could be related to short fan lifetime of my last two laptops to some extent.  :Crying or Very sad:  Maybe gentoo is better for Desktop PCs rather than laptops, especially when you want to use "~amd64" keywords. Anyway, when you get something, you have to pay for it.

----------

## cach0rr0

 *lowsfer wrote:*   

>  I guess high compiling load in gentoo could be related to short fan lifetime of my last two laptops to some extent.  Maybe gentoo is better for Desktop PCs rather than laptops, especially when you want to use "~amd64" keywords. Anyway, when you get something, you have to pay for it.

 

I just run -j2 on a dual core, but use zen-sources which incorporates the BFS scheduler. Makes all the difference in the world. Not as far as heating, but as far as not having to crank the jobs way up in order to get a speedy compile, and still having good interactivity on the desktop. 

If you DO have a fairly beefy desktop PC somewhere, it might be worthwhile setting up distcc - I do this with my laptops, and have them offload the heavy builds to the desktop. Each laptop gets 2 jobs, the desktop gets 4 (quad phenom), things finish in a fraction of the time, and my CPU doesn't have to be out of control longer than is absolutely necessary  :Smile: 

----------

## lowsfer

 *cach0rr0 wrote:*   

>  *lowsfer wrote:*    I guess high compiling load in gentoo could be related to short fan lifetime of my last two laptops to some extent.  Maybe gentoo is better for Desktop PCs rather than laptops, especially when you want to use "~amd64" keywords. Anyway, when you get something, you have to pay for it. 
> 
> I just run -j2 on a dual core, but use zen-sources which incorporates the BFS scheduler. Makes all the difference in the world. Not as far as heating, but as far as not having to crank the jobs way up in order to get a speedy compile, and still having good interactivity on the desktop. 
> 
> If you DO have a fairly beefy desktop PC somewhere, it might be worthwhile setting up distcc - I do this with my laptops, and have them offload the heavy builds to the desktop. Each laptop gets 2 jobs, the desktop gets 4 (quad phenom), things finish in a fraction of the time, and my CPU doesn't have to be out of control longer than is absolutely necessary 

 

No I don't have a desktop PC.

I do have a dedicated workstation with openSUSE installed for my numerical computation work in my uni. It is a really really powerful machine. I'm a new PhD student. I'm not sure about this but I think setting up that machine for personal use might be some kind of abuse, which may cause trouble for me...

----------

## vaxbrat

Here's a trick to do on your running system if you want to look at any static /dev nodes that might be overlayed by udev:

```
mkdir /mnt/rawroot

mount --bind / /mnt/rawroot 
```

Then look in /mnt/rawroot/dev and tweak or clean up.  The key is that the --bind option doesn't mount up any overlaid mounts on the rawroot mountpoint.

----------

