# "Filesystem couldn't be fixed" error on boot

## azdruid

Following a fresh install and a long compile run, I rebooted my system only to receive the error "Filesystem couldn't be fixed" on boot. The failure occurs during the startup script on the "Checking root filesystem" step.

"Failed to open the device '/dev/hda4': No such file or directory."

My disk layout looks like:

  /dev/hda1 - /boot, ext2

  /dev/hda2 - swap

  /dev/hda3 - /home, reiserfs

  /dev/hda4 - /, reiserfs

When entering the special root mode for maintenance, I ls'd /dev and noticed that all hda nodes (inluding hda1, hda2....) were MISSING.

This really just happened out of the blue. Could it be a result of upgrading udev? I believe the 2005.1 version was 059, and the latest stable one in portage is 079-r1.

Oddly enough, there are no problems when booting from the livecd and chrooting in. Even more interesting, when not mounting the livecd dev, and looking at the actual hda4 dev,  the hda nodes DO EXIST. I really have no idea why the normal boot process fails to see them.

Any ideas, folks? I'm out of them.

----------

## NeddySeagoon

azdruid,

Whats in your /etc/fstab ?

Errors here will produce some of the symptoms you describe.

e.g. /dev/BOOT, /dev/ROOT and /dev/SWAP  all need to be fixed, as do the filesystem types.

----------

## azdruid

Thanks for the response.

My fstab looks like this:

/dev/hda1 /boot ext2 noauto,noatime 1 2

/dev/hda4 / reiserfs noatime,notail 0 1

/dev/hda2 none swap sw 0 0

/dev/hda3 /home reiserfs noatime,notail 0 1

/dev/cdroms/cdrom0 /mnt/cdrom iso9660 noauto,ro,user 0 0

#/dev/fd0 /mnt/floppy auto noauto 0 0

proc /proc proc defaults 0 0

shm /dev/shm tmpfs nodev,nosuid,noexec

AFAIK, this is correct, because the system ran perfectly on first boot. It was only after that long compile run and etc-update did the problem begin.

Thanks again.

----------

## NeddySeagoon

azdruid,

That looks good. etc-update can make a mess of things, but its not reverted your /etc/fstab.

Did you make a new kernel ?

----------

## azdruid

No, the current kernel was the one built in the livecd chroot upon installation.

I'd build a new one, but gcc doesn't like working in the emergency root environment. /var isn't writeable, you see.

----------

## NeddySeagoon

azdruid,

You can make things writable with

```
 mount -o remount,rw /dev/...  /mount/point
```

You cannot move a mounted filesystem this way.

----------

## azdruid

Well, that made the disk writable. Thanks!

However, how do I resolve the error on boot, and restore the missing hda device nodes?

For example, "fsck.reiserfs /dev/hda4" still fails, complaining that there is no /dev/hda4.

----------

## NeddySeagoon

azdruid,

You can create your missing /dev nodes manually but you should not need to.

Are you running udev or devfs ? 

For kernels after 2.6.12, its must be udev.

Try udevstart and see if your /dev entries come back.

----------

## azdruid

udevstart did it

Thanks a lot! I really appreciate your help.

----------

## rolfrnm

I just did a "emerge --update world" yesterday and now I have the same problem: Entries for

the partitions I use (/dev/sda, /dev/sda1, /dev/sda2, and /dev/sda3) are missing after each

boot. Others (/dev/sda4 ... /dev/sda15) are there...

Going into maintenance mode and typing udevstart creates the needed entries. When I

then Ctrl-D the system reboots, the entries disappear[/quote], and I'm back where I started.

How do I fix this manually? Remember, my system is not fully up yet (e.g. networking

has not been enabled yet), so emergeing things is not really an option.

Thanks for any help,

Rolf

----------

## NeddySeagoon

rolfrnm,

Create your missing /dev nodes as you have been. Then edit /etc/conf.d/rc to change 

```
# UDEV OPTION:

# Set to "yes" if you want to save /dev to a tarball on shutdown

# and restore it on startup.  This is useful if you have a lot of

# custom device nodes that udev does not handle/know about.

RC_DEVICE_TARBALL="no"
```

to 

```
RC_DEVICE_TARBALL="yes"
```

You really should not need this to make the nodes persist over reboots. When you have run your 

```
emerge --sync

emerge -uDN world
```

try reveting this setting to no again. This controls saving your /dev to a tarball during shutdown and restoring /dev from the tarball on startup

----------

## rolfrnm

Thanks for the reply. Your suggestion didn't work, but gave me an idea for something else to try. In /etc/conf.d/rc I did this:

[code]RC_DEVICES="static"[/code]

Now my system boots all the way to kdm. My konsole windows are broken now. I can use the menus to change the background, fonts, etc. But the cursor stays in the top left corner and no text is displayed and it accepts no input. That's probably unrelated to this thread, though.

Doing an emerge -UDN world now. Once that finishes, I'll reset the settings in /etc/conf.d/rc and see what happens. I'll report back.

Rolf

----------

## rolfrnm

 *rolfrnm wrote:*   

> Thanks for the reply. Your suggestion didn't work, but gave me an idea for something else to try. In /etc/conf.d/rc I did this:
> 
> ```
> RC_DEVICES="static"
> ```
> ...

 

So, RC_DEVICES="auto" still doesn't work, and the konsole problem is related. It says "not enough ptys".

For an intermediate solution, I leave RC_DEVICES set to "static", and put the command udevstart into /etc/conf.d/local.start. That makes things work for me, but I don't think that was the intention of the designers...

Rolf

----------

## NeddySeagoon

rolfrnm,

As you say, its not supposed to need that to make it work.

How did you make your kernel?

Does it have hotplug support enabled ?

One of the very first boot messages is 

```
mounting /dev for ... 
```

what do you get in place of the dots ?

What kernel are you using ? 

```
uname -a
```

will show it.

If its 2.6.12 or lower, you have the choice of devfs or udev. Put udev on the kernel line line in grub.conf to help the system make up its mind. devfs was removed at 2.6.13, so its not an issue with later kernels.

----------

## rolfrnm

The kernel is the same one I had from before the update:

```

localhost linux # uname -a

Linux localhost 2.6.15-gentoo-r4.rolf #3 Tue Feb 14 19:25:58 MST 2006 x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux

```

Hotplug support is not enabled. I'm not sure what you mean by "How did you make your kernel?" I can post the config file, if that would help.

When things do not work (RC_DEVICES="auto"), then I get the following output:

```

 * Mounting proc at /proc ...                                            [ OK ]

 * Mounting sysfs at /sys ...                                            [ OK ]

 * Mounting /dev for udev ...                                            [ OK ]

 * Seeding /dev/ with needed nodes ...                                   [ OK ]

 * Setting up proper hotplug agent ...                                   [ OK ]

 * Starting udevd ...                                                    [ OK ]

 * Populating /dev with existing devices through uevents ...             [ OK ]

 * Letting udev process events ...                                       [ OK ]

 * Finalizing udev configuration ...                                     [ OK ]

 * Mounting devpts at /dev/pts ...                                       [ OK ]

 * Activating (possible) swap ...                                        [ OK ]

 * Checking root fielsystem ...                                          [ OK ]

Failed to open the device '/dev/sda2': No such file or directory

 * Filesystem couldn't be fixed :(                                       [ !! ]

```

Rolf

----------

## NeddySeagoon

rolfrnm,

That looks good. Try 

```
grep HOTPLUG /usr/src/linux/.config

CONFIG_HOTPLUG=y

# CONFIG_HOTPLUG_PCI is not set
```

You need CONFIG_HOTPLUG=y the PCI one does not matter

----------

## rolfrnm

 *NeddySeagoon wrote:*   

> rolfrnm,
> 
> That looks good. Try 
> 
> ```
> ...

 

That was it! I rebuilt my kernel with HOTPLUG enabled, got rid of udevstart in my /etc/conf.d/local.start file, set RC_DEVICES="auto" again,

and rebooted.

Thanks a lot for your help NeddySeagoon.

I guess the only disturbing thing is that "emerge -u world" should have somehow told me that I needed to reconfigure my kernel.

Rolf

----------

## vladgrigorescu

I am having this problem.  I have hotplug support in my kernel, and I have tried a udevstart, both with no success.  RC_DEVICE_TARBALL="yes" and RC_DEVICES="auto".

Any other ideas?

----------

## NeddySeagoon

vladgrigorescu,

Maybe you are not running the kernel you think you are ?

Does the date/time from 

```
uname -a 
```

match the date/time you last compiled your kernel ?

----------

## vladgrigorescu

 *NeddySeagoon wrote:*   

> Does the date/time from 
> 
> ```
> uname -a 
> ```
> ...

 

Yes, it's correct.  Maybe I'm just not doing things in the right order...  After I go into maintenance mode, here's what I do:

```
mount -o remount,rw /dev/hda1 /

mknod /dev/hda2 b 3 2

swapon /dev/hda2

udevstart

umount /dev/hda1

logout
```

Should I be changing RC_DEVICE_TARBALL somewhere in there?  It's just always set to "yes", but I don't see where the tarball gets saved upon exiting maintenance mode, all the system says is "Unmounting Filesystems" and "Rebooting".

----------

## vladgrigorescu

Oh well... I tried upgrading udev, downgrading udev, messing with conf.d/rc, everything I could think of and I still couldn't get it to work.  Back to devfs I go...

----------

