# [Solved] Root Filesystem Read-only / Wrong boot

## phobos13013

Wow, this one is super weird....

I was briefly on linux-4.0.9 then got the upgrade for linux-4.1.12.  Did the usual, copied over my old kernel config, built it in the new kernel, copied it to boot and that was that.  I did edit my fstab some but I see no major issues there.  I continued to use the system but eventually it started lagging after a few days so i restarted.... Now when I start into my new kernel, the filesystem is readonly and I can do nothing as root or normal user.  Also, my boot directory is jacked.  None of my old kernels are there nor grub the only thing is some old kernel files from 3.1.0 which I havent used since I first turned this box back on about 1.5 years ago....

How did i overwrite my previous /boot with files from over a year and half ago and how can I get it back?

----------

## charles17

Sounds like you forgot to mount /boot before make install modules_install?

----------

## phobos13013

Good thought, and I am not denying this is not a super basic issue... but I am still not so certain.  That said, I can confirm that /boot is still alive and well just not properly mounted (mounting /dev/sda1 shows my correct kernel files).  I did just notice that the whole netmount thing that was in the news awhile back was not done, so I added that to boot.  But it was not the solution.  

I was about to post my fstab but borked the syntax and now my fstab is dead... is there a place I get a template version or do I have to make it from scratch?

----------

## phobos13013

I rebuilt the fstab (posted here: http://bpaste.net/show/020ca9f48633), assuming this is correct, but maybe I am not being clear on the primary issue here...., it is that my system is booting read-only and I do not know why.  During the boot sequence, I am being told that:

 *Quote:*   

> /run/lvm/lvmetad.socket: connect failed: No such file or directory
> 
> WARNING: Failed to connect to lvmetad.  Falling back to internal scanning
> 
> Checking local filesystems ...
> ...

 

the drive is determined to be inconsistent then takes me to the login page

Of course I am certain this is a symptom and not the ailment.

In addition to this, when I do login (read-only) my /boot drive is not there but instead apparently a /boot that is written to the root directory.  Why is my boot drive not getting mounted?  It historically was in the past....

----------

## charles17

 *phobos13013 wrote:*   

> I rebuilt the fstab (posted here: http://bpaste.net/show/020ca9f48633), assuming this is correct, 

 IMHO the /dev/proc line should be removed. See https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#About_fstab

 *phobos13013 wrote:*   

>  *Quote:*   /run/lvm/lvmetad.socket: connect failed: No such file or directory
> 
> WARNING: Failed to connect to lvmetad.  Falling back to internal scanning 

 

Are you using LVM? And shouldn't the fstab entries for LVM look like https://wiki.gentoo.org/wiki/Sakaki's_EFI_Install_Guide/Final_Preparations_and_Reboot_into_EFI#Setting_up_the_Mountpoint_Tables?

----------

## phobos13013

Well I can comment out proc pretty easy, dont think that is my problem.  And I can try to follow this guide for EFI... I hadnt been using it before although I was using LVM with no problems; why is this a problem all of a sudden?  I am not really sure what the EFI settings mean, though so do not feel I understand this guide.  For instance, what is this mapper directory for?  Is there any other explanation for all this other EFI stuff?  I am not using an initiramfs and do not really know how or why to set it up.  So any guidance towards this is appreciated.

----------

## charles17

Forget about EFI.  It is LVM where I was trying to point your attention.

But maybe your fstab would also work for LVM.  Did you set up for LVM at all?

----------

## phobos13013

Well, I have it in my kernel, I guess I am not sure what it is that I need to do to set up for LVM in the first place.  The thing is, I have had this same configuration for boot on this system since at least July but upgraded a new kernel and now I am stuck with a non-mounting boot and a read-only root and there is no clear reason why.  If you can point me to a way to configure LVM (which I have never done before), its appreciated.

----------

## krinn

/dev/sda1	/boot		ext2	defaults,noauto	0 2

the noauto mean you will have not /boot mount until you ask it to mount.

that's normal and you should keep it like that, your previous fstab wasn't using it if it was mounting /boot always.

as /boot partition isn't mount, what you see in /boot comes from your /boot subdirectory of your /

so what you see in it, is that times ago, you forget to mount /boot and install a kernel and it was copied there.

if your system works without initram, it mean your system doesn't specially need something to mount /

if your / is mount read only, you have some issue.

you can 

```
mount -o rw,remount /
```

to allow you to fix the issue. But as you probability guess, you need to find first, what is your issue  :Smile: 

per example, lvm is only an issue if you use lvm, something i don't think you do.

And if per example, your kernel lack tmpfs, you cannot mount /run, udev will not works, lvm will complain /run/lvm/lvmetad.socket doesn't exists because nobody could write in /run if /run is not on a tmpfs but in your / that is also ro to makes things worst.

If you are lost with what could be your issue, look in dmesg, because many clues should be there.

----------

## phobos13013

This I think has gotten to the root of the matter, PUN INTENDED!

I had seen this remount command somehwere in the forums but hadnt actually tried it yet and unfortunately time is short for the rest of my American day so wont be able to test but I have faith in that temporary fix.

I will explore dmesg and do the basic diagnostic also, but I think you are right LVM complaints are a symptom and not a disease.  The question is where is the disease, this I am afraid is above my paygrade... which for Linux is 0.  Any other ideas or info is appreciated until my next post which is inevitably forthcoming.

----------

## phobos13013

Well, I have finally gotten some time to do some testing.

You are correct in assuming that previously I mounted /boot unabashedly, I hope that hasnt gotten me in to trouble.  From now on I will definitely be locking that down.  You are also correct that I have a bum /boot from when I was trying to reinstall a kernel to it when I first turned this box back on a few years ago.

The mount -o command worked but none of my major services are working so this just won't do.  One weird thing I will say is that when I run the desktop manager after the remount happens is a desktop that is typically setup for a sub instance of X I had setup for vnc and such activities.  It is not my primary user desktop.

You actually may be right that I am not using lvm.... but then where did it come from?  My understanding is that it is used to access partitions above sda3, am I incorrect here?

regarding /run and tmpfs, at the start of openrc, I get the following info, not sure if it means anything.

 *Quote:*   

> Mounting /run ...
> 
> /run/openrc: creating directory
> 
> /run/lock: creating directory
> ...

 

The stuff about lvm not working and root inconsistencies follows.

My dmesg is found here... http://bpaste.net/show/61914cbdd3d9

Not seeing anywhere to start; clearly VFS is reporting that root gets mounted ro pretty unceremoniously... There is clearly an ACPI hiccup there but do not see how this can lock me out of my system.  So the only thing left to say is HELP!

----------

## Syl20

 *phobos13013 wrote:*   

> Did the usual, copied over my old kernel config,

 

So you didn't make oldconfig ? All the new or renamed options don't appear in your pasted .config, so I suppose the building process is setting for these options default values that may be different that you expect.

----------

## charles17

 *phobos13013 wrote:*   

> My dmesg is found here... http://bpaste.net/show/61914cbdd3d9

 I don't see your lvmetad stuff there.

 *phobos13013 wrote:*   

>  *Quote:*   /run/lvm/lvmetad.socket: connect failed: No such file or directory
> 
> WARNING: Failed to connect to lvmetad.  Falling back to internal scanning 

 Did that come before the grub prompt or after?

----------

## krinn

```
Command line: BOOT_IMAGE=/vmlinuz-4.1.12-gentoo root=/dev/sda5 ro
```

Hmmm, do you know what "ro" could mean?

----------

## phobos13013

Whoa!  All good info folks, I appreciate everyone thinking this through.  I have been looking at some or all of these issues....  Please see below for individual responses:

CneGroumF:  I have evolved my kernel upgrade process over the years even within the last few years... My current process is to just do a cp of the .config from the old linux directory to the new kernel version then do an eselect to change the kernel version then run make and build the kernel again from the previous .config which I know is valid.  I then copy the bzImage to /boot renaming it to linux-version convention.  Then I remake grub.cfg.  I have no doubt there are all sorts of issues and problems with doing it this way but really it has worked for me for the last few version upgrades (since at least the last few 3.X I was running); I have thought about make oldconfig but just have never run it.  I do this, though, for exactly the reason you mention.  Awhile ago I rebuit a kernel without copying the oldconfig and had a hell of a time trying to figure out why my X was bad my net was bad and all that.

charles17:  Thanks for sticking with me here; yeh good point I do not see any mention of lvm... which doesnt surprise me since it couldnt work on boot and the complaint from openrc is that it cant find lvm.  I do not know how to confirm but for some reason when I did the emerge that brought in the new kernel, i want to say that an lvm upgrade was in there... but it could just be priming bias from the current conundrum.  I am musing at this point actually.  Anyway, the lvm complaint is after grub, its during the drive mouning after openrc picks up the initscript.

krinn:  I know what you are thinking and I had this same idea, since I did notice that... But from what I can tell, that is normal behavior for my system.  First looking back at all my grub.cfg (from past kernels also), this is also the case on the boot script line.  My understanding from the research i did after seeing that is the initial action of the boot script is to mount all drives ro until later in the script where it then makes them rw, it has something to do with timing of services and such... I am not an expert on any of this so may be talking out of my lower regions but they are probably more informed than my upper regions anyway.  Whatever the made up excuse, the fact is I went into my grub.cfg and removed any mention of ro after the script line for the boot location and it had zero effect.  That said, now that I think about it, maybe that line isnt the same as the grub.cfg line and there is somewhere else that is telling this thing to boot ro... I just do not understand why it would have happened all of a sudden.. and where for that matter the thing is that would say to do that.

Any other suggestions are appreciated.

----------

## charles17

 *phobos13013 wrote:*   

> I am musing at this point actually.  Anyway, the lvm complaint is after grub, its during the drive mouning after openrc picks up the initscript.

 

What about doing an interactive boot (have rc_interactive="YES" in /etc/rc.conf) in order to see which of the init scripts does this?

----------

## phobos13013

I am bumping this looking for some fresh ideas on this... I have an interactive boot set currently, but dont know what if any of the services are contributing to this.  Not sure which services to add or deny to triage it.  So far what I have tried comes out the same, read-only.

----------

## charles17

 *phobos13013 wrote:*   

> I have an interactive boot set currently, 

 

Press the i button very early at boot, so it should prompt you for starting or skipping each service (initscript) separately.

----------

## krinn

 *phobos13013 wrote:*   

> 
> 
> krinn:  I know what you are thinking and I had this same idea, since I did notice that... But from what I can tell, that is normal behavior for my system.

 

If you ask kernel to boot ro, it will boot ro, it's only people that ask a first boot ro, then boot gets remount to rw later (that's for people that remount their system because of initram or temporary boot usage that will be remount later)

So if your root is not remount, it should remains ro (edit: i have checked, and openrc service will remount any ro to rw, check rc-update | grep root and make sure it is start at boot, and look if it works)

 *phobos13013 wrote:*   

>  That said, now that I think about it, maybe that line isnt the same as the grub.cfg line and there is somewhere else that is telling this thing to boot ro

 

Command line can be force thru kernel options. So even removing ro from grub may not change it.

```
grep "CMDLINE=" /usr/src/linux/.config 

CONFIG_CMDLINE="clocksource=hpet"

grub.cfg: kernel /vmlinuz root=/dev/sdb2

[    0.000000] Kernel command line: clocksource=hpet root=/dev/sdb2

```

You can either clean grub config and check dmesg to see if ro has really disappears, or check your kernel CMDLINE to see if its enable and have ro set.

If you use an initramfs, make sure it remount it rw.

----------

## phobos13013

 *Quote:*   

> So if your root is not remount, it should remains ro (edit: i have checked, and openrc service will remount any ro to rw, check rc-update | grep root and make sure it is start at boot, and look if it works)
> 
> 

 

So this is the rub... yes rc-update has root on boot and it always has, so not sure why my most recent update would change any of this.  BUT no it is not being remount as rw for some reason later down the sequence.  I did an interactive mount and it is certainly being requested to mount then remount but its not happening, nonetheless.

 *Quote:*   

> You can either clean grub config and check dmesg to see if ro has really disappears, or check your kernel CMDLINE to see if its enable and have ro set. 

 

Well if this is the solution, I think im too dumb to implement this without some additional info.  I run the grep command and this is returned, CONFIG_FB_CMDLINE=y

dmesg does not seem to be reporting a remount and i do not have an initiramfs.  Something changed in terms of my boot expectation and I am not clear when, how or why.

----------

## krinn

 *phobos13013 wrote:*   

> I did an interactive mount and it is certainly being requested to mount then remount but its not happening, nonetheless.

 

Have an rc.log to see the reason?

 *Quote:*   

> 
> 
> dmesg does not seem to be reporting a remount and i do not have an initiramfs.  Something changed in terms of my boot expectation and I am not clear when, how or why.

 

You will not see a remount in dmesg, it will appears as a mount info, the kernel mount boot, it doesn't remount it itself. that's openrc that do that.

You may check dmesg still to see if the mount options were properly tried, some limits may prevent a remount or bad option.

Speaking of bad options i've look at your fstab, and i could tell you that noatimes is not noatime

----------

## NeddySeagoon

phobos13013,

post your dmesg after root comes up read only.

I suspect that when the root fs is checked, it may have some damage that cannot be fixed on the fly.

-- edit --

Logical Volume Manager is not used to access Logical Partitions.  The naming is an unfortunate coincidence.

A Logical Volumes are like partitions but they can be dynamically resized, providing the filesystem supports the operation too.

Unlike a partition, where all the blocks are in one contiguious chunk on the disk, a logical volume need not be in one piece and can even be spread over several disks.

Logical Volume manager hides all this from you.

In short, if you didn't set up your install on logical volumes, you are not using logical volume manager.

----------

## phobos13013

Still truding along here... but I think I had a memory flash that is the cause of what got here, more on that below but let me provide responses first:

krinn:

I am not getting rc.logged right now unless its in /var/log/messages; I cannot confirm this because it is bloated to GBs in size cannot open, will have to scrub it i guess.  I am posting dmesg for Neddy below.

NeddySeagoon:

I have the old dmesg from when I first got this error towards the beginning of the thread (see the only other bpaste link).  Go ahead and compare it to the most recent one here:  https://bpaste.net/show/91c716edf624.  But I think you are right, it is seeing some kind of error to the root drive which is triggering e2fsck but it cannot resolve it.  Before it was getting through the whole check, but now it panics at like 18% and continues with the boot.  Is there a way to fix this my transfering my root partition to another one and try to boot to that one?  

I have changed some kernel and config settings, but nothing has helped.  There is one last possiblity here.  I recently installed a file manager that I have not used before as a test.  I was transferring files to backup and then deleteing them from the hdd.  There are some partitions on here that I used to not be able to modify contents of and I am wondering if the file manager was able to exceed its permissions which borked the drive.  If this is the issue, is there anything that can be done to salvage this short of formatting, if that?

----------

## krinn

About my query for rc.log, the file is alone and made up by openrc if you have enable the option.

That's in /etc/rc.conf

```
# rc_logger launches a logging daemon to log the entire rc process to

# /var/log/rc.log

# NOTE: Linux systems require the devfs service to be started before

# logging can take place and as such cannot log the sysinit runlevel.

#rc_logger="YES"

```

I'm seeing i didn't think about it, but of course creating that /var/log/rc.log on a ro fs to see where rc report problems is chiken/egg problem

It might be possible to gave it some tmpfs location to bypass the limitation, but i never tried something like that myself.

As usual, solution always comes when NeddySeagoon comes in, i really start to think it's not coincidence  :Smile: 

```
[code][  266.923622] EXT4-fs (sda5): warning: mounting fs with errors, running e2fsck is recommended[/code]
```

Didn't saw it because it appears at end of dmesg (and really late), that must be you are remounting it rw by hands.

Use a livecd and try fix it from there as from your fstab, sda5 hold your /

----------

## steveL

 *phobos13013 wrote:*   

> Well I can comment out proc pretty easy, dont think that is my problem.  And I can try to follow this guide for EFI... I hadnt been using it before although I was using LVM with no problems; why is this a problem all of a sudden?  I am not really sure what the EFI settings mean, though so do not feel I understand this guide.  For instance, what is this mapper directory for?  Is there any other explanation for all this other EFI stuff?  I am not using an initiramfs and do not really know how or why to set it up.  So any guidance towards this is appreciated.

 

If you're using LVM without initramfs, you probably need these settings worked out by saellaven in /etc/lvm/lvm.conf (if you don't already):

```
obtain_device_list_from_udev = 0 

udev_sync = 1 

udev_rules = 0
```

I wouldn't worry about the warning for lvmetad; though i do recall thinking it seemed like one we might want as a dependency in any eventuality, it's not strictly needed, and is for networked filesystems iirc (could well be wrong on its actual purpose; was a while ago.)

WRT logging output in early init, you can try the same thing I did with lvm, for a previous bug.

Though it seems your issue may be a fsck required; touch /forcefsck if you had root.. ;)

Sorry I haven't been updating that thread; was thinking to leave Gentoo (again) thanks to developer idiocy (again.)

Forums users persuaded me otherwise, just by being themselves (again.)

----------

## NeddySeagoon

phobos13013,

Your root is /dev/sda5 - no LVM is in use.

As krinn says, try booting a liveCD/USB and running fsck on /dev/sda5.

root is checked to be clean every boot and fixed if possible.

Keep a backup of /home before you run fsck, since fsck can make a bad situation worse.

All it does is make the filesystem data structures on disk self consistent.  

That says nothing about any user data that may be on the filesystem.  

/home is where your user data is. Everything else can be recreated.

If you want to be a bit more adventureous, you can try to remount root using a backup superblock.  

If a corrupt primary superblock is the issue, this will fix it.  If not you will still see the error in dmesg.  

Read 

```
man mount
```

find the options for ext2, they apply to ext4 too.

See the sb=n entry.

----------

## phobos13013

Hello all!  Sorry for the delay in response but I was called out of town unexpectedly and was not able to attend to this until today.  But during the time away it gave me lots of opportunity to think about the issue.  In addition, I can confirm NeddySeagoon you did it again; you are the finest of all Scots (no true Scotsman my ARSE!).  I kid!

But seriously folks, I did an e2fsck on the disk unmounted and it found innumerable issues.  Not sure if its a drive thats on the way out (it is super old), or if its that filemanager i was using to shuffle lots of files around, but anyway, I went the simple route and just did e2fsck since I didnt want to invest any more time into the superblock method and luckily it fixed all isues with no apparent irreprable harm to the fs.  that said I did save /home just in case.

Consider this HAPPILY CLOSED!

----------

