# root(/dev/sda3 & ext4) mounting read-only! [SOLVED]

## upengan78

Hi,

I am having an issue. My root fs is mounting read only. I don't see a new /var/log/dmesg file generated, so I am wondering how to troubleshoot this issue.

/etc/fstab

```

/dev/sda6                /boot           ext2            noatime 1 2

/dev/sda3                /              ext4            defaults,noatime         0 3

/dev/sda7               none            swap            sw               0 0

shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

/dev/sda4 /nfs ext4 noatime,acl 0 1

/dev/sda8 /nfs1 auto noatime 0 1

```

 df -kh

```
Filesystem      Size  Used Avail Use% Mounted on

/dev/root       165G   19G  138G  12% /

devtmpfs        1.9G     0  1.9G   0% /dev

tmpfs           1.9G  652K  1.9G   1% /run

cgroup_root      10M     0   10M   0% /sys/fs/cgroup

shm             1.9G     0  1.9G   0% /dev/shm
```

mount 

```
/dev/root on / type ext4 (rw,noatime)

devtmpfs on /dev type devtmpfs (rw,relatime,size=1985860k,nr_inodes=496465,mode=755)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)

cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)

openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)

cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)

cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)

freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)

```

One more strange thing is df doesn't show /nfs /nfs1 and /boot in its o/p, howevever mount -a complains those are mounted. Ofcourse those are, I checked if I see my data in mounted partitions, it's there.

mount -a

```
mount: /dev/sda6 already mounted or /boot busy

mount: /dev/sda4 already mounted or /nfs busy

mount: /dev/sda8 already mounted or /nfs1 busy
```

```
fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x958b958b

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *          63    83891429    41945683+   7  HPFS/NTFS/exFAT

/dev/sda2        83891430   312496379   114302475    f  W95 Ext'd (LBA)

/dev/sda3       312498176   662585343   175043584   83  Linux

/dev/sda4       662585344   868386815   102900736   83  Linux

/dev/sda5        83891493    83987819       48163+   b  W95 FAT32

/dev/sda6        83987883    84196664      104391   83  Linux

/dev/sda7        84196728    86172659      987966   82  Linux swap / Solaris

/dev/sda8       158658003   312496379    76919188+  83  Linux

```

```
emerge --info

Portage 2.1.11.50 (default/linux/amd64/13.0, gcc-4.5.4, glibc-2.15-r3, 3.6.11-gentoo x86_64)

=================================================================

System uname: Linux-3.6.11-gentoo-x86_64-Intel-R-_Core-TM-_i7_CPU_860_@_2.80GHz-with-gentoo-2.1

KiB Mem:     3974836 total,   3380420 free

KiB Swap:     987960 total,    987960 free

Timestamp of tree: Tue, 19 Feb 2013 17:45:01 +0000

ld GNU ld (GNU Binutils) 2.22

ccache version 3.1.9 [enabled]

app-shells/bash:          4.2_p37

dev-java/java-config:     2.1.12-r1

dev-lang/python:          2.5.4-r4, 2.7.3-r2, 3.1.5, 3.2.3

dev-util/ccache:          3.1.9

dev-util/cmake:           2.8.9

dev-util/pkgconfig:       0.28

sys-apps/baselayout:      2.1-r1

sys-apps/openrc:          0.11.8

sys-apps/sandbox:         2.5

sys-devel/autoconf:       2.13, 2.69

sys-devel/automake:       1.9.6-r3, 1.10.3, 1.11.6

sys-devel/binutils:       2.22-r1

sys-devel/gcc:            4.4.7, 4.5.4, 4.6.3

sys-devel/gcc-config:     1.7.3

sys-devel/libtool:        2.4-r1

sys-devel/make:           3.82-r4

sys-kernel/linux-headers: 3.6 (virtual/os-headers)

sys-libs/glibc:           2.15-r3

Repositories: gentoo bleeding-edge

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="*"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-O2 -march=native -pipe"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/maven-bin-2.2/conf /usr/share/maven-bin-3.0/conf /var/lib/hsqldb /var/spool/munin-async/.ssh"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"

CXXFLAGS="-O2 -march=native -pipe"

DISTDIR="/usr/portage/distfiles"

EMERGE_DEFAULT_OPTS="--autounmask=n"

FCFLAGS="-O2 -pipe"

FEATURES="assume-digests binpkg-logs ccache config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"

FFLAGS="-O2 -pipe"

GENTOO_MIRRORS="http://mirror.mcs.anl.gov/pub/gentoo/ rsync://mirror.mcs.anl.gov/gentoo/ ftp://mirror.mcs.anl.gov/pub/gentoo/ ftp://gentoo.mirrors.tds.net/gentoo rsync://gentoo.mirrors.tds.net/gentoo http://gentoo.mirrors.tds.net/gentoo"

LANG="en_US.UTF-8"

LDFLAGS="-Wl,-O1 -Wl,--as-needed"

MAKEOPTS="-j8 -l16"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_EXTRA_OPTS="--timeout=50"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/nfs1/var/tmp/portage"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/var/lib/layman/bleeding-edge"

SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"

USE="X acl alsa amd64 aspell berkdb bzip2 cli consolekit cracklib crypt cxx dbus dri gdbm gpm gstreamer gtk hal iconv ipv6 jpeg ldap mmx modules mudflap multilib ncurses nls nptl nptlonly nsplugin openmp pam pcre readline session sse sse2 ssl svg tcpd unicode zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgi cgid" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, USE_PYTHON

```

Kernel .config http://bpaste.net/show/78453/

Just added:

tune2fs -l /dev/sda3

```
tune2fs 1.42.1 (17-Feb-2012)

Filesystem volume name:   <none>

Last mounted on:          /

Filesystem UUID:          83044d5d-bde1-4cc8-9e8c-f31c08ba32d3

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Filesystem flags:         signed_directory_hash

Default mount options:    user_xattr acl

Filesystem state:         clean

Errors behavior:          Continue

Filesystem OS type:       Linux

Inode count:              10944512

Block count:              43760896

Reserved block count:     2188044

Free blocks:              38178990

Free inodes:              10165999

First block:              0

Block size:               4096

Fragment size:            4096

Reserved GDT blocks:      1013

Blocks per group:         32768

Fragments per group:      32768

Inodes per group:         8192

Inode blocks per group:   512

Flex block group size:    16

Filesystem created:       Fri Feb 15 16:00:09 2013

Last mount time:          Tue Feb 19 22:03:49 2013

Last write time:          Tue Feb 19 11:56:29 2013

Mount count:              25

Maximum mount count:      -1

Last checked:             Fri Feb 15 16:00:09 2013

Check interval:           0 (<none>)

Lifetime writes:          25 GB

Reserved blocks uid:      0 (user root)

Reserved blocks gid:      0 (group root)

First inode:              11

Inode size:               256

Required extra isize:     28

Desired extra isize:      28

Journal inode:            8

First orphan inode:       1310769

Default directory hash:   half_md4

Directory Hash Seed:      87a8997c-4ccd-4953-9a09-be419bf40315

Journal backup:           inode blocks
```

ls -al /dev/root

```
ls: cannot access /dev/root: No such file or directory

```

Just added , dmseg command's o/p http://bpaste.net/show/78458/

What do you think is going on?Last edited by upengan78 on Wed Feb 20, 2013 8:21 am; edited 1 time in total

----------

## DONAHUE

in menuconfig:

 *Quote:*   

> CONFIG_DEVTMPFS=y 
> 
> CONFIG_DEVTMPFS_MOUNT=y 
> 
> CONFIG_HOTPLUG=y 
> ...

 

----------

## upengan78

Thanks. Most of the Kernel config options are correct except this -> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"

I can change this config option if it's going to help.

----------

## DONAHUE

no that is the correct path 

just finished  noting that you had all the options, was late seeing your config

# CONFIG_EXT2_FS is not set

# CONFIG_EXT3_FS is not set

CONFIG_EXT4_FS=y

CONFIG_EXT4_USE_FOR_EXT23=y

# CONFIG_EXT4_FS_POSIX_ACL is not set

# CONFIG_EXT4_FS_SECURITY is not set

# CONFIG_EXT4_DEBUG is not set

has been reported to help[, so has 'rootfstype=ext4' in the kernel line of grub.conf

----------

## upengan78

I had CONFIG_EXT4_USE_FOR_EXT23=y and CONFIG_EXT2_FS NOT SET BEFORE. CONFIG_EXT3_FS was set. Still I had problems. So, I added EXT2 config options again and unset CONFIG_EXT4_USE_FOR_EXT23  :Sad: 

I will try to do exactly as in your post.

Also I use lilo, I do use rootfstype=ext4. Here is my lilo.conf

http://bpaste.net/show/78463/

----------

## DONAHUE

I would delete  *Quote:*   

> shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

  from fstab ; the news item for udev 1.97 has  *Quote:*   

> need to verify the fstype for
> 
>   possible /dev line in /etc/fstab is devtmpfs (and not, for example, tmpfs)

 

the need for shm in fstab was eliminated with the inotify/dnotify kernel choices IIRC

check that /var/run is a symlink to /run

the /dev/root that appears in mount and df appoears to be anomalous; i don't have it

----------

## DONAHUE

try, in /etc/rc.conf uncomment rc_parallel="NO"

----------

## upengan78

Thanks. Finished compiling kernel with those EXT4_23 parameters configured. I have commented out rc_parallel="NO" in /etc/rc.conf, commented shm configuration in /etc/fstab.

I am going to reboot within next 10 minutes.

----------

## upengan78

Read-only again.

 # df -kh

```
Filesystem      Size  Used Avail Use% Mounted on

/dev/root       165G   19G  138G  13% /

devtmpfs        1.9G     0  1.9G   0% /dev

tmpfs           1.9G  636K  1.9G   1% /run

cgroup_root      10M     0   10M   0% /sys/fs/cgroup

shm             1.9G     0  1.9G   0% /dev/shm
```

mount

```
/dev/root on / type ext4 (rw,noatime)

devtmpfs on /dev type devtmpfs (rw,relatime,size=1985860k,nr_inodes=496465,mode=755)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)

cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)

openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)

cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)

cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)

freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)

mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).

       It's possible that information reported by mount(8) is not

       up to date. For actual information about system mount points

       check the /proc/mounts file.
```

----------

## upengan78

Forgot one point you mentioned before. 

ls -ald /var/run/

```
drwxr-xr-x 9 root root 4096 Feb 20 00:50 /var/run/
```

No symlink..

Let me create one. 

Update: Added symlink, rebooted, read-only root fs continues..  :Rolling Eyes: 

----------

## upengan78

One of this did the trick finally.

```
host79 ~ # rc-update add root

 * service root added to runlevel default

host79 ~ # rc-update add swap

 * service swap added to runlevel default

host79 ~ # rc-update add udev-mount

 * service udev-mount added to runlevel default

host79 ~ # rc-update add net.lo

 * service net.lo added to runlevel default

host79 ~ # rc-update add udev-mount

```

Above was followed by a reboot and I got the shell access fast this time, hinting some success.  :Very Happy: 

I see df -kh shows /nfs/ nfs1 and /boot in the o/p in addition to existing ones.

 df -kh

```
Filesystem      Size  Used Avail Use% Mounted on

rootfs          165G   19G  138G  13% /

/dev/root       165G   19G  138G  13% /

devtmpfs        1.9G     0  1.9G   0% /dev

tmpfs           1.9G  780K  1.9G   1% /run

shm             1.9G     0  1.9G   0% /dev/shm

cgroup_root      10M     0   10M   0% /sys/fs/cgroup

/dev/sda6        99M   66M   28M  71% /boot

/dev/sda4        97G   46G   47G  50% /nfs

/dev/sda8        74G   48G   20G  72% /nfs1

```

mount shows all the mounted partitions/filesystems in its o/p

```
rootfs on / type rootfs (rw)

/dev/root on / type ext4 (rw,relatime,data=ordered)

devtmpfs on /dev type devtmpfs (rw,relatime,size=1985868k,nr_inodes=496467,mode=755)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)

cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)

openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)

cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)

cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)

freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

/dev/sda6 on /boot type ext2 (rw,noatime)

/dev/sda4 on /nfs type ext4 (rw,noatime,acl)

/dev/sda8 on /nfs1 type btrfs (rw,noatime)

```

Thanks Donahue  :Cool: 

----------

## DONAHUE

good work

is this a new install? 

from the response to the add commands those items were missing and should not have been. thanks I'll add this to my list.

----------

## upengan78

Good morning,

No it wasn't completely new install. There is a story behind this one.

FYI. this was good working machine, with no such problems. With some crazy idea I decided to grow sda3 which is root file system,  online. I changed END sectors(fdisk) for sda3 while system booted and ran resize2fs. That's when kernel crashed. Then booting with gparted I ran couple of fsck's which moved some data to lost+found. I didn't get anything back and lost all 40GB file system then. I recreated ext4 on it and rsynced everything over ssh while gparted was running from other working gentoo system. Omitted  dev/ /mnt/ /sys.. 

I couldn't boot system with other system's lilo and its kernel, so I did install a newer kernel and recovered some things off of lost+found dir(via test disk image before ext4fs formatting). I had installed kernel in ch-rooted env which itself was pain for me to get it to work even with 64bit kernel sysrescue. I chrooted with /bin/tcsh shell which I am not familiar much. bash was failing due libtermcap.so I wonder why! This was one of the causes system didn't boot after kernel compilation but I had taken care of it before posting my issue here.

Anyway, I wonder why some services went missing for boot or default runlevel. I did come across a thread on gentoo forums where a user added 'root' to default to resolve similar problem.

In the end, I guess this was one of the things I had to check..

Just thought I'd post my rsync command. It's run from a good system running gentoo.

```
rsync -avz --stats --progress -e "ssh -c arcfour" --exclude-from=file / root@<problem machine's IP>:/sda3/
```

cat file

```
dev

sys

mnt

proc

nfs

nfs1

boot

run

```

----------

