# Not able to bring the server up.

## manu_leo

Hi Experts, I am completely out of ideas now. One of our Gentoo server seems to have gone down and I am not sure how to recover it.  The prompt I am getting is 

 *Quote:*   

> (none) ~#

 

There is no hostname that is coming up. If I run route or ifconfig commands, I get the below error :-

```
route -n

/proc/net/route: No such file or directory

INET (IPv4) not configured in this system.

ifconfig

Warning: cannot open /proc/net/dev (No such file or directory). Limited output.

```

If I type df -h to see the mount I get arrays of missing file system errors.

```
This is (none).unknown_domain (Linux x86_64 3.17.7-gentoo) 19:33:04

(none) login: root

Password:

[ 52.405911] random: nonblocking pool is initialized

(none)  -  It  df -h

df: m/dev/shms: Ho such file or directory

df: m/sys/fs/cgroupm: Ho such file or directory

df: m/sys/fs/cgroup/openrcm: Ho such file or directory

df: m/sys/fs/cgroup/cpusetm: Ho such file or directory

df: m/sys/fs/cgroup/cpum: Ho such file or directory

df: m/sys/fs/cgroup/cpuacctm: Ho such file or directory

Filesystem Size Used Avail Usex Mounted on

/dev/sdal 40G 9.1G 29G 25x /

devtmpfs 12G 0 12G Ox /dev

tmpfs 40G 9.1G 29G 25x /run

/dev/sda3 40G 9.1G 29G 25x /var

(none) - It  /etc/init.d/sshd status

/sbin/runscript: error while loading shared libraries: librc.so.1: cannot open s

hared object file: Ho such file or directory

(none)  -  
```

I tried booting from Live CD and I can see all the info which I configured before like the hostname, fstab entries. I did a emerge --sync but no go. 

Could you please help me in getting this fixed. Appreciate all your help in advance.

----------

## The_Great_Sephiroth

I would boot System Rescue CD or similar and check your disk! First check SMART to see if it has tripped.

```

smartctl -a /dev/sda

```

If the SMART data has passed, check your file-systems.

```

fsck /dev/sda1

fsck /dev/sda3

```

If those pass, I have no clue! To me it looks like a dying drive or bad file-system. If that is not the case, then maybe some core configuration file got messed up. I am sure a guru will be along shortly to be a tad more insightful.

----------

## manu_leo

Thanks for looking into this. I ran fsck before and all looks clean. However I dont think smartctl utility is installed on it.

```

(none)  -   [ 27.449033] random: nonblocking pool is initialized

(none) -  cd /sw/

(none)  /sw   /etc/init.d/sshd status

/sbin/runscript: error while loading shared libraries: librc.so.1: cannot open s

hared object file: Ho such file or directory

(none)  /sw  smartctl -a /dev/sda

-bash: smartctl: command not found

(none)  /sw   fsck /dev/sdal

fsck from util-linux 2.24.1

e2fsck 1.42.10 (18 -May -2014)

/dev/sdal: clean, 368057/2621440 files, 2584344/10485760 blocks

(none)  /sw   fsck /dev/sda3

fsck from util-linux 2.24.1

e2fsck 1.42.10 (18 -May -2014)

/dev/sda3: clean, 144859/1048576 files, 1260033/4194304 blocks

(none)  /sw   fsck /dev/sda4

fsck from util-linux 2.24.1

e2fsck 1.42.10 (18 -May -2014)

/dev/sda4: clean, 73219/30048256 files, 66456657/120192768 blocks
```

----------

## NeddySeagoon

manu_leo,

It looks like root is staying read only.  Can you pasfebin dmesg from a failed boot?

----------

## manu_leo

Thanks Neddy.This is what I have from dmesg

```
[ 3.6952211 input: Avocent USB Composite Device -0 as /devices/pci0000:00/0000

:00:1d.O/usb5/5-2/5-2:1.0/0003:0624:0248.0001/input/input4

[ 3.6954751 hid -generic 0003:0624:0248.0001: input,hidraw0: USB HID v1.00 Key

board [Avocent USB Composite Device -0] on usb-0000:00:1d.0-2/inputO

[ 3.7011631 input: Avocent USB Composite Device -0 as /devices/pci0000:00/0000

:00:1d.0/usb5/5-2/5-2:1.1/0003:0624:0248.0002/input/input5

[ 3.7014891 hid -generic 0003:0624:0248.0002: input,hidrawl: USB HID v1.00 Mou

se [Avocent USB Composite Device -0] on usb-0000:00:1d.0-2/inputl

[ 4.238268] scsi 3:0:0:0: Direct -Access iDRAC LCDRIVE 0323 PQ

: 0 ANSI: 0 CCS

[ 4.238547] sd 3:0:0:0: Attached scsi generic sg3 type 0

[ 4.239637] scsi 4:0:0:0: CD-ROM iDRAC Virtual CD 0323 PQ

: 0 ANSI: 0

[ 4.242062] sd 3:0:0:0: [sdb] Attached SCSI removable disk

[ 4.347774] sr 4:0:0:0: [srl] scsi3-mmc drive: Ox/Ox cd/rw caddy

[ 4.348124] sr 4:0:0:0: Attached scsi CD-ROM srl

[ 4.348284] sr 4:0:0:0: Attached scsi generic sg4 type 5

[ 4.350839] scsi 4:0:0:1: Direct -Access iDRAC Virtual Floppy 0323 PQ

: 0 ANSI: 0 CCS

[ 4.351213] sd 4:0:0:1: Attached scsi generic sg5 type 0

[ 4.353393] sd 4:0:0:1: [sdc] Attached SCSI removable disk

[ 27.449033] random: nonblocking pool is initialized

[ 1250.420809] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts:

(null)

(none)  -
```

----------

## manu_leo

Actually Neddy, its not a failed boot but the serves comes up when I issue - shutdown -r now with the prompt as below 

(none) ~#

----------

## manu_leo

Is there a way this can be fixed or repaired using a LiveCD without loosing any of the stuff/data ?

Thanks and appreciate all your help.

----------

## manu_leo

I did a mount -o rw,remount / and now I am able to create directories and delete them. But the issue is still the same - Not able to restart any service, df -h gives the same error and so on.

----------

## NeddySeagoon

manu_leo,

That looks normal but I would have liked it all.

With System Rescue CD, mount your root at /mnt/gentoo.  Do not chroot.

Edit /mnt/gentoo/etc/rc.conf to be sure the following options aare set.

```
rc_parallel="NO"

rc_interactive="YES"
```

Reboot normally and press I when openrc starts.  There may even be a

prompt.

Let each service start in turn and note any error messages you get along

with the service that generated the errors.

Once we know the cause of the problem we can fix it.  No promises about data loss yet, since we cannot rule out hardware issues.

----------

## manu_leo

I have mounted /dev/sda1 on /mnt/gentoo, however I dont see the /mnt/gentoo/etc/rc.conf. Do I need to manually create it and then put these options under this fine ? 

Please suggest.

----------

## NeddySeagoon

manu_leo,

If you are using OpenRC the file should exist and start with

```
# Global OpenRC configuration settings

# Set to "YES" if you want the rc system to try and start services

# in parallel for a slight speed improvement. When running in parallel we

# prefix the service output with its name as the output will get

# jumbled up.

# WARNING: whilst we have improved parallel, it can still potentially lock

# the boot process. Don't file bugs about this unless you can supply

# patches that fix it without breaking other things!

#rc_parallel="NO"

rc_parallel="NO"
```

Maybe sda1 is not your root filesystem?

If you use systemd, I can't help.  You should probably start a new thread.

----------

## manu_leo

Neddy, I am using openrc.

I did a chroot , and then ran locale-gen. It gave me an error as 

```
/usr/sbin/locale-gen: line 17: /etc/init.d/functions.sh: No such file or directory
```

When I went under /etc/init.d, and did an ls , the functions.sh file was missing

 *Quote:*   

> (chroot) livecd /etc/init.d # ls
> 
> busybox-ntpd     
> 
>  git-daemon         
> ...

 

I then went under /etc/runlevels/sysinit to check the status and the softlinks for devfs , dmesg , sysfs and tmpfiles.dev are in RED.

```
(chroot) livecd # cd /etc/runlevels/sysinit

ls -lth

total 0

lrwxrwxrwx 1 root root 16 Dec  4  2014 udev -> /etc/init.d/udev

lrwxrwxrwx 1 root root 22 Dec  4  2014 udev-mount -> /etc/init.d/udev-mount

[color=darkred]lrwxrwxrwx 1 root root 17 Dec  4  2014 devfs -> /etc/init.d/devfs

lrwxrwxrwx 1 root root 17 Dec  4  2014 dmesg -> /etc/init.d/dmesg

lrwxrwxrwx 1 root root 17 Dec  4  2014 sysfs -> /etc/init.d/sysfs

lrwxrwxrwx 1 root root 24 Dec  4  2014 tmpfiles.dev -> /etc/init.d/tmpfiles.dev[/color]

lrwxrwxrwx 1 root root 29 Dec  4  2014 kmod-static-nodes -> /etc/init.d/kmod-static-nodes
```

There is something which is broken, but I am not sure what it is .  Please suggest some way as I am running out of ideas.

Thanks in advance.Last edited by manu_leo on Thu Jul 16, 2015 5:58 am; edited 1 time in total

----------

## manu_leo

I booted from the livecd and here is how the mounts look like 

```
/dev/sda1 on /mnt/gentoo/

/dev/sda3 on /mnt/gentoo/var

/dev/sda4 on /mnt/gentoo/home

proc on /mnt/gentoo/proc

/sys/ on /mnt/gentoo/sys

/dev/ on /mnt/gentoo/dev

```

So I am using /dev/sda1 as my root partition. 

Under /mnt/gentoo where previously I did an untar to the stage3 ball, I see all the directories intact, so nothing is lost, its just the repair/scan I believe we need to do to get the system back to normal.

----------

