# Kernel panic init 1 segfault at 0 ip error 4 in libc-2.19.so

## wdsci

I restarted my computer this afternoon and got a kernel panic about two seconds into the boot process:

```
init[1]: segfault at 0 ip 00007ff10ea3fe05 sp 00007fff7cb49148 error 4 in libc-2.19.so[7ff10e919000+19e000]

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
```

It occurs just after USB device detection, but it doesn't matter whether I have USB devices plugged into the computer or not, the error still occurs.

I was able to boot the computer just fine in the morning, and the only significant change I can remember making to the system since then was installing Astrill (a VPN client, not from the Portage tree, I used the installer downloaded from the company's website but I changed the file locations from /usr/local to /opt). In particular I did not make any changes to the kernel or drivers between when the system worked and when it started giving this error. (Unless Astrill did something, but it shouldn't have, I would think.)

Since I first got the error, I've recompiled my kernel (3.14.14), glibc (2.19.0), sysvinit (2.88-r7), and openrc (0.12.4) (using an Ubuntu live USB because I can't boot the system natively). I also recompiled gcc 4.8.3 but didn't realize that my system profile is set to gcc 4.7.3, so in the absence of any other ideas I guess I might try recompiling GCC 4.7.3 next, not that I really expect it to help... I've also upgraded to a new kernel version (3.16.5) to ensure that the kernel really is compiled from scratch, although I did use `make oldconfig` to configure the new kernel. Still, I think it shouldn't be a bad option in the kernel configuration because I've been using the same configuration for a long time without any trouble. Needless to say, none of this made any apparent difference in the error. It still occurs every time I boot up.

Any idea what's going on and/or how I can fix this?

Here is a photo of the context of the error: https://www.dropbox.com/s/1ls6381k8f23c2h/P_20141113_221430.jpg?dl=0 Let me know if it would help to transcribe any of it to make it more readable.

----------

## steveL

 *wdsci wrote:*   

> 
> 
> ```
> init[1]: segfault at 0 ip 00007ff10ea3fe05 sp 00007fff7cb49148 error 4 in libc-2.19.so[7ff10e919000+19e000]
> 
> ...

 

That looks like a bug in pid1, though it shouldn't really have anything to do with USB devices (ie it shouldn't be affected whatever happens to a process currently talking to USB/device netlink, or trying to modprobe.)

Can't tell from that image; are you using systemd or openrc?

If the latter, then it's something more serious than an application level bug.

----------

## quilosaq

Hi!

The problem seems related to smp. Try disabling in the kernel. 

```
CONFIG_SMP=n
```

.

----------

## wdsci

quilosaq: I tried that, recompiled and reinstalled the kernel, with no change in the behavior. It still panics at the same point in the boot process with the same message (except the sp and ip pointers, of course). There was only one penguin at the top, instead of the usual four, so I know the configuration change took effect.

steveL: I'm using OpenRC, though this is so early in the boot process it hasn't even started yet (I think).

----------

## N8Fear

 *wdsci wrote:*   

> ... There was only one penguin at the top, instead of the usual four, so I know the configuration change took effect.
> 
> 

 

Reenable SMP: without SMP you use just one of your 4 (logical) cores.

Do you use an initrd?

----------

## steveL

Hmm you said that you "also recompiled gcc 4.8.3 but didn't realize that my system profile is set to gcc 4.7.3," and there's been quite a few problems with that specific upgrade (4.7 -> 4.8), to do with C11 and also the ld ordering.

Just to check, can you show us the output (in code tags) of 

```
cat /etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf
```

 (or whatever you have for gcc.)

I have a feeling you need to switch to 4.8 and perhaps rebuild some stuff against the new gcc.

----------

## wdsci

NBFear: yep, I'm going right back to enabling SMP.

I don't use an initrd.

steveL: here's the contents of /etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf:

```
/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/32

/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3

/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/32

/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3
```

If switching to GCC 4.8 fixes the problem, that's great, but then I would wonder, why did I only start having trouble yesterday? I'd been using 4.7 for two months without any issues.

Just to be clear, I mean my GCC system profile has been set to 4.7 for two months (the lifetime of the computer), so everything I've emerged has been compiled with 4.7.

```
# gcc-config -l

 [1] x86_64-pc-linux-gnu-4.7.3 *

 [2] x86_64-pc-linux-gnu-4.8.3
```

----------

## steveL

 *wdsci wrote:*   

> here's the contents of /etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf:
> 
> ```
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/32
> 
> ...

 

That's fine if eg your libc is compiled with 4.8.3, but since the ABI has changed, I'd expect hassle with a glibc compiled against 4.7 running with lookup of gcc-4.8.3 runtime.

 *Quote:*   

> If switching to GCC 4.8 fixes the problem, that's great, but then I would wonder, why did I only start having trouble yesterday? I'd been using 4.7 for two months without any issues.
> 
> Just to be clear, I mean my GCC system profile has been set to 4.7 for two months (the lifetime of the computer), so everything I've emerged has been compiled with 4.7.

 

Yeah that's all good until you emerged 4.8 which has put its runtime libpaths first; though it shouldn't affect C apps in any case, I've no idea what's happening under the hood.

That the fault was in libc-2.19.so makes me think this is the issue; though you should note you'll have to compile a few things against/with the newer gcc. a revdep-rebuild on libstdc++.so.6 would also be advisable, from what I've read so far. (Not sure what the bug # is right now, and a bit tired atm.)

Or you could just swap the order in the file above; though I thought gcc-config is meant to handle that.

----------

## wdsci

I always thought gcc-config worked with symlinks, though maybe I'm mixing it up with eselect kernel? Anyway, it probably doesn't matter. I decided to switch the system compiler to 4.8.3 to keep everything consistent, then I recompiled libtool, gcc, glibc, sysvinit, and openrc, and compiled and reinstalled the kernel. I'm still getting the error.

Also, the revdep-rebuild you suggested didn't turn up anything that needed to be rebuilt.

Could it be indicative of a hardware problem of some sort? I should mention that this laptop dual-boots Windows 8.1, and that OS still works with no problems as far as I can tell, which limits the possibilities for a hardware issue.

----------

## N8Fear

You could try to pass init=/bin/bash (or a statically compiled busybox shell) an see if you can boot into that system at all.

----------

## wdsci

Ah, yeah I should have tried that. But I decided to go ahead and reinstall the system, and I've already formatted the relevant partitions so it's too late. I can't really afford to have this computer nonfunctional for longer than it needs to be.

----------

## wdsci

Just to update: I managed to reproduce the problem (this time being fully prepared to restore the system to a working state), and it turns out Astrill was at fault. The installer adds a custom library to /etc/ld.so.preload, and it seems that library conflicts with init to cause the crash. I was able to fix it by removing the line from /etc/ld.so.preload.

----------

## steveL

Oh man; sorry you had to reinstall.

----------

## wdsci

Yeah... though ironically I didn't have to reinstall; I could have fixed it just by clearing /etc/ld.so.preload. Anyway I got myself into this by letting an unknown program loose on my system, so I can't be too mad.  :Razz: 

----------

## padde

 *wdsci wrote:*   

> Just to update: I managed to reproduce the problem (this time being fully prepared to restore the system to a working state), and it turns out Astrill was at fault. The installer adds a custom library to /etc/ld.so.preload, and it seems that library conflicts with init to cause the crash. I was able to fix it by removing the line from /etc/ld.so.preload.

 

Unbelievable, the same thing just happened to me. Only that I hadn't rebooted in days since I had installed the terrible Astrill client. I would never have made the link between that and my box not booting with "error 4"... Thanks a lot for posting the update, saved my ass  :Smile: 

----------

## wdsci

Cool! Glad to know my days of tinkering were useful to someone else.

I know some people who use Astrill on Ubuntu without any problems, so I guess my next project is to figure out why it works there but not on Gentoo.

----------

## padde

Yep, that would be interesting to know. As I remember the entry was something like "/lib/$LIB/..." - the $LIB was there literally - I guess that has something to do with the issue. This was the only line in the file on my system.

I'm running Astrill's DD-WRT extension now on my router, so luckily I don't need the client on my Linux box anymore. The client on Android seems to be somewhat ok, works decently for me.

By the way, where behind the Great Firewall are you? I'm in Shanghai.

----------

