# segmentation fault - apache

## rastman

Witam, 

After 2 years uptime, now I got such errors:

```
Sep 23 16:01:41 ns310834 grsec: From 65.52.109.198: Segmentation fault occurred at 0000000000000018 in /usr/sbin/apache2[apache2:11255] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:10809] uid/euid:0/0 gid/egid:0/0

Sep 23 16:01:41 ns310834 grsec: From 65.52.109.198: signal 11 sent to /usr/sbin/apache2[apache2:11241] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:10809] uid/euid:0/0 gid/egid:0/0 by /usr/sbin/apache2[apache2:11255] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:10809] uid/euid:0/0 gid/egid:0/0

```

After few segfaults server goes into such state, that it takes about 10 minutes to log in via ssh and restart apache. After that - it works fine for few hours.

other logs:

```
Sep 23 15:29:40 ns310834 apache2[5261]: segfault at 18 ip 00007f8b9d9f0644 sp 00007f8b8b60e640 error 4 in libphp5.so[7f8b9d662000+962000]

```

http://wklej.org/id/598460/

```
Sep 22 18:30:07 ns310834 vim: page allocation failure. order:1, mode:0x20

Sep 22 18:30:07 ns310834 Pid: 28683, comm: vim Not tainted 2.6.32.2-xxxx-grs-ipv4-64 #1

Sep 22 18:30:07 ns310834 Call Trace:

Sep 22 18:30:07 ns310834 [<ffffffff810c52de>] __alloc_pages_nodemask+0x49e/0x730
```

I've tested hdd with fsck, CPU with cpuburn and RAM with memtest. No errors present. 

Now even while compiling php I got:

```

Sep 23 23:55:17 ns310834 conftest[20084]: segfault at 1 ip 00000000004054d4 sp 00007fffc567a1d0 error 4 in conftest[400000+a2000]

Sep 23 23:55:17 ns310834 grsec: From 89.174.34.11: Segmentation fault occurred at 0000000000000001 in /var/tmp/portage/dev-lang/php-5.3.8/work/sapis-build/apache2/conftest[conftest:20084] uid/euid:0/0 gid/egid:0/0, parent /var/tmp/portage/dev-lang/php-5.3.8/work/sapis-build/apache2/configure[configure:20083] uid/euid:0/0 gid/egid:0/0

```

What should I do in order to find what causes segfaults?

----------

## eccerr0r

You definitely have some sort of hardware issue here...

Maybe something related to overheating, hdd power or psu... no idea...  but you definitely do not have a common issue there...

A segfault with an oops is not a good sign.

----------

## rastman

Is there any easy way to debug those segfaults?

Currently it looks so:

```
Sep 26 19:28:59 ns310834 grsec: From 207.46.204.224: Segmentation fault occurred at (null) in /usr/sbin/apache2[apache2:31961] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:23531] uid/euid:0/0 gid/egid:0/0

Sep 26 19:28:59 ns310834 grsec: From 207.46.204.224: signal 11 sent to /usr/sbin/apache2[apache2:31944] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:23531] uid/euid:0/0 gid/egid:0/0 by /usr/sbin/apache2[apache2:31961] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:23531] uid/euid:0/0 gid/egid:0/0
```

Even while compiling php:

```
Sep 26 22:17:16 ns310834 conftest[4169]: segfault at 1 ip 00000000004054d4 sp 00007fff895731b0 error 4 in conftest[400000+a2000]

Sep 26 22:17:16 ns310834 grsec: From 89.174.34.11: Segmentation fault occurred at 0000000000000001 in /var/tmp/portage/dev-lang/php-5.3.8/work/sapis-build/cli/conftest[conftest:4169] uid/euid:0/0 gid/egid:0/0, parent /var/tmp/portage/dev-lang/php-5.3.8/work/sapis-build/cli/configure[configure:4168] uid/euid:0/0 gid/egid:0/0
```

----------

## eccerr0r

You might also want to upgrade your kernel (running 2.6.32?).  Looking at the oops dumps it seems a network driver is causing issues... perhaps there was a bug that someone's maliciously exploiting from the outside... highly unlikely but possible.

Still should check your hardware and make sure it's all clean from dust, cooling properly, etc.

----------

