# [SOLVED] Apache/PHP Segfault with Gallery

## mryoung_fr

Hi there,

I'm currently experiencing problem with Gallery...

To give a summary of my config, I'm currently using Gentoo Hardened, with Apache2.2 and PHP 5.3 (. :Cool:  ... I've got several web apps on this server (including drupal, roundcube, squirrelmail, phpmyadmin, phpldapadmin, and such) which are running without any problem...

I also have several gallery2 webapps installed, but since sometimes (I suppose since upgrading to php 5.3, but not really sure), i'm experiencing segfault with apache, stopping my web service completly until apache kill and restart ...

This problem is occuring randomly, and seems to be related to thumbnails creation or read (occurs when browsing album, displaying thumbnails ... time to time, some thumbnail doesn't display, and if I look at my log, I can see Apache SegFault)...

I suppose this problem is php related, but as it occurs only when using Gallery2, I already made a forum post on Gallery2 forums... But maybe someone have gone through this issue here ...

Gallery config:

```

Gallery version = 2.3.1 noyau 1.3.0.1

API = Noyau 7.54, Module 3.9, Thème 2.6, Intégré 1.5

PHP version = 5.3.8-pl0-gentoo apache2handler

Serveur Web = Apache

Base de données = mysqli 5.1.56-log, lock.system=flock

Boîtes à outils = Exif, ImageMagick, jpegtran

Accélération = full/21600, partial/21600

Système d'exploitation = Linux sdbsv001 3.0.4-hardened-r4 #2 SMP Sun Nov 6 02:32:03 CET 2011 x86_64

Thème par défaut = matrix

gettext = activé

Langage = fr_FR

Navigateur Web = Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0

Lignes dans la table GalleryAccessMap = 517

Lignes dans la table GalleryAccessSubscriberMap = 5052

Lignes dans la table GalleryUser = 14

Lignes dans la table GalleryItem = 5052

Lignes dans la table GalleryAlbumItem = 37

Lignes dans la table GalleryCacheMap = 301

```

Apache SegFault:

```

Dec 27 07:37:58 kernel: grsec: From 10.1.1.1: signal 11 sent to /usr/sbin/apache2[apache2:25106] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:25093] uid/euid:0/0 gid/egid:0/0 by /usr/sbin/apache2[apache2:25150] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:25093] uid/euid:0/0 gid/egid:0/0

```

Some times (not ervery time), just before segfault, i can see in user.log ...

```

Dec 27 00:42:38 apache2: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 120150874768 bytes) in Unknown on line 0

```

Apache / PHP / Gallery are installed with the following Gentoo USE Flags:

```

[ebuild   R    ] www-servers/apache-2.2.21-r1  USE="ldap ssl suexec

threads -debug -doc (-selinux) -static" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_http rewrite setenvif speling status unique_id userdir usertrack vhost_alias -asis -auth_digest -authn_dbd -cern_meta -charset_lite -dbd -dumpio -ident -imagemap -log_forensic -proxy_ajp -proxy_balancer -proxy_connect -proxy_ftp -proxy_scgi -reqtimeout -substitute -version"

APACHE2_MPMS="-event -itk -peruser -prefork -worker" 0 kB

[ebuild   R    ] dev-lang/php-5.3.8  USE="apache2 berkdb bzip2 cli crypt

ctype curl exif fileinfo filter ftp gd gdbm hash iconv imap intl ipv6 json ldap mysql mysqli nls pdo phar pic posix readline session simplexml snmp soap sockets spell sqlite ssl threads tokenizer truetype unicode xml xmlreader xmlwriter xsl zip zlib (-adabas) -bcmath (-birdstep) -calendar -cdb -cgi -cjk -curlwrappers -db2 (-dbmaker) -debug -doc -embed (-empress) (-empress-bcs) -enchant (-esoob) -firebird -flatfile -fpm (-frontbase) -gd-external -gmp -inifile -interbase -iodbc -kerberos -kolab -ldap-sasl -libedit -mhash -mssql -mysqlnd -oci8 -oci8-instant-client -odbc -pcntl -postgres -qdbm -recode -sapdb -sharedext -sharedmem (-solid) -sqlite3 -suhosin (-sybase-ct) -sysvipc -tidy -wddx -xmlrpc -xpm" 0 kB

[ebuild   R    ] www-apps/gallery-2.3.1  USE="ffmpeg gd imagemagick

mysql raw sqlite unzip vhosts zip -netpbm -postgres" 0 kB

```

As i googled a lot, i made a lot of test:

. PHP

 -> Disabled zend GC: Still SegFaulting

 -> Compiling without PIC: Still SegFaulting

 -> Compiling without berkdb: Still SegFaulting

 -> Compiling without berkdb & PIC: Still SegFaulting

. Gallery

 -> Disabling most of modules (rewrite): Still SegFaulting

 -> Using only GD: Still SegFaulting

 -> Using only ImageMagick: Still SegFaulting

. Apache

 -> Trying to enable SetEnv Broken (gallery config): Still SegFaulting

Finally, i tried to upgrade to manually Gallery3 (as the architecture is completly different ... but I'm getting same segfault, and approximatly at same time (thumbnail creation or read)...

For now, I have no more idea where to look...

Can anyone help me on this issue ?

Thanks in advance for your answer.

Cya

Baptiste

----------

## mocsokmike

Is there anything in the apache error log?

What amount of physical memory (RAM) does your server have?

----------

## mryoung_fr

Thanks for your answer ...

Nothing in Apache log, except the normal "segfault" message (child process exit)...

The serveur is a physical box, with 4GB RAM ...

php memory_limit is set to 128MB, but as i said, i tried to raise it to 512MB even 1GB (overkill I know  :Smile: ), without any success...

Cya

----------

## mocsokmike

I think 128 MB should be enough for everything.

If you have a 2.6 kernel still in your box, try with that one.

You can also try enabling E_WARNING in your php.ini (add E_WARNING to the error_reporting line), and set the report_memleaks to On, if it isn't that already. This will help you to find out whether the source of your troubles is apache/php, or Gallery. But be careful with this E_WARNING, if the mentioned box is a production server, it could dump warning messages in every web app you have...  :Wink: 

----------

## mryoung_fr

Thanks for the trick ... I'll try it ASAP and give you feedback.

Thanks again.

----------

## mryoung_fr

Hi again,

So, i recompiled apache and php with debug use flag ... and i haven't any new information ... log stay empty ...

I decided to strace the apache process ... but for the moment, i don't get any useful information. .. if someone can help ...

For now, this is where I am

--

1. Compile PHP with Debug

2. Compile Apache with Debug

3. Check log... Nothing more

4. Start apache process with strace:

```
strace -o/home/toto2 -f /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D LANGUAGE -D SSL -D SSL_DEFAULT_VHOST -D PHP5 -D AUTHNZ_LDAP -D ERRORDOCS -D LDAP -D USERDIR -D STATUS -D PROXY -d /usr/lib64/apache2 -f /etc/apache2/httpd.conf -k start
```

5. Browsing, and waiting for a process segfault ... Got it (PID: 11994)

```
kernel: grsec: From 10.1.1.1: signal 11 sent to /usr/sbin/apache2[apache2:11994] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:11989] uid/euid:0/0 gid/egid:0/0 by /usr/sbin/apache2[apache2:12028] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:11989] uid/euid:0/0 gid/egid:0/0
```

6. Looking at last strace messages

```
# grep ^11994 toto2 | tail 

11994 <... rt_sigprocmask resumed> NULL, 8) = 0

11994 clone( <unfinished ...>

11994 <... clone resumed> child_stack=0x2530025dfb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2530025e9d0, tls=0x2530025e700, child_tidptr=0x2530025e9d0) = 11995

11994 rt_sigprocmask(SIG_UNBLOCK, [TERM],  <unfinished ...>

11994 <... rt_sigprocmask resumed> NULL, 8) = 0

11994 rt_sigaction(SIGTERM, {0x2db7fbc8e6, [], SA_RESTORER|SA_INTERRUPT, 0x2530d1aba80},  <unfinished ...>

11994 <... rt_sigaction resumed> {0x2db7fbeda7, [], SA_RESTORER|SA_INTERRUPT, 0x2530d1aba80}, 8) = 0

11994 read(7,  <unfinished ...>

11994 <... read resumed> 0x3ae12aaa787, 1) = ? ERESTARTSYS (To be restarted)

11994 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
```

7. Hmm seems to crash during reading

8. Apache is not completly dead, i can continue browsing ... Second crash (apache is now definitly dead): PID: 11993

```
kernel: grsec: From 10.1.1.1: signal 11 sent to /usr/sbin/apache2[apache2:11993] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:11989] uid/euid:0/0 gid/egid:0/0 by /usr/sbin/apache2[apache2:12018] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:11989] uid/euid:0/0 gid/egid:0/0
```

9. Looking at strace

```
# grep ^11993 toto2 | tail

11993 <... rt_sigprocmask resumed> NULL, 8) = 0

11993 clone( <unfinished ...>

11993 <... clone resumed> child_stack=0x2530025dfb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2530025e9d0, tls=0x2530025e700, child_tidptr=0x2530025e9d0) = 11996

11993 rt_sigprocmask(SIG_UNBLOCK, [TERM],  <unfinished ...>

11993 <... rt_sigprocmask resumed> NULL, 8) = 0

11993 rt_sigaction(SIGTERM, {0x2db7fbc8e6, [], SA_RESTORER|SA_INTERRUPT, 0x2530d1aba80},  <unfinished ...>

11993 <... rt_sigaction resumed> {0x2db7fbeda7, [], SA_RESTORER|SA_INTERRUPT, 0x2530d1aba80}, 8) = 0

11993 read(7,  <unfinished ...>

11993 <... read resumed> 0x3ae12aaa787, 1) = ? ERESTARTSYS (To be restarted)

11993 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
```

=> for now, i know that 2 process are crashing during read (just before Apache went completly out).

These 2 process are reading a FD=7, and store it in the 0x3ae12aaa787 buffer... Exact same buffer for the 2 process...

If I look at strace messages, no other process are trying to store in this buffer ... only these 2 buggies process

# grep 0x3ae12aaa787 toto2 

11994 <... read resumed> 0x3ae12aaa787, 1) = ? ERESTARTSYS (To be restarted)

11993 <... read resumed> 0x3ae12aaa787, 1) = ? ERESTARTSYS (To be restarted)[/code]

I'm not really an strace expert, and I'm not sure I'm reading correctly ... Is there anyone around able to help me with this issue ?

Thanks in advance for your answer.

Cya

----------

## mryoung_fr

Finally, i figure it out ...

Apache and PHP were compiled with THREADS use flag, which means MPM-WORKER for Apache (in other words, multithreading worker)... and i use SAPI mode...

As said here: http://overlays.gentoo.org/proj/php/wiki/CommonQuestions it seems to be a supported configuration:

 *Quote:*   

> If you compile Apache2 with any of the following MPMs: mpm-event mpm-leader metuxmpm mpm-perchild mpm-threadpool mpm-worker
> 
> then make sure you compile PHP with the threads USE flag: echo "dev-lang/php threads" >> /etc/portage/package.use

 

But finally, if i look at PHP FAQ, PHP team doesn't recommand it at all (or at least, they say to use FastCGI instead of SAPI)

http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2

 *Quote:*   

> Why shouldn't I use Apache2 with a threaded MPM in a production environment?
> 
>     PHP is glue. It is the glue used to build cool web applications by sticking dozens of 3rd-party libraries together and making it all appear as one coherent entity through an intuitive and easy to learn language interface. The flexibility and power of PHP relies on the stability and robustness of the underlying platform. It needs a working OS, a working web server and working 3rd-party libraries to glue together. When any of these stop working PHP needs ways to identify the problems and fix them quickly. When you make the underlying framework more complex by not having completely separate execution threads, completely separate memory segments and a strong sandbox for each request to play in, further weaknesses are introduced into PHP's system.
> 
>     If you want to use a threaded MPM, look at a FastCGI configuration where PHP is running in its own memory space.

 

So finally, i compiled Apache+PHP without the Threads USE flag ... and segfault are gone !!!

Thanks for all your advices.

Cya

----------

