« Return to Thread: Seg-faults on highly-trafficed APC site
Seg-faults on highly-trafficed APC site
by SinistralEnigma
::
Rate this Message:
Hi,
I had a decent number of segfaults that appeared to be caused by APC (I am running APC v3.0.19 and PHP v5.1.6). Rasmus recommended that I bump up the space allotted to APC for caching and that made the problem go away for about a month but it recently came back and caused my servers to crash again.
I have APC running on two Apache servers (behind a squid caching server) and I watched apc_INFO.php closely for the whole month, checking several times a day. Neither of the servers were using even 50% of the space allotted to them, and they both had a "Cache full count" of 0 the last time I checked (which would have been only a matter of hours before the error occurred).
The only anomaly I saw ahead of time is that the one of the servers (where I hadn't restarted httpd during the whole 4 weeks) started to have values on the apc_INFO panel that were negative. For example, the hits in the hits/misses were a large negative number - around 1.9 billion if I recall. The "Request Rate" was also negative.
If it matters, this is being used for a MediaWiki v1.13.2 site.
Any hints would be appreciated. I've attached a good deal of what I feel is probably helpful debugging information below.
Thanks for your time,
- Sean Colombo [LyricWiki.org]
Looking at my apache error logs showed some interesting hints. For a day or two prior, I had occasional lines like this:
[Tue Dec 02 19:42:56 2008] [notice] child pid 4980 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:01 2008] [notice] child pid 4969 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:02 2008] [notice] child pid 4982 exit signal Segmentation fault (11)
Then when the outage happened, they went from occasional clumps to an almost constant stream:
[Wed Dec 03 10:45:54 2008] [notice] child pid 14167 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:29 2008] [notice] child pid 14166 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:42 2008] [notice] child pid 14000 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:48 2008] [notice] child pid 14154 exit signal Segmentation fault (11)
[Wed Dec 03 10:47:26 2008] [notice] child pid 14181 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:45 2008] [notice] child pid 14011 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14041 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14222 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:48 2008] [notice] child pid 14220 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14183 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14192 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 10:53:52 2008] [notice] child pid 14040 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:53 2008] [notice] child pid 14224 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14176 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14168 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:56 2008] [notice] child pid 14209 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:57 2008] [notice] child pid 14157 exit signal Segmentation fault (11)
After another 30 minutes or so of those segfault notices, and three of the invalid-pointer errors (all with the same hex address at the end), the log-file had a stacktrace:
[Wed Dec 03 11:27:43 2008] [notice] child pid 14501 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x09d4daf4 ***
======= Backtrace: =========
/lib/libc.so.6(cfree+0x1bb)[0xc4e19b]
/etc/httpd/modules/libphp5.so(_efree+0x87)[0x11291a7]
/etc/httpd/modules/libphp5.so(_zval_dtor_func+0x5a)[0x11412ea]
/etc/httpd/modules/libphp5.so[0x1187609]
/etc/httpd/modules/libphp5.so[0x11876dd]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so(zend_execute_scripts+0x2f0)[0x11418b0]
/etc/httpd/modules/libphp5.so(php_execute_script+0x255)[0x11010f5]
/etc/httpd/modules/libphp5.so[0x11c47f5]
/usr/sbin/httpd(ap_run_handler+0x6d)[0x25c97d]
/usr/sbin/httpd(ap_invoke_handler+0x68)[0x260338]
/usr/sbin/httpd(ap_process_request+0x1ae)[0x26c3ae]
/usr/sbin/httpd[0x26918f]
/usr/sbin/httpd(ap_run_process_connection+0x6d)[0x26485d]
/usr/sbin/httpd(ap_process_connection+0x4c)[0x26495c]
/usr/sbin/httpd[0x270972]
/usr/sbin/httpd[0x270c81]
/usr/sbin/httpd(ap_mpm_run+0x913)[0x271683]
/usr/sbin/httpd(main+0x8b7)[0x248147]
/lib/libc.so.6(__libc_start_main+0xdc)[0xbf7dec]
/usr/sbin/httpd[0x247221]
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:06 52669849 /usr/lib/libaprutil-1.so.0.2.7
0012c000-0012d000 rw-p 0001b000 08:06 52669849 /usr/lib/libaprutil-1.so.0.2.7
0012d000-00221000 r-xp 00000000 08:06 16844304 /lib/libdb-4.3.so
00221000-00224000 rw-p 000f3000 08:06 16844304 /lib/libdb-4.3.so
00224000-00227000 r-xp 00000000 08:06 16842959 /lib/libuuid.so.1.2
00227000-00228000 rw-p 00003000 08:06 16842959 /lib/libuuid.so.1.2
00228000-00230000 r-xp 00000000 08:06 52669719 /usr/lib/libkrb5support.so.0.1
00230000-00231000 rw-p 00007000 08:06 52669719 /usr/lib/libkrb5support.so.0.1
00231000-00233000 r-xp 00000000 08:06 16844286 /lib/libkeyutils-1.2.so
00233000-00234000 rw-p 00001000 08:06 16844286 /lib/libkeyutils-1.2.so
00234000-00236000 r-xp 00000000 08:06 52921622 /usr/lib/httpd/modules/mod_auth_basic.so
00236000-00238000 rw-p 00001000 08:06 52921622 /usr/lib/httpd/modules/mod_auth_basic.so
00238000-00283000 r-xp 00000000 08:06 52669857 /usr/sbin/httpd
00283000-00285000 rw-p 0004a000 08:06 52669857 /usr/sbin/httpd
00285000-00288000 rw-p 00285000 00:00 0
00288000-00289000 r-xp 00000000 08:06 52921627 /usr/lib/httpd/modules/mod_authn_dbm.so
00289000-0028b000 rw-p 00000000 08:06 52921627 /usr/lib/httpd/modules/mod_authn_dbm.so
0028b000-0028c000 r-xp 00000000 08:06 52921649 /usr/lib/httpd/modules/mod_env.so
0028c000-0028e000 rw-p 00000000 08:06 52921649 /usr/lib/httpd/modules/mod_env.so
0028e000-002a1000 r-xp 00000000 08:06 16844275 /lib/libpthread-2.5.so
002a1000-002a2000 r--p 00012000 08:06 16844275 /lib/libpthread-2.5.so
002a2000-002a3000 rw-p 00013000 08:06 16844275 /lib/libpthread-2.5.so
002a3000-002a5000 rw-p 002a3000 00:00 0
002a5000-002c4000 r-xp 00000000 08:06 52669829 /usr/lib/libpq.so.4.1
002c4000-002c5000 rw-p 0001f000 08:06 52669829 /usr/lib/libpq.so.4.1
002c5000-002d4000 r-xp 00000000 08:06 16843031 /lib/libresolv-2.5.so
002d4000-002d5000 r--p 0000e000 08:06 16843031 /lib/libresolv-2.5.so
002d5000-002d6000 rw-p 0000f000 08:06 16843031 /lib/libresolv-2.5.so
002d6000-002d8000 rw-p 002d6000 00:00 0
002d8000-002de000 r-xp 00000000 08:06 52921630 /usr/lib/httpd/modules/mod_authnz_ldap.so
002de000-002e0000 rw-p 00005000 08:06 52921630 /usr/lib/httpd/modules/mod_authnz_ldap.so
002e0000-00319000 r-xp 00000000 08:06 52663998 /usr/lib/libldap-2.3.so.0.2.15
00319000-0031a000 rw-p 00039000 08:06 52663998 /usr/lib/libldap-2.3.so.0.2.15
0031a000-00355000 r-xp 00000000 08:06 16844298 /lib/libsepol.so.1
00355000-00356000 rw-p 0003a000 08:06 16844298 /lib/libsepol.so.1
00356000-00360000 rw-p 00356000 00:00 0
00360000-00378000 r-xp 00000000 08:06 52669106 /usr/lib/libsasl2.so.2.0.22
00378000-00379000 rw-p 00017000 08:06 52669106 /usr/lib/libsasl2.so.2.0.22
00379000-00409000 r-xp 00000000 08:06 52670754 /usr/lib/libkrb5.so.3.3
00409000-0040c000 rw-p 0008f000 08:06 52670754 /usr/lib/libkrb5.so.3.3
0040e000-00428000 r-xp 00000000 08:06 16844256 /lib/ld-2.5.so
00428000-00429000 r--p 00019000 08:06 16844256 /lib/ld-2.5.so
00429000-0042a000 rw-p 0001a000 08:06 16844256 /lib/ld-2.5.so
0042a000-0043d000 r-xp 00000000 08:06 16844259 /lib/libnsl-2.5.so
0043d000-0043e000 r--p 00012000 08:06 16844259 /lib/libnsl-2.5.so
0043e000-0043f000 rw-p 00013000 08:06 16844259 /lib/libnsl-2.5.so
0043f000-00441000 rw-p 0043f000 00:00 0
00443000-00468000 r-xp 00000000 08:06 52670747 /usr/lib/libk5crypto.so.3.1
00468000-00469000 rw-p 00025000 08:06 52670747 /usr/lib/libk5crypto.so.3.1
00469000-0047b000 r-xp 00000000 08:06 52659359 /usr/lib/libz.so.1.2.3
0047b000-0047c000 rw-p 00011000 08:06 52659359 /usr/lib/libz.so.1.2.3
0047c000-00481000 r-xp 00000000 08:06 52921660 /usr/lib/httpd/modules/mod_log_config.so
00481000-00483000 rw-p 00004000 08:06 52921660 /usr/lib/httpd/modules/mod_log_config.so
00483000-00488000 r-xp 00000000 08:06 52921665 /usr/lib/httpd/modules/mod_mime_magic.so
00488000-0048a000 rw-p 00004000 08:06 52921665 /usr/lib/httpd/modules/mod_mime_magic.so
0048a000-0048c000 r-xp 00000000 08:06 52921650 /usr/lib/httpd/modules/mod_expires.so
0048c000-0048e000 rw-p 00001000 08:06 52921650 /usr/lib/httpd/modules/mod_expires.so
0048e000-00491000 r-xp 00000000 08:06 52921654 /usr/lib/httpd/modules/mod_headers.so
00491000-00493000 rw-p 00002000 08:06 52921654 /usr/lib/httpd/modules/mod_headers.so
00493000-00495000 r-xp 00000000 08:06 52921680 /usr/lib/httpd/modules/mod_usertrack.so
00495000-00497000 rw-p 00001000 08:06 52921680 /usr/lib/httpd/modules/mod_usertrack.so
00497000-00499000 r-xp 00000000 08:06 52921674 /usr/lib/httpd/modules/mod_setenvif.so
00499000-0049b000 rw-p 00001000 08:06 52921674 /usr/lib/httpd/modules/mod_setenvif.so
0049b000-0049e000 r-xp 00000000 08:06 52921664 /usr/lib/httpd/modules/mod_mime.so
0049e000-004a0000 rw-p 00003000 08:06 52921664 /usr/lib/httpd/modules/mod_mime.so
004a0000-004b5000 r-xp 00000000 08:06 52921642 /usr/lib/httpd/modules/mod_dav.so
004b5000-004b7000 rw-p 00014000 08:06 52921642 /usr/lib/httpd/modules/mod_dav.so
004b7000-004bb000 r-xp 00000000 08:06 52921676 /usr/lib/httpd/modules/mod_status.so
004bb000-004bd000 rw-p 00003000 08:06 52921676 /usr/lib/httpd/modules/mod_status.so
004bd000-004c4000 r-xp 00000000 08:06 52921637 /usr/lib/httpd/modules/mod_autoindex.so
004c4000-004c6000 rw-p 00007000 08:06 52921637 /usr/lib/httpd/modules/mod_autoindex.so
004c6000-004c7000 r-xp 00000000 08:06 52921621 /usr/lib/httpd/modules/mod_asis.so
004c7000-004c9000 rw-p 00000000 08:06 52921621 /usr/lib/httpd/modules/mod_asis.so
004c9000-004cd000 r-xp 00000000 08:06 52921658 /usr/lib/httpd/modules/mod_info.so
004cd000-004cf000 rw-p 00003000 08:06 52921658 /usr/lib/httpd/modules/mod_info.so
004cf000-004d1000 r-xp 00000000 08:06 52921682 /usr/lib/httpd/modules/mod_vhost_alias.so
004d1000-004d3000 rw-p 00001000 08:06 52921682 /usr/lib/httpd/modules/mod_vhost_alias.so
004d3000-004da000 r-xp 00000000 08:06 52921666 /usr/lib/httpd/modules/mod_negotiation.so
004da000-004dc000 rw-p 00006000 08:06 52921666 /usr/lib/httpd/modules/mod_negotiation.so
004dc000-004dd000 r-xp 00000000 08:06 52921646 /usr/lib/httpd/modules/mod_dir.so
004dd000-004df000 rw-p 00001000 08:06 52921646 /usr/lib/httpd/modules/mod_dir.so
004df000-004e0000 r-xp 00000000 08:06 52921619 /usr/lib/httpd/modules/mod_actions.so
004e0000-004e2000 rw-p 00001000 08:06 52921619 /usr/lib/httpd/modules/mod_actions.so
004e2000-004e4000 r-xp 00000000 08:06 52921679 /usr/lib/httpd/modules/mod_userdir.so
004e4000-004e6000 rw-p 00001000 08:06 52921679 /usr/lib/httpd/modules/mod_userdir.so
004e6000-004e8000 r-xp 00000000 08:06 52921620 /usr/lib/httpd/modules/mod_alias.so
004e8000-004ea000 rw-p 00002000 08:06 52921620 /usr/lib/httpd/modules/mod_alias.so
004ea000-004f9000 r-xp 00000000 08:06 52921667 /usr/lib/httpd/modules/mod_proxy.so
004f9000-004fb000 rw-p 0000f000 08:06 52921667 /usr/lib/httpd/modules/mod_proxy.so
004fb000-004fd000 r-xp 00000000 08:06 52921670 /usr/lib/httpd/modules/mod_proxy_connect.so
004fd000-004ff000 rw-p 00001000 08:06 52921670 /usr/lib/httpd/modules/mod_proxy_connect.so
004ff000-0051b000 r-xp 00000000 08:06 16844305 /lib/libpcre.so.0.0.1
0051b000-0051c000 rw-p 0001c000 08:06 16844305 /lib/libpcre.so.0.0.1
0051c000-00523000 r-xp 00000000 08:06 52921671 /usr/lib/httpd/modules/mod_proxy_ftp.so
00523000-00525000 rw-p 00006000 08:06 52921671 /usr/lib/httpd/modules/mod_proxy_ftp.so
00525000-0052b000 r-xp 00000000 08:06 52921672 /usr/lib/httpd/modules/mod_proxy_http.so
0052b000-0052d000 rw-p 00005000 08:06 52921672 /usr/lib/httpd/modules/mod_proxy_http.so
0052d000-00534000 r-xp 00000000 08:06 52921638 /usr/lib/httpd/modules/mod_cache.so
00534000-00536000 rw-p 00006000 08:06 52921638 /usr/lib/httpd/modules/mod_cache.so
00536000-00537000 r-xp 00000000 08:06 52921677 /usr/lib/httpd/modules/mod_suexec.so
00537000-00539000 rw-p 00000000 08:06 52921677 /usr/lib/httpd/modules/mod_suexec.so
00539000-0053d000 r-xp 00000000 08:06 52921647 /usr/lib/httpd/modules/mod_disk_cache.so
0053d000-0053f000 rw-p 00003000 08:06 52921647 /usr/lib/httpd/modules/mod_disk_cache.so
0053f000-00541000 r-xp 00000000 08:06 52921652 /usr/lib/httpd/modules/mod_file_cache.so
00541000-00543000 rw-p 00001000 08:06 52921652 /usr/lib/httpd/modules/mod_file_cache.so
00543000-00548000 r-xp 00000000 08:06 52921663 /usr/lib/httpd/modules/mod_mem_cache.so
00548000-0054a000 rw-p 00004000 08:06 52921663 /usr/lib/httpd/modules/mod_mem_cache.so
0054a000-0054f000 r-xp 00000000 08:06 52921640 /usr/lib/httpd/modules/mod_cgi.so
0054f000-00551000 rw-p 00004000 08:06 52921640 /usr/lib/httpd/modules/mod_cgi.so
00551000-00584000 r-xp 00000000 08:06 52671478 /usr/lib/sse2/libgmp.so.3.3.3
00584000-00585000 rw-p 00032000 08:06 52671478 /usr/lib/sse2/libgmp.so.3.3.3
00585000-00590000 r-xp 00000000 08:06 16844301 /lib/libgcc_s-4.1.2-20080102.so.1
00590000-00591000 rw-p 0000a000 08:06 16844301 /lib/libgcc_s-4.1.2-20080102.so.1
00593000-005c0000 r-xp 00000000 08:06 52670808 /usr/lib/libgssapi_krb5.so.2.2
005c0000-005c1000 rw-p 0002d000 08:06 52670808 /usr/lib/libgssapi_krb5.so.2.2
005c2000-005cf000 r-xp 00000000 08:06 52666461 /usr/lib/liblber-2.3.so.0.2.15
005cf000-005d0000 rw-p 0000c000 08:06 52666461 /usr/lib/liblber-2.3.so.0.2.15
005d0000-005e0000 r-xp 00000000 08:06 52664228 /usr/lib/libbz2.so.1.0.3
005e0000-005e1000 rw-p 00010000 08:06 52664228 /usr/lib/libbz2.so.1.0.3
005e1000-005e8000 r-xp 00000000 08:06 52921668 /usr/lib/httpd/modules/mod_proxy_ajp.so
005e8000-005ea000 rw-p 00006000 08:06 52921668 /usr/lib/httpd/modules/mod_proxy_ajp.so
005ea000-005ee000 r-xp 00000000 08:06 52671473 /usr/lib/libdistcache.so.1.0.1
005ee000-005ef000 rw-p 00003000 08:06 52671473 /usr/lib/libdistcache.so.1.0.1
005ef000-005f7000 r-xp 00000000 08:06 52670131 /usr/lib/libnal.so.1.0.1
005f7000-005f8000 rw-p 00007000 08:06 52670131 /usr/lib/libnal.so.1.0.1
005f8000-005fb000 r-xp 00000000 08:06 52857818 /usr/lib/php/modules/fileinfo.so
005fb000-005fc000 rw-p 00002000 08:06 52857818 /usr/lib/php/modules/fileinfo.so
005fc000-00601000 r-xp 00000000 08:06 52857813 /usr/lib/php/modules/pdo_sqlite.so
00601000-00602000 rw-p 00004000 08:06 52857813 /usr/lib/php/modules/pdo_sqlite.so
00602000-0060d000 r-xp 00000000 08:06 52921643 /usr/lib/httpd/modules/mod_dav_fs.so
0060d000-0060f000 rw-p 0000a000 08:06 52921643 /usr/lib/httpd/modules/mod_dav_fs.so
0060f000-0061d000 r-xp 00000000 08:06 52663481 /usr/lib/libmagic.so.1.0.0
0061d000-0061e000 rw-p 0000e000 08:06 52663481 /usr/lib/libmagic.so.1.0.0
00622000-0062b000 r-xp 00000000 08:06 16842793 /lib/libnss_files-2.5.so
0062b000-0062c000 r--p 00008000 08:06 16842793 /lib/libnss_files-2.5.so
0062c000-0062d000 rw-p 00009000 08:06 16842793 /lib/libnss_files-2.5.so
0062d000-00634000 r-xp 00000000 08:06 52857811 /usr/lib/php/modules/dbase.so
00634000-00635000 rw-p 00006000 08:06 52857811 /usr/lib/php/modules/dbase.so
00635000-00640000 r-xp 00000000 08:06 52857814 /usr/lib/php/modules/mysql.so
00640000-00641000 rw-p 0000b000 08:06 52857814 /usr/lib/php/modules/mysql.so
00646000-0064c000 r-xp 00000000 08:06 52857816 /usr/lib/php/modules/pdo_mysql.so
0064c000-0064d000 rw-p 00005000 08:06 52857816 /usr/lib/php/modules/pdo_mysql.so
00655000-00657000 r-xp 00000000 08:06 52921639 /usr/lib/httpd/modules/mod_cern_meta.so
00657000-00659000 rw-p 00001000 08:06 52921639 /usr/lib/httpd/modules/mod_cern_meta.so
00659000-0066c000 r-xp 00000000 08:06 52857812 /usr/lib/php/modules/pdo.so
0066c000-0066e000 rw-p 00012000 08:06 52857812 /usr/lib/php/modules/pdo.so
00672000-00679000 r-xp 00000000 08:06 16844276 /lib/librt-2.5.so
00679000-0067a000 r--p 00006000 08:06 16844276 /lib/librt-2.5.so
0067a000-0067b000 rw-p 00007000 08:06 16844276 /lib/librt-2.5.so
0067e000-00680000 r-xp 00000000 08:06 16844269 /lib/libdl-2.5.so
00680000-00681000 r--p 00001000 08:06 16844269 /lib/libdl-2.5.so
00681000-00682000 rw-p 00002000 08:06 16844269 /lib/libdl-2.5.so
00682000-006bd000 r-xp 00000000 08:06 52671188 /usr/lib/libcurl.so.3.0.0
006bd000-006be000 rw-p 0003b000 08:06 52671188 /usr/lib/libcurl.so.3.0.0
006be000-006ee000 r-xp 00000000 08:06 52670247 /usr/lib/libidn.so.11.5.19
006ee000-006ef000 rw-p 0002f000 08:06 52670247 /usr/lib/libidn.so.11.5.19
00706000-0070c000 r-xp 00000000 08:06 52921623 /usr/lib/httpd/modules/mod_auth_digest.so
0070c000-0070e000 rw-p 00006000 08:06 52921623 /usr/lib/httpd/modules/mod_auth_digest.so
0070e000-00726000 r-xp 00000000 08:06 52857815 /usr/lib/php/modules/mysqli.so
00726000-00728000 rw-p 00018000 08:06 52857815 /usr/lib/php/modules/mysqli.so
0072e000-0084b000 r-xp 00000000 08:06 16844284 /lib/libcrypto.so.0.9.8b
0084b000-0085e000 rw-p 0011c000 08:06 16844284 /lib/libcrypto.so.0.9.8b
0085e000-00861000 rw-p 0085e000 00:00 0
00879000-0087d000 r-xp 00000000 08:06 52921645 /usr/lib/httpd/modules/mod_deflate.so
0087d000-0087f000 rw-p 00003000 08:06 52921645 /usr/lib/httpd/modules/mod_deflate.so
00881000-00882000 r-xp 00881000 00:00 0 [vdso]
00882000-008a9000 r-xp 00000000 08:06 52922693 /usr/lib/httpd/modules/mod_ssl.so
008a9000-008ab000 rw-p 00027000 08:06 52922693 /usr/lib/httpd/modules/mod_ssl.so
008ab000-008ad000 rw-p 008ab000 00:00 0
008c3000-008e2000 r-xp 00000000 08:06 16842961 /lib/libexpat.so.0.5.0
008e2000-008e4000 rw-p 0001e000 08:06 16842961 /lib/libexpat.so.0.5.0
008e4000-00916000 r-xp 00000000 08:06 52857817 /usr/lib/php/modules/gd.so
00916000-00936000 rw-p 00031000 08:06 52857817 /usr/lib/php/modules/gd.so
00936000-0094a000 rw-p 00936000 00:00 0
00959000-0099a000 r-xp 00000000 08:06 16843617 /lib/libssl.so.0.9.8b
0099a000-0099e000 rw-p 00040000 08:06 16843617 /lib/libssl.so.0.9.8b
009a0000-009a9000 r-xp 00000000 08:06 52921657 /usr/lib/httpd/modules/mod_include.so
009a9000-009ab000 rw-p 00009000 08:06 52921657 /usr/lib/httpd/modules/mod_include.so
009ab000-009d0000 r-xp 00000000 08:06 52666993 /usr/lib/libpng12.so.0.10.0
009d0000-009d1000 rw-p 00024000 08:06 52666993 /usr/lib/libpng12.so.0.10.0
009de000-009e0000 r-xp 00000000 08:06 52921675 /usr/lib/httpd/modules/mod_speling.so
009e0000-009e2000 rw-p 00001000 08:06 52921675 /usr/lib/httpd/modules/mod_speling.so
009e2000-00a03000 r-xp 00000000 08:06 52671469 /usr/lib/libjpeg.so.62.0.0
00a03000-00a04000 rw-p 00020000 08:06 52671469 /usr/lib/libjpeg.so.62.0.0
00a06000-00a0f000 r-xp 00000000 08:06 16844279 /lib/libcrypt-2.5.so
00a0f000-00a10000 r--p 00008000 08:06 16844279 /lib/libcrypt-2.5.so
00a10000-00a11000 rw-p 00009000 08:06 16844279 /lib/libcrypt-2.5.so
00a11000-00a38000 rw-p 00a11000 00:00 0
00a38000-00af1000 r-xp 00000000 08:06 52670718 /usr/lib/libaspell.so.15.1.3
00af1000-00af5000 rw-p 000b8000 08:06 52670718 /usr/lib/libaspell.so.15.1.3
00af5000-00af9000 rw-p 00af5000 00:00 0
00b12000-00b13000 r-xp 00000000 08:06 52921625 /usr/lib/httpd/modules/mod_authn_anon.so
00b13000-00b15000 rw-p 00000000 08:06 52921625 /usr/lib/httpd/modules/mod_authn_anon.so
00b2e000-00b30000 r-xp 00000000 08:06 52921634 /usr/lib/httpd/modules/mod_authz_host.so
00b30000-00b32000 rw-p 00001000 08:06 52921634 /usr/lib/httpd/modules/mod_authz_host.so
00b5c000-00b67000 r-xp 00000000 08:06 52921659 /usr/lib/httpd/modules/mod_ldap.so
00b67000-00b69000 rw-p 0000a000 08:06 52921659 /usr/lib/httpd/modules/mod_ldap.so
00be2000-00d1f000 r-xp 00000000 08:06 16844266 /lib/libc-2.5.so
00d1f000-00d21000 r--p 0013c000 08:06 16844266 /lib/libc-2.5.so
00d21000-00d22000 rw-p 0013e000 08:06 16844266 /lib/libc-2.5.so
00d22000-00d25000 rw-p 00d22000 00:00 0
00d26000-00d34000 r-xp 00000000 08:06 52921673 /usr/lib/httpd/modules/mod_rewrite.so
00d34000-00d36000 rw-p 0000d000 08:06 52921673 /usr/lib/httpd/modules/mod_rewrite.so
00d72000-00d74000 r-xp 00000000 08:06 16844287 /lib/libcom_err.so.2.1
00d74000-00d75000 rw-p 00001000 08:06 16844287 /lib/libcom_err.so.2.1
00d9d000-00dfa000 r-xp 00000000 08:06 52663637 /usr/lib/libsqlite3.so.0.8.6
00dfa000-00dfc000 rw-p 0005c000 08:06 52663637 /usr/lib/libsqlite3.so.0.8.6
00e69000-00e8e000 r-xp 00000000 08:06 16844268 /lib/libm-2.5.so
00e8e000-00e8f000 r--p 00024000 08:06 16844268 /lib/libm-2.5.so
00e8f000-00e90000 rw-p 00025000 08:06 16844268 /lib/libm-2.5.so
00ec8000-00edb000 r-xp 00000000 08:06 3866824 /usr/lib/php/modules/apc.so
00edb000-00edc000 rw-p 00013000 08:06 3866824 /usr/lib/php/modules/apc.so
00edc000-00ee1000 rw-p 00edc000 00:00 0
00f1f000-00f20000 r-xp 00000000 08:06 52671477 /usr/lib/libpspell.so.15.1.3
00f20000-00f21000 rw-p 00000000 08:06 52671477 /usr/lib/libpspell.so.15.1.3
00f22000-00f38000 r-xp 00000000 08:06 16844299 /lib/libselinux.so.1
00f38000-00f3a000 rw-p 00015000 08:06 16844299 /lib/libselinux.so.1
00f5a000-00f81000 r-xp 00000000 08:06 52669012 /usr/lib/libapr-1.so.0.2.7
00f81000-00f82000 rw-p 00026000 08:06 52669012 /usr/lib/libapr-1.so.0.2.7
00f82000-01241000 r-xp 00000000 08:06 52922692 /usr/lib/httpd/modules/libphp5.so
01241000-01260000 rw-p 002bf000 08:06 52922692 /usr/lib/httpd/modules/libphp5.so
01260000-01268000 rw-p 01260000 00:00 0
01268000-013f9000 r-xp 00000000 08:06 52857819 /usr/lib/php/modules/mbstring.so
013f9000-0140c000 rw-p 00191000 08:06 52857819 /usr/lib/php/modules/mbstring.so
02a7d000-02afa000 r-xp 00000000 08:06 52670281 /usr/lib/libfreetype.so.6.3.10
02afa000-02afd000 rw-p 0007d000 08:06 52670281 /usr/lib/libfreetype.so.6.3.10
05f33000-06013000 r-xp 00000000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06013000-06017000 r--p 000df000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06017000-06018000 rw-p 000e3000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06018000-0601e000 rw-p 06018000 00:00 0
06813000-0693f000 r-xp 00000000 08:06 52666505 /usr/lib/libxml2.so.2.6.26
0693f000-06944000 rw-p 0012c000 08:06 52666505 /usr/lib/libxml2.so.2.6.26
06944000-06945000 rw-p 06944000 00:00 0
0757a000-0769d000 r-xp 00000000 08:06 52954267 /usr/lib/mysql/libmysqlclient.so.15.0.0
0769d000-076df000 rw-p 00122000 08:06 52954267 /usr/lib/mysql/libmysqlclient.so.15.0.0
076df000-076e0000 rw-p 076df000 00:00 0
09892000-09ed5000 rw-p 09892000 00:00 0
afe63000-afe77000 rw-s 00000000 00:08 888627839 /dev/zero (deleted)
afe77000-afef5000 rw-s 00000000 00:08 1021609716 /dev/zero (deleted)
afef5000-b7ef5000 rw-s 00000000 08:06 29949955 /tmp/apc.RumXNU (deleted)
b7ef5000-b7f0e000 rw-s 00000000 00:08 1021609714 /dev/zero (deleted)
b7f0e000-b7f16000 rw-p b7f0e000 00:00 0
bfd91000-bfda7000 rw-p bfd91000 00:00 0 [stack]
[Wed Dec 03 11:27:48 2008] [notice] child pid 14537 exit signal Aborted (6)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 11:27:54 2008] [notice] child pid 14543 exit signal Segmentation fault (11)
[Wed Dec 03 11:27:55 2008] [notice] child pid 14542 exit signal Segmentation fault (11)
I had a decent number of segfaults that appeared to be caused by APC (I am running APC v3.0.19 and PHP v5.1.6). Rasmus recommended that I bump up the space allotted to APC for caching and that made the problem go away for about a month but it recently came back and caused my servers to crash again.
I have APC running on two Apache servers (behind a squid caching server) and I watched apc_INFO.php closely for the whole month, checking several times a day. Neither of the servers were using even 50% of the space allotted to them, and they both had a "Cache full count" of 0 the last time I checked (which would have been only a matter of hours before the error occurred).
The only anomaly I saw ahead of time is that the one of the servers (where I hadn't restarted httpd during the whole 4 weeks) started to have values on the apc_INFO panel that were negative. For example, the hits in the hits/misses were a large negative number - around 1.9 billion if I recall. The "Request Rate" was also negative.
If it matters, this is being used for a MediaWiki v1.13.2 site.
Any hints would be appreciated. I've attached a good deal of what I feel is probably helpful debugging information below.
Thanks for your time,
- Sean Colombo [LyricWiki.org]
Looking at my apache error logs showed some interesting hints. For a day or two prior, I had occasional lines like this:
[Tue Dec 02 19:42:56 2008] [notice] child pid 4980 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:01 2008] [notice] child pid 4969 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:02 2008] [notice] child pid 4982 exit signal Segmentation fault (11)
Then when the outage happened, they went from occasional clumps to an almost constant stream:
[Wed Dec 03 10:45:54 2008] [notice] child pid 14167 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:29 2008] [notice] child pid 14166 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:42 2008] [notice] child pid 14000 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:48 2008] [notice] child pid 14154 exit signal Segmentation fault (11)
[Wed Dec 03 10:47:26 2008] [notice] child pid 14181 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:45 2008] [notice] child pid 14011 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14041 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14222 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:48 2008] [notice] child pid 14220 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14183 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14192 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 10:53:52 2008] [notice] child pid 14040 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:53 2008] [notice] child pid 14224 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14176 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14168 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:56 2008] [notice] child pid 14209 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:57 2008] [notice] child pid 14157 exit signal Segmentation fault (11)
After another 30 minutes or so of those segfault notices, and three of the invalid-pointer errors (all with the same hex address at the end), the log-file had a stacktrace:
[Wed Dec 03 11:27:43 2008] [notice] child pid 14501 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x09d4daf4 ***
======= Backtrace: =========
/lib/libc.so.6(cfree+0x1bb)[0xc4e19b]
/etc/httpd/modules/libphp5.so(_efree+0x87)[0x11291a7]
/etc/httpd/modules/libphp5.so(_zval_dtor_func+0x5a)[0x11412ea]
/etc/httpd/modules/libphp5.so[0x1187609]
/etc/httpd/modules/libphp5.so[0x11876dd]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so(zend_execute_scripts+0x2f0)[0x11418b0]
/etc/httpd/modules/libphp5.so(php_execute_script+0x255)[0x11010f5]
/etc/httpd/modules/libphp5.so[0x11c47f5]
/usr/sbin/httpd(ap_run_handler+0x6d)[0x25c97d]
/usr/sbin/httpd(ap_invoke_handler+0x68)[0x260338]
/usr/sbin/httpd(ap_process_request+0x1ae)[0x26c3ae]
/usr/sbin/httpd[0x26918f]
/usr/sbin/httpd(ap_run_process_connection+0x6d)[0x26485d]
/usr/sbin/httpd(ap_process_connection+0x4c)[0x26495c]
/usr/sbin/httpd[0x270972]
/usr/sbin/httpd[0x270c81]
/usr/sbin/httpd(ap_mpm_run+0x913)[0x271683]
/usr/sbin/httpd(main+0x8b7)[0x248147]
/lib/libc.so.6(__libc_start_main+0xdc)[0xbf7dec]
/usr/sbin/httpd[0x247221]
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:06 52669849 /usr/lib/libaprutil-1.so.0.2.7
0012c000-0012d000 rw-p 0001b000 08:06 52669849 /usr/lib/libaprutil-1.so.0.2.7
0012d000-00221000 r-xp 00000000 08:06 16844304 /lib/libdb-4.3.so
00221000-00224000 rw-p 000f3000 08:06 16844304 /lib/libdb-4.3.so
00224000-00227000 r-xp 00000000 08:06 16842959 /lib/libuuid.so.1.2
00227000-00228000 rw-p 00003000 08:06 16842959 /lib/libuuid.so.1.2
00228000-00230000 r-xp 00000000 08:06 52669719 /usr/lib/libkrb5support.so.0.1
00230000-00231000 rw-p 00007000 08:06 52669719 /usr/lib/libkrb5support.so.0.1
00231000-00233000 r-xp 00000000 08:06 16844286 /lib/libkeyutils-1.2.so
00233000-00234000 rw-p 00001000 08:06 16844286 /lib/libkeyutils-1.2.so
00234000-00236000 r-xp 00000000 08:06 52921622 /usr/lib/httpd/modules/mod_auth_basic.so
00236000-00238000 rw-p 00001000 08:06 52921622 /usr/lib/httpd/modules/mod_auth_basic.so
00238000-00283000 r-xp 00000000 08:06 52669857 /usr/sbin/httpd
00283000-00285000 rw-p 0004a000 08:06 52669857 /usr/sbin/httpd
00285000-00288000 rw-p 00285000 00:00 0
00288000-00289000 r-xp 00000000 08:06 52921627 /usr/lib/httpd/modules/mod_authn_dbm.so
00289000-0028b000 rw-p 00000000 08:06 52921627 /usr/lib/httpd/modules/mod_authn_dbm.so
0028b000-0028c000 r-xp 00000000 08:06 52921649 /usr/lib/httpd/modules/mod_env.so
0028c000-0028e000 rw-p 00000000 08:06 52921649 /usr/lib/httpd/modules/mod_env.so
0028e000-002a1000 r-xp 00000000 08:06 16844275 /lib/libpthread-2.5.so
002a1000-002a2000 r--p 00012000 08:06 16844275 /lib/libpthread-2.5.so
002a2000-002a3000 rw-p 00013000 08:06 16844275 /lib/libpthread-2.5.so
002a3000-002a5000 rw-p 002a3000 00:00 0
002a5000-002c4000 r-xp 00000000 08:06 52669829 /usr/lib/libpq.so.4.1
002c4000-002c5000 rw-p 0001f000 08:06 52669829 /usr/lib/libpq.so.4.1
002c5000-002d4000 r-xp 00000000 08:06 16843031 /lib/libresolv-2.5.so
002d4000-002d5000 r--p 0000e000 08:06 16843031 /lib/libresolv-2.5.so
002d5000-002d6000 rw-p 0000f000 08:06 16843031 /lib/libresolv-2.5.so
002d6000-002d8000 rw-p 002d6000 00:00 0
002d8000-002de000 r-xp 00000000 08:06 52921630 /usr/lib/httpd/modules/mod_authnz_ldap.so
002de000-002e0000 rw-p 00005000 08:06 52921630 /usr/lib/httpd/modules/mod_authnz_ldap.so
002e0000-00319000 r-xp 00000000 08:06 52663998 /usr/lib/libldap-2.3.so.0.2.15
00319000-0031a000 rw-p 00039000 08:06 52663998 /usr/lib/libldap-2.3.so.0.2.15
0031a000-00355000 r-xp 00000000 08:06 16844298 /lib/libsepol.so.1
00355000-00356000 rw-p 0003a000 08:06 16844298 /lib/libsepol.so.1
00356000-00360000 rw-p 00356000 00:00 0
00360000-00378000 r-xp 00000000 08:06 52669106 /usr/lib/libsasl2.so.2.0.22
00378000-00379000 rw-p 00017000 08:06 52669106 /usr/lib/libsasl2.so.2.0.22
00379000-00409000 r-xp 00000000 08:06 52670754 /usr/lib/libkrb5.so.3.3
00409000-0040c000 rw-p 0008f000 08:06 52670754 /usr/lib/libkrb5.so.3.3
0040e000-00428000 r-xp 00000000 08:06 16844256 /lib/ld-2.5.so
00428000-00429000 r--p 00019000 08:06 16844256 /lib/ld-2.5.so
00429000-0042a000 rw-p 0001a000 08:06 16844256 /lib/ld-2.5.so
0042a000-0043d000 r-xp 00000000 08:06 16844259 /lib/libnsl-2.5.so
0043d000-0043e000 r--p 00012000 08:06 16844259 /lib/libnsl-2.5.so
0043e000-0043f000 rw-p 00013000 08:06 16844259 /lib/libnsl-2.5.so
0043f000-00441000 rw-p 0043f000 00:00 0
00443000-00468000 r-xp 00000000 08:06 52670747 /usr/lib/libk5crypto.so.3.1
00468000-00469000 rw-p 00025000 08:06 52670747 /usr/lib/libk5crypto.so.3.1
00469000-0047b000 r-xp 00000000 08:06 52659359 /usr/lib/libz.so.1.2.3
0047b000-0047c000 rw-p 00011000 08:06 52659359 /usr/lib/libz.so.1.2.3
0047c000-00481000 r-xp 00000000 08:06 52921660 /usr/lib/httpd/modules/mod_log_config.so
00481000-00483000 rw-p 00004000 08:06 52921660 /usr/lib/httpd/modules/mod_log_config.so
00483000-00488000 r-xp 00000000 08:06 52921665 /usr/lib/httpd/modules/mod_mime_magic.so
00488000-0048a000 rw-p 00004000 08:06 52921665 /usr/lib/httpd/modules/mod_mime_magic.so
0048a000-0048c000 r-xp 00000000 08:06 52921650 /usr/lib/httpd/modules/mod_expires.so
0048c000-0048e000 rw-p 00001000 08:06 52921650 /usr/lib/httpd/modules/mod_expires.so
0048e000-00491000 r-xp 00000000 08:06 52921654 /usr/lib/httpd/modules/mod_headers.so
00491000-00493000 rw-p 00002000 08:06 52921654 /usr/lib/httpd/modules/mod_headers.so
00493000-00495000 r-xp 00000000 08:06 52921680 /usr/lib/httpd/modules/mod_usertrack.so
00495000-00497000 rw-p 00001000 08:06 52921680 /usr/lib/httpd/modules/mod_usertrack.so
00497000-00499000 r-xp 00000000 08:06 52921674 /usr/lib/httpd/modules/mod_setenvif.so
00499000-0049b000 rw-p 00001000 08:06 52921674 /usr/lib/httpd/modules/mod_setenvif.so
0049b000-0049e000 r-xp 00000000 08:06 52921664 /usr/lib/httpd/modules/mod_mime.so
0049e000-004a0000 rw-p 00003000 08:06 52921664 /usr/lib/httpd/modules/mod_mime.so
004a0000-004b5000 r-xp 00000000 08:06 52921642 /usr/lib/httpd/modules/mod_dav.so
004b5000-004b7000 rw-p 00014000 08:06 52921642 /usr/lib/httpd/modules/mod_dav.so
004b7000-004bb000 r-xp 00000000 08:06 52921676 /usr/lib/httpd/modules/mod_status.so
004bb000-004bd000 rw-p 00003000 08:06 52921676 /usr/lib/httpd/modules/mod_status.so
004bd000-004c4000 r-xp 00000000 08:06 52921637 /usr/lib/httpd/modules/mod_autoindex.so
004c4000-004c6000 rw-p 00007000 08:06 52921637 /usr/lib/httpd/modules/mod_autoindex.so
004c6000-004c7000 r-xp 00000000 08:06 52921621 /usr/lib/httpd/modules/mod_asis.so
004c7000-004c9000 rw-p 00000000 08:06 52921621 /usr/lib/httpd/modules/mod_asis.so
004c9000-004cd000 r-xp 00000000 08:06 52921658 /usr/lib/httpd/modules/mod_info.so
004cd000-004cf000 rw-p 00003000 08:06 52921658 /usr/lib/httpd/modules/mod_info.so
004cf000-004d1000 r-xp 00000000 08:06 52921682 /usr/lib/httpd/modules/mod_vhost_alias.so
004d1000-004d3000 rw-p 00001000 08:06 52921682 /usr/lib/httpd/modules/mod_vhost_alias.so
004d3000-004da000 r-xp 00000000 08:06 52921666 /usr/lib/httpd/modules/mod_negotiation.so
004da000-004dc000 rw-p 00006000 08:06 52921666 /usr/lib/httpd/modules/mod_negotiation.so
004dc000-004dd000 r-xp 00000000 08:06 52921646 /usr/lib/httpd/modules/mod_dir.so
004dd000-004df000 rw-p 00001000 08:06 52921646 /usr/lib/httpd/modules/mod_dir.so
004df000-004e0000 r-xp 00000000 08:06 52921619 /usr/lib/httpd/modules/mod_actions.so
004e0000-004e2000 rw-p 00001000 08:06 52921619 /usr/lib/httpd/modules/mod_actions.so
004e2000-004e4000 r-xp 00000000 08:06 52921679 /usr/lib/httpd/modules/mod_userdir.so
004e4000-004e6000 rw-p 00001000 08:06 52921679 /usr/lib/httpd/modules/mod_userdir.so
004e6000-004e8000 r-xp 00000000 08:06 52921620 /usr/lib/httpd/modules/mod_alias.so
004e8000-004ea000 rw-p 00002000 08:06 52921620 /usr/lib/httpd/modules/mod_alias.so
004ea000-004f9000 r-xp 00000000 08:06 52921667 /usr/lib/httpd/modules/mod_proxy.so
004f9000-004fb000 rw-p 0000f000 08:06 52921667 /usr/lib/httpd/modules/mod_proxy.so
004fb000-004fd000 r-xp 00000000 08:06 52921670 /usr/lib/httpd/modules/mod_proxy_connect.so
004fd000-004ff000 rw-p 00001000 08:06 52921670 /usr/lib/httpd/modules/mod_proxy_connect.so
004ff000-0051b000 r-xp 00000000 08:06 16844305 /lib/libpcre.so.0.0.1
0051b000-0051c000 rw-p 0001c000 08:06 16844305 /lib/libpcre.so.0.0.1
0051c000-00523000 r-xp 00000000 08:06 52921671 /usr/lib/httpd/modules/mod_proxy_ftp.so
00523000-00525000 rw-p 00006000 08:06 52921671 /usr/lib/httpd/modules/mod_proxy_ftp.so
00525000-0052b000 r-xp 00000000 08:06 52921672 /usr/lib/httpd/modules/mod_proxy_http.so
0052b000-0052d000 rw-p 00005000 08:06 52921672 /usr/lib/httpd/modules/mod_proxy_http.so
0052d000-00534000 r-xp 00000000 08:06 52921638 /usr/lib/httpd/modules/mod_cache.so
00534000-00536000 rw-p 00006000 08:06 52921638 /usr/lib/httpd/modules/mod_cache.so
00536000-00537000 r-xp 00000000 08:06 52921677 /usr/lib/httpd/modules/mod_suexec.so
00537000-00539000 rw-p 00000000 08:06 52921677 /usr/lib/httpd/modules/mod_suexec.so
00539000-0053d000 r-xp 00000000 08:06 52921647 /usr/lib/httpd/modules/mod_disk_cache.so
0053d000-0053f000 rw-p 00003000 08:06 52921647 /usr/lib/httpd/modules/mod_disk_cache.so
0053f000-00541000 r-xp 00000000 08:06 52921652 /usr/lib/httpd/modules/mod_file_cache.so
00541000-00543000 rw-p 00001000 08:06 52921652 /usr/lib/httpd/modules/mod_file_cache.so
00543000-00548000 r-xp 00000000 08:06 52921663 /usr/lib/httpd/modules/mod_mem_cache.so
00548000-0054a000 rw-p 00004000 08:06 52921663 /usr/lib/httpd/modules/mod_mem_cache.so
0054a000-0054f000 r-xp 00000000 08:06 52921640 /usr/lib/httpd/modules/mod_cgi.so
0054f000-00551000 rw-p 00004000 08:06 52921640 /usr/lib/httpd/modules/mod_cgi.so
00551000-00584000 r-xp 00000000 08:06 52671478 /usr/lib/sse2/libgmp.so.3.3.3
00584000-00585000 rw-p 00032000 08:06 52671478 /usr/lib/sse2/libgmp.so.3.3.3
00585000-00590000 r-xp 00000000 08:06 16844301 /lib/libgcc_s-4.1.2-20080102.so.1
00590000-00591000 rw-p 0000a000 08:06 16844301 /lib/libgcc_s-4.1.2-20080102.so.1
00593000-005c0000 r-xp 00000000 08:06 52670808 /usr/lib/libgssapi_krb5.so.2.2
005c0000-005c1000 rw-p 0002d000 08:06 52670808 /usr/lib/libgssapi_krb5.so.2.2
005c2000-005cf000 r-xp 00000000 08:06 52666461 /usr/lib/liblber-2.3.so.0.2.15
005cf000-005d0000 rw-p 0000c000 08:06 52666461 /usr/lib/liblber-2.3.so.0.2.15
005d0000-005e0000 r-xp 00000000 08:06 52664228 /usr/lib/libbz2.so.1.0.3
005e0000-005e1000 rw-p 00010000 08:06 52664228 /usr/lib/libbz2.so.1.0.3
005e1000-005e8000 r-xp 00000000 08:06 52921668 /usr/lib/httpd/modules/mod_proxy_ajp.so
005e8000-005ea000 rw-p 00006000 08:06 52921668 /usr/lib/httpd/modules/mod_proxy_ajp.so
005ea000-005ee000 r-xp 00000000 08:06 52671473 /usr/lib/libdistcache.so.1.0.1
005ee000-005ef000 rw-p 00003000 08:06 52671473 /usr/lib/libdistcache.so.1.0.1
005ef000-005f7000 r-xp 00000000 08:06 52670131 /usr/lib/libnal.so.1.0.1
005f7000-005f8000 rw-p 00007000 08:06 52670131 /usr/lib/libnal.so.1.0.1
005f8000-005fb000 r-xp 00000000 08:06 52857818 /usr/lib/php/modules/fileinfo.so
005fb000-005fc000 rw-p 00002000 08:06 52857818 /usr/lib/php/modules/fileinfo.so
005fc000-00601000 r-xp 00000000 08:06 52857813 /usr/lib/php/modules/pdo_sqlite.so
00601000-00602000 rw-p 00004000 08:06 52857813 /usr/lib/php/modules/pdo_sqlite.so
00602000-0060d000 r-xp 00000000 08:06 52921643 /usr/lib/httpd/modules/mod_dav_fs.so
0060d000-0060f000 rw-p 0000a000 08:06 52921643 /usr/lib/httpd/modules/mod_dav_fs.so
0060f000-0061d000 r-xp 00000000 08:06 52663481 /usr/lib/libmagic.so.1.0.0
0061d000-0061e000 rw-p 0000e000 08:06 52663481 /usr/lib/libmagic.so.1.0.0
00622000-0062b000 r-xp 00000000 08:06 16842793 /lib/libnss_files-2.5.so
0062b000-0062c000 r--p 00008000 08:06 16842793 /lib/libnss_files-2.5.so
0062c000-0062d000 rw-p 00009000 08:06 16842793 /lib/libnss_files-2.5.so
0062d000-00634000 r-xp 00000000 08:06 52857811 /usr/lib/php/modules/dbase.so
00634000-00635000 rw-p 00006000 08:06 52857811 /usr/lib/php/modules/dbase.so
00635000-00640000 r-xp 00000000 08:06 52857814 /usr/lib/php/modules/mysql.so
00640000-00641000 rw-p 0000b000 08:06 52857814 /usr/lib/php/modules/mysql.so
00646000-0064c000 r-xp 00000000 08:06 52857816 /usr/lib/php/modules/pdo_mysql.so
0064c000-0064d000 rw-p 00005000 08:06 52857816 /usr/lib/php/modules/pdo_mysql.so
00655000-00657000 r-xp 00000000 08:06 52921639 /usr/lib/httpd/modules/mod_cern_meta.so
00657000-00659000 rw-p 00001000 08:06 52921639 /usr/lib/httpd/modules/mod_cern_meta.so
00659000-0066c000 r-xp 00000000 08:06 52857812 /usr/lib/php/modules/pdo.so
0066c000-0066e000 rw-p 00012000 08:06 52857812 /usr/lib/php/modules/pdo.so
00672000-00679000 r-xp 00000000 08:06 16844276 /lib/librt-2.5.so
00679000-0067a000 r--p 00006000 08:06 16844276 /lib/librt-2.5.so
0067a000-0067b000 rw-p 00007000 08:06 16844276 /lib/librt-2.5.so
0067e000-00680000 r-xp 00000000 08:06 16844269 /lib/libdl-2.5.so
00680000-00681000 r--p 00001000 08:06 16844269 /lib/libdl-2.5.so
00681000-00682000 rw-p 00002000 08:06 16844269 /lib/libdl-2.5.so
00682000-006bd000 r-xp 00000000 08:06 52671188 /usr/lib/libcurl.so.3.0.0
006bd000-006be000 rw-p 0003b000 08:06 52671188 /usr/lib/libcurl.so.3.0.0
006be000-006ee000 r-xp 00000000 08:06 52670247 /usr/lib/libidn.so.11.5.19
006ee000-006ef000 rw-p 0002f000 08:06 52670247 /usr/lib/libidn.so.11.5.19
00706000-0070c000 r-xp 00000000 08:06 52921623 /usr/lib/httpd/modules/mod_auth_digest.so
0070c000-0070e000 rw-p 00006000 08:06 52921623 /usr/lib/httpd/modules/mod_auth_digest.so
0070e000-00726000 r-xp 00000000 08:06 52857815 /usr/lib/php/modules/mysqli.so
00726000-00728000 rw-p 00018000 08:06 52857815 /usr/lib/php/modules/mysqli.so
0072e000-0084b000 r-xp 00000000 08:06 16844284 /lib/libcrypto.so.0.9.8b
0084b000-0085e000 rw-p 0011c000 08:06 16844284 /lib/libcrypto.so.0.9.8b
0085e000-00861000 rw-p 0085e000 00:00 0
00879000-0087d000 r-xp 00000000 08:06 52921645 /usr/lib/httpd/modules/mod_deflate.so
0087d000-0087f000 rw-p 00003000 08:06 52921645 /usr/lib/httpd/modules/mod_deflate.so
00881000-00882000 r-xp 00881000 00:00 0 [vdso]
00882000-008a9000 r-xp 00000000 08:06 52922693 /usr/lib/httpd/modules/mod_ssl.so
008a9000-008ab000 rw-p 00027000 08:06 52922693 /usr/lib/httpd/modules/mod_ssl.so
008ab000-008ad000 rw-p 008ab000 00:00 0
008c3000-008e2000 r-xp 00000000 08:06 16842961 /lib/libexpat.so.0.5.0
008e2000-008e4000 rw-p 0001e000 08:06 16842961 /lib/libexpat.so.0.5.0
008e4000-00916000 r-xp 00000000 08:06 52857817 /usr/lib/php/modules/gd.so
00916000-00936000 rw-p 00031000 08:06 52857817 /usr/lib/php/modules/gd.so
00936000-0094a000 rw-p 00936000 00:00 0
00959000-0099a000 r-xp 00000000 08:06 16843617 /lib/libssl.so.0.9.8b
0099a000-0099e000 rw-p 00040000 08:06 16843617 /lib/libssl.so.0.9.8b
009a0000-009a9000 r-xp 00000000 08:06 52921657 /usr/lib/httpd/modules/mod_include.so
009a9000-009ab000 rw-p 00009000 08:06 52921657 /usr/lib/httpd/modules/mod_include.so
009ab000-009d0000 r-xp 00000000 08:06 52666993 /usr/lib/libpng12.so.0.10.0
009d0000-009d1000 rw-p 00024000 08:06 52666993 /usr/lib/libpng12.so.0.10.0
009de000-009e0000 r-xp 00000000 08:06 52921675 /usr/lib/httpd/modules/mod_speling.so
009e0000-009e2000 rw-p 00001000 08:06 52921675 /usr/lib/httpd/modules/mod_speling.so
009e2000-00a03000 r-xp 00000000 08:06 52671469 /usr/lib/libjpeg.so.62.0.0
00a03000-00a04000 rw-p 00020000 08:06 52671469 /usr/lib/libjpeg.so.62.0.0
00a06000-00a0f000 r-xp 00000000 08:06 16844279 /lib/libcrypt-2.5.so
00a0f000-00a10000 r--p 00008000 08:06 16844279 /lib/libcrypt-2.5.so
00a10000-00a11000 rw-p 00009000 08:06 16844279 /lib/libcrypt-2.5.so
00a11000-00a38000 rw-p 00a11000 00:00 0
00a38000-00af1000 r-xp 00000000 08:06 52670718 /usr/lib/libaspell.so.15.1.3
00af1000-00af5000 rw-p 000b8000 08:06 52670718 /usr/lib/libaspell.so.15.1.3
00af5000-00af9000 rw-p 00af5000 00:00 0
00b12000-00b13000 r-xp 00000000 08:06 52921625 /usr/lib/httpd/modules/mod_authn_anon.so
00b13000-00b15000 rw-p 00000000 08:06 52921625 /usr/lib/httpd/modules/mod_authn_anon.so
00b2e000-00b30000 r-xp 00000000 08:06 52921634 /usr/lib/httpd/modules/mod_authz_host.so
00b30000-00b32000 rw-p 00001000 08:06 52921634 /usr/lib/httpd/modules/mod_authz_host.so
00b5c000-00b67000 r-xp 00000000 08:06 52921659 /usr/lib/httpd/modules/mod_ldap.so
00b67000-00b69000 rw-p 0000a000 08:06 52921659 /usr/lib/httpd/modules/mod_ldap.so
00be2000-00d1f000 r-xp 00000000 08:06 16844266 /lib/libc-2.5.so
00d1f000-00d21000 r--p 0013c000 08:06 16844266 /lib/libc-2.5.so
00d21000-00d22000 rw-p 0013e000 08:06 16844266 /lib/libc-2.5.so
00d22000-00d25000 rw-p 00d22000 00:00 0
00d26000-00d34000 r-xp 00000000 08:06 52921673 /usr/lib/httpd/modules/mod_rewrite.so
00d34000-00d36000 rw-p 0000d000 08:06 52921673 /usr/lib/httpd/modules/mod_rewrite.so
00d72000-00d74000 r-xp 00000000 08:06 16844287 /lib/libcom_err.so.2.1
00d74000-00d75000 rw-p 00001000 08:06 16844287 /lib/libcom_err.so.2.1
00d9d000-00dfa000 r-xp 00000000 08:06 52663637 /usr/lib/libsqlite3.so.0.8.6
00dfa000-00dfc000 rw-p 0005c000 08:06 52663637 /usr/lib/libsqlite3.so.0.8.6
00e69000-00e8e000 r-xp 00000000 08:06 16844268 /lib/libm-2.5.so
00e8e000-00e8f000 r--p 00024000 08:06 16844268 /lib/libm-2.5.so
00e8f000-00e90000 rw-p 00025000 08:06 16844268 /lib/libm-2.5.so
00ec8000-00edb000 r-xp 00000000 08:06 3866824 /usr/lib/php/modules/apc.so
00edb000-00edc000 rw-p 00013000 08:06 3866824 /usr/lib/php/modules/apc.so
00edc000-00ee1000 rw-p 00edc000 00:00 0
00f1f000-00f20000 r-xp 00000000 08:06 52671477 /usr/lib/libpspell.so.15.1.3
00f20000-00f21000 rw-p 00000000 08:06 52671477 /usr/lib/libpspell.so.15.1.3
00f22000-00f38000 r-xp 00000000 08:06 16844299 /lib/libselinux.so.1
00f38000-00f3a000 rw-p 00015000 08:06 16844299 /lib/libselinux.so.1
00f5a000-00f81000 r-xp 00000000 08:06 52669012 /usr/lib/libapr-1.so.0.2.7
00f81000-00f82000 rw-p 00026000 08:06 52669012 /usr/lib/libapr-1.so.0.2.7
00f82000-01241000 r-xp 00000000 08:06 52922692 /usr/lib/httpd/modules/libphp5.so
01241000-01260000 rw-p 002bf000 08:06 52922692 /usr/lib/httpd/modules/libphp5.so
01260000-01268000 rw-p 01260000 00:00 0
01268000-013f9000 r-xp 00000000 08:06 52857819 /usr/lib/php/modules/mbstring.so
013f9000-0140c000 rw-p 00191000 08:06 52857819 /usr/lib/php/modules/mbstring.so
02a7d000-02afa000 r-xp 00000000 08:06 52670281 /usr/lib/libfreetype.so.6.3.10
02afa000-02afd000 rw-p 0007d000 08:06 52670281 /usr/lib/libfreetype.so.6.3.10
05f33000-06013000 r-xp 00000000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06013000-06017000 r--p 000df000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06017000-06018000 rw-p 000e3000 08:06 52669786 /usr/lib/libstdc++.so.6.0.8
06018000-0601e000 rw-p 06018000 00:00 0
06813000-0693f000 r-xp 00000000 08:06 52666505 /usr/lib/libxml2.so.2.6.26
0693f000-06944000 rw-p 0012c000 08:06 52666505 /usr/lib/libxml2.so.2.6.26
06944000-06945000 rw-p 06944000 00:00 0
0757a000-0769d000 r-xp 00000000 08:06 52954267 /usr/lib/mysql/libmysqlclient.so.15.0.0
0769d000-076df000 rw-p 00122000 08:06 52954267 /usr/lib/mysql/libmysqlclient.so.15.0.0
076df000-076e0000 rw-p 076df000 00:00 0
09892000-09ed5000 rw-p 09892000 00:00 0
afe63000-afe77000 rw-s 00000000 00:08 888627839 /dev/zero (deleted)
afe77000-afef5000 rw-s 00000000 00:08 1021609716 /dev/zero (deleted)
afef5000-b7ef5000 rw-s 00000000 08:06 29949955 /tmp/apc.RumXNU (deleted)
b7ef5000-b7f0e000 rw-s 00000000 00:08 1021609714 /dev/zero (deleted)
b7f0e000-b7f16000 rw-p b7f0e000 00:00 0
bfd91000-bfda7000 rw-p bfd91000 00:00 0 [stack]
[Wed Dec 03 11:27:48 2008] [notice] child pid 14537 exit signal Aborted (6)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 11:27:54 2008] [notice] child pid 14543 exit signal Segmentation fault (11)
[Wed Dec 03 11:27:55 2008] [notice] child pid 14542 exit signal Segmentation fault (11)
« Return to Thread: Seg-faults on highly-trafficed APC site
| Free embeddable forum powered by Nabble | Forum Help |
