| 
Pecl - Dev

Seg-faults on highly-trafficed APC site

View: New views
4 Messages — Rating Filter:   Alert me  

Seg-faults on highly-trafficed APC site

by SinistralEnigma :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi,
I had a decent number of segfaults that appeared to be caused by APC (I am running APC v3.0.19 and PHP v5.1.6). Rasmus recommended that I bump up the space allotted to APC for caching and that made the problem go away for about a month but it recently came back and caused my servers to crash again.

I have APC running on two Apache servers (behind a squid caching server) and I watched apc_INFO.php closely for the whole month, checking several times a day.  Neither of the servers were using even 50% of the space allotted to them, and they both had a "Cache full count" of 0 the last time I checked (which would have been only a matter of hours before the error occurred).

The only anomaly I saw ahead of time is that the one of the servers (where I hadn't restarted httpd during the whole 4 weeks) started to have values on the apc_INFO panel that were negative.  For example, the hits in the hits/misses were a large negative number - around 1.9 billion if I recall.  The "Request Rate" was also negative.

If it matters, this is being used for a MediaWiki v1.13.2 site.

Any hints would be appreciated.  I've attached a good deal of what I feel is probably helpful debugging information below.
Thanks for your time,
 - Sean Colombo [LyricWiki.org]



Looking at my apache error logs showed some interesting hints.  For a day or two prior, I had occasional lines like this:
[Tue Dec 02 19:42:56 2008] [notice] child pid 4980 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:01 2008] [notice] child pid 4969 exit signal Segmentation fault (11)
[Tue Dec 02 19:43:02 2008] [notice] child pid 4982 exit signal Segmentation fault (11)

Then when the outage happened, they went from occasional clumps to an almost constant stream:
[Wed Dec 03 10:45:54 2008] [notice] child pid 14167 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:29 2008] [notice] child pid 14166 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:42 2008] [notice] child pid 14000 exit signal Segmentation fault (11)
[Wed Dec 03 10:46:48 2008] [notice] child pid 14154 exit signal Segmentation fault (11)
[Wed Dec 03 10:47:26 2008] [notice] child pid 14181 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:45 2008] [notice] child pid 14011 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14041 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:46 2008] [notice] child pid 14222 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:48 2008] [notice] child pid 14220 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14183 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:49 2008] [notice] child pid 14192 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 10:53:52 2008] [notice] child pid 14040 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:53 2008] [notice] child pid 14224 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14176 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:54 2008] [notice] child pid 14168 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:56 2008] [notice] child pid 14209 exit signal Segmentation fault (11)
[Wed Dec 03 10:53:57 2008] [notice] child pid 14157 exit signal Segmentation fault (11)

After another 30 minutes or so of those segfault notices, and three of the invalid-pointer errors (all with the same hex address at the end), the log-file had a stacktrace:
[Wed Dec 03 11:27:43 2008] [notice] child pid 14501 exit signal Segmentation fault (11)
*** glibc detected *** /usr/sbin/httpd: munmap_chunk(): invalid pointer: 0x09d4daf4 ***
======= Backtrace: =========
/lib/libc.so.6(cfree+0x1bb)[0xc4e19b]
/etc/httpd/modules/libphp5.so(_efree+0x87)[0x11291a7]
/etc/httpd/modules/libphp5.so(_zval_dtor_func+0x5a)[0x11412ea]
/etc/httpd/modules/libphp5.so[0x1187609]
/etc/httpd/modules/libphp5.so[0x11876dd]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so[0x1168aa8]
/etc/httpd/modules/libphp5.so(execute+0x188)[0x11613f8]
/etc/httpd/modules/libphp5.so(zend_execute_scripts+0x2f0)[0x11418b0]
/etc/httpd/modules/libphp5.so(php_execute_script+0x255)[0x11010f5]
/etc/httpd/modules/libphp5.so[0x11c47f5]
/usr/sbin/httpd(ap_run_handler+0x6d)[0x25c97d]
/usr/sbin/httpd(ap_invoke_handler+0x68)[0x260338]
/usr/sbin/httpd(ap_process_request+0x1ae)[0x26c3ae]
/usr/sbin/httpd[0x26918f]
/usr/sbin/httpd(ap_run_process_connection+0x6d)[0x26485d]
/usr/sbin/httpd(ap_process_connection+0x4c)[0x26495c]
/usr/sbin/httpd[0x270972]
/usr/sbin/httpd[0x270c81]
/usr/sbin/httpd(ap_mpm_run+0x913)[0x271683]
/usr/sbin/httpd(main+0x8b7)[0x248147]
/lib/libc.so.6(__libc_start_main+0xdc)[0xbf7dec]
/usr/sbin/httpd[0x247221]
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:06 52669849   /usr/lib/libaprutil-1.so.0.2.7
0012c000-0012d000 rw-p 0001b000 08:06 52669849   /usr/lib/libaprutil-1.so.0.2.7
0012d000-00221000 r-xp 00000000 08:06 16844304   /lib/libdb-4.3.so
00221000-00224000 rw-p 000f3000 08:06 16844304   /lib/libdb-4.3.so
00224000-00227000 r-xp 00000000 08:06 16842959   /lib/libuuid.so.1.2
00227000-00228000 rw-p 00003000 08:06 16842959   /lib/libuuid.so.1.2
00228000-00230000 r-xp 00000000 08:06 52669719   /usr/lib/libkrb5support.so.0.1
00230000-00231000 rw-p 00007000 08:06 52669719   /usr/lib/libkrb5support.so.0.1
00231000-00233000 r-xp 00000000 08:06 16844286   /lib/libkeyutils-1.2.so
00233000-00234000 rw-p 00001000 08:06 16844286   /lib/libkeyutils-1.2.so
00234000-00236000 r-xp 00000000 08:06 52921622   /usr/lib/httpd/modules/mod_auth_basic.so
00236000-00238000 rw-p 00001000 08:06 52921622   /usr/lib/httpd/modules/mod_auth_basic.so
00238000-00283000 r-xp 00000000 08:06 52669857   /usr/sbin/httpd
00283000-00285000 rw-p 0004a000 08:06 52669857   /usr/sbin/httpd
00285000-00288000 rw-p 00285000 00:00 0
00288000-00289000 r-xp 00000000 08:06 52921627   /usr/lib/httpd/modules/mod_authn_dbm.so
00289000-0028b000 rw-p 00000000 08:06 52921627   /usr/lib/httpd/modules/mod_authn_dbm.so
0028b000-0028c000 r-xp 00000000 08:06 52921649   /usr/lib/httpd/modules/mod_env.so
0028c000-0028e000 rw-p 00000000 08:06 52921649   /usr/lib/httpd/modules/mod_env.so
0028e000-002a1000 r-xp 00000000 08:06 16844275   /lib/libpthread-2.5.so
002a1000-002a2000 r--p 00012000 08:06 16844275   /lib/libpthread-2.5.so
002a2000-002a3000 rw-p 00013000 08:06 16844275   /lib/libpthread-2.5.so
002a3000-002a5000 rw-p 002a3000 00:00 0
002a5000-002c4000 r-xp 00000000 08:06 52669829   /usr/lib/libpq.so.4.1
002c4000-002c5000 rw-p 0001f000 08:06 52669829   /usr/lib/libpq.so.4.1
002c5000-002d4000 r-xp 00000000 08:06 16843031   /lib/libresolv-2.5.so
002d4000-002d5000 r--p 0000e000 08:06 16843031   /lib/libresolv-2.5.so
002d5000-002d6000 rw-p 0000f000 08:06 16843031   /lib/libresolv-2.5.so
002d6000-002d8000 rw-p 002d6000 00:00 0
002d8000-002de000 r-xp 00000000 08:06 52921630   /usr/lib/httpd/modules/mod_authnz_ldap.so
002de000-002e0000 rw-p 00005000 08:06 52921630   /usr/lib/httpd/modules/mod_authnz_ldap.so
002e0000-00319000 r-xp 00000000 08:06 52663998   /usr/lib/libldap-2.3.so.0.2.15
00319000-0031a000 rw-p 00039000 08:06 52663998   /usr/lib/libldap-2.3.so.0.2.15
0031a000-00355000 r-xp 00000000 08:06 16844298   /lib/libsepol.so.1
00355000-00356000 rw-p 0003a000 08:06 16844298   /lib/libsepol.so.1
00356000-00360000 rw-p 00356000 00:00 0
00360000-00378000 r-xp 00000000 08:06 52669106   /usr/lib/libsasl2.so.2.0.22
00378000-00379000 rw-p 00017000 08:06 52669106   /usr/lib/libsasl2.so.2.0.22
00379000-00409000 r-xp 00000000 08:06 52670754   /usr/lib/libkrb5.so.3.3
00409000-0040c000 rw-p 0008f000 08:06 52670754   /usr/lib/libkrb5.so.3.3
0040e000-00428000 r-xp 00000000 08:06 16844256   /lib/ld-2.5.so
00428000-00429000 r--p 00019000 08:06 16844256   /lib/ld-2.5.so
00429000-0042a000 rw-p 0001a000 08:06 16844256   /lib/ld-2.5.so
0042a000-0043d000 r-xp 00000000 08:06 16844259   /lib/libnsl-2.5.so
0043d000-0043e000 r--p 00012000 08:06 16844259   /lib/libnsl-2.5.so
0043e000-0043f000 rw-p 00013000 08:06 16844259   /lib/libnsl-2.5.so
0043f000-00441000 rw-p 0043f000 00:00 0
00443000-00468000 r-xp 00000000 08:06 52670747   /usr/lib/libk5crypto.so.3.1
00468000-00469000 rw-p 00025000 08:06 52670747   /usr/lib/libk5crypto.so.3.1
00469000-0047b000 r-xp 00000000 08:06 52659359   /usr/lib/libz.so.1.2.3
0047b000-0047c000 rw-p 00011000 08:06 52659359   /usr/lib/libz.so.1.2.3
0047c000-00481000 r-xp 00000000 08:06 52921660   /usr/lib/httpd/modules/mod_log_config.so
00481000-00483000 rw-p 00004000 08:06 52921660   /usr/lib/httpd/modules/mod_log_config.so
00483000-00488000 r-xp 00000000 08:06 52921665   /usr/lib/httpd/modules/mod_mime_magic.so
00488000-0048a000 rw-p 00004000 08:06 52921665   /usr/lib/httpd/modules/mod_mime_magic.so
0048a000-0048c000 r-xp 00000000 08:06 52921650   /usr/lib/httpd/modules/mod_expires.so
0048c000-0048e000 rw-p 00001000 08:06 52921650   /usr/lib/httpd/modules/mod_expires.so
0048e000-00491000 r-xp 00000000 08:06 52921654   /usr/lib/httpd/modules/mod_headers.so
00491000-00493000 rw-p 00002000 08:06 52921654   /usr/lib/httpd/modules/mod_headers.so
00493000-00495000 r-xp 00000000 08:06 52921680   /usr/lib/httpd/modules/mod_usertrack.so
00495000-00497000 rw-p 00001000 08:06 52921680   /usr/lib/httpd/modules/mod_usertrack.so
00497000-00499000 r-xp 00000000 08:06 52921674   /usr/lib/httpd/modules/mod_setenvif.so
00499000-0049b000 rw-p 00001000 08:06 52921674   /usr/lib/httpd/modules/mod_setenvif.so
0049b000-0049e000 r-xp 00000000 08:06 52921664   /usr/lib/httpd/modules/mod_mime.so
0049e000-004a0000 rw-p 00003000 08:06 52921664   /usr/lib/httpd/modules/mod_mime.so
004a0000-004b5000 r-xp 00000000 08:06 52921642   /usr/lib/httpd/modules/mod_dav.so
004b5000-004b7000 rw-p 00014000 08:06 52921642   /usr/lib/httpd/modules/mod_dav.so
004b7000-004bb000 r-xp 00000000 08:06 52921676   /usr/lib/httpd/modules/mod_status.so
004bb000-004bd000 rw-p 00003000 08:06 52921676   /usr/lib/httpd/modules/mod_status.so
004bd000-004c4000 r-xp 00000000 08:06 52921637   /usr/lib/httpd/modules/mod_autoindex.so
004c4000-004c6000 rw-p 00007000 08:06 52921637   /usr/lib/httpd/modules/mod_autoindex.so
004c6000-004c7000 r-xp 00000000 08:06 52921621   /usr/lib/httpd/modules/mod_asis.so
004c7000-004c9000 rw-p 00000000 08:06 52921621   /usr/lib/httpd/modules/mod_asis.so
004c9000-004cd000 r-xp 00000000 08:06 52921658   /usr/lib/httpd/modules/mod_info.so
004cd000-004cf000 rw-p 00003000 08:06 52921658   /usr/lib/httpd/modules/mod_info.so
004cf000-004d1000 r-xp 00000000 08:06 52921682   /usr/lib/httpd/modules/mod_vhost_alias.so
004d1000-004d3000 rw-p 00001000 08:06 52921682   /usr/lib/httpd/modules/mod_vhost_alias.so
004d3000-004da000 r-xp 00000000 08:06 52921666   /usr/lib/httpd/modules/mod_negotiation.so
004da000-004dc000 rw-p 00006000 08:06 52921666   /usr/lib/httpd/modules/mod_negotiation.so
004dc000-004dd000 r-xp 00000000 08:06 52921646   /usr/lib/httpd/modules/mod_dir.so
004dd000-004df000 rw-p 00001000 08:06 52921646   /usr/lib/httpd/modules/mod_dir.so
004df000-004e0000 r-xp 00000000 08:06 52921619   /usr/lib/httpd/modules/mod_actions.so
004e0000-004e2000 rw-p 00001000 08:06 52921619   /usr/lib/httpd/modules/mod_actions.so
004e2000-004e4000 r-xp 00000000 08:06 52921679   /usr/lib/httpd/modules/mod_userdir.so
004e4000-004e6000 rw-p 00001000 08:06 52921679   /usr/lib/httpd/modules/mod_userdir.so
004e6000-004e8000 r-xp 00000000 08:06 52921620   /usr/lib/httpd/modules/mod_alias.so
004e8000-004ea000 rw-p 00002000 08:06 52921620   /usr/lib/httpd/modules/mod_alias.so
004ea000-004f9000 r-xp 00000000 08:06 52921667   /usr/lib/httpd/modules/mod_proxy.so
004f9000-004fb000 rw-p 0000f000 08:06 52921667   /usr/lib/httpd/modules/mod_proxy.so
004fb000-004fd000 r-xp 00000000 08:06 52921670   /usr/lib/httpd/modules/mod_proxy_connect.so
004fd000-004ff000 rw-p 00001000 08:06 52921670   /usr/lib/httpd/modules/mod_proxy_connect.so
004ff000-0051b000 r-xp 00000000 08:06 16844305   /lib/libpcre.so.0.0.1
0051b000-0051c000 rw-p 0001c000 08:06 16844305   /lib/libpcre.so.0.0.1
0051c000-00523000 r-xp 00000000 08:06 52921671   /usr/lib/httpd/modules/mod_proxy_ftp.so
00523000-00525000 rw-p 00006000 08:06 52921671   /usr/lib/httpd/modules/mod_proxy_ftp.so
00525000-0052b000 r-xp 00000000 08:06 52921672   /usr/lib/httpd/modules/mod_proxy_http.so
0052b000-0052d000 rw-p 00005000 08:06 52921672   /usr/lib/httpd/modules/mod_proxy_http.so
0052d000-00534000 r-xp 00000000 08:06 52921638   /usr/lib/httpd/modules/mod_cache.so
00534000-00536000 rw-p 00006000 08:06 52921638   /usr/lib/httpd/modules/mod_cache.so
00536000-00537000 r-xp 00000000 08:06 52921677   /usr/lib/httpd/modules/mod_suexec.so
00537000-00539000 rw-p 00000000 08:06 52921677   /usr/lib/httpd/modules/mod_suexec.so
00539000-0053d000 r-xp 00000000 08:06 52921647   /usr/lib/httpd/modules/mod_disk_cache.so
0053d000-0053f000 rw-p 00003000 08:06 52921647   /usr/lib/httpd/modules/mod_disk_cache.so
0053f000-00541000 r-xp 00000000 08:06 52921652   /usr/lib/httpd/modules/mod_file_cache.so
00541000-00543000 rw-p 00001000 08:06 52921652   /usr/lib/httpd/modules/mod_file_cache.so
00543000-00548000 r-xp 00000000 08:06 52921663   /usr/lib/httpd/modules/mod_mem_cache.so
00548000-0054a000 rw-p 00004000 08:06 52921663   /usr/lib/httpd/modules/mod_mem_cache.so
0054a000-0054f000 r-xp 00000000 08:06 52921640   /usr/lib/httpd/modules/mod_cgi.so
0054f000-00551000 rw-p 00004000 08:06 52921640   /usr/lib/httpd/modules/mod_cgi.so
00551000-00584000 r-xp 00000000 08:06 52671478   /usr/lib/sse2/libgmp.so.3.3.3
00584000-00585000 rw-p 00032000 08:06 52671478   /usr/lib/sse2/libgmp.so.3.3.3
00585000-00590000 r-xp 00000000 08:06 16844301   /lib/libgcc_s-4.1.2-20080102.so.1
00590000-00591000 rw-p 0000a000 08:06 16844301   /lib/libgcc_s-4.1.2-20080102.so.1
00593000-005c0000 r-xp 00000000 08:06 52670808   /usr/lib/libgssapi_krb5.so.2.2
005c0000-005c1000 rw-p 0002d000 08:06 52670808   /usr/lib/libgssapi_krb5.so.2.2
005c2000-005cf000 r-xp 00000000 08:06 52666461   /usr/lib/liblber-2.3.so.0.2.15
005cf000-005d0000 rw-p 0000c000 08:06 52666461   /usr/lib/liblber-2.3.so.0.2.15
005d0000-005e0000 r-xp 00000000 08:06 52664228   /usr/lib/libbz2.so.1.0.3
005e0000-005e1000 rw-p 00010000 08:06 52664228   /usr/lib/libbz2.so.1.0.3
005e1000-005e8000 r-xp 00000000 08:06 52921668   /usr/lib/httpd/modules/mod_proxy_ajp.so
005e8000-005ea000 rw-p 00006000 08:06 52921668   /usr/lib/httpd/modules/mod_proxy_ajp.so
005ea000-005ee000 r-xp 00000000 08:06 52671473   /usr/lib/libdistcache.so.1.0.1
005ee000-005ef000 rw-p 00003000 08:06 52671473   /usr/lib/libdistcache.so.1.0.1
005ef000-005f7000 r-xp 00000000 08:06 52670131   /usr/lib/libnal.so.1.0.1
005f7000-005f8000 rw-p 00007000 08:06 52670131   /usr/lib/libnal.so.1.0.1
005f8000-005fb000 r-xp 00000000 08:06 52857818   /usr/lib/php/modules/fileinfo.so
005fb000-005fc000 rw-p 00002000 08:06 52857818   /usr/lib/php/modules/fileinfo.so
005fc000-00601000 r-xp 00000000 08:06 52857813   /usr/lib/php/modules/pdo_sqlite.so
00601000-00602000 rw-p 00004000 08:06 52857813   /usr/lib/php/modules/pdo_sqlite.so
00602000-0060d000 r-xp 00000000 08:06 52921643   /usr/lib/httpd/modules/mod_dav_fs.so
0060d000-0060f000 rw-p 0000a000 08:06 52921643   /usr/lib/httpd/modules/mod_dav_fs.so
0060f000-0061d000 r-xp 00000000 08:06 52663481   /usr/lib/libmagic.so.1.0.0
0061d000-0061e000 rw-p 0000e000 08:06 52663481   /usr/lib/libmagic.so.1.0.0
00622000-0062b000 r-xp 00000000 08:06 16842793   /lib/libnss_files-2.5.so
0062b000-0062c000 r--p 00008000 08:06 16842793   /lib/libnss_files-2.5.so
0062c000-0062d000 rw-p 00009000 08:06 16842793   /lib/libnss_files-2.5.so
0062d000-00634000 r-xp 00000000 08:06 52857811   /usr/lib/php/modules/dbase.so
00634000-00635000 rw-p 00006000 08:06 52857811   /usr/lib/php/modules/dbase.so
00635000-00640000 r-xp 00000000 08:06 52857814   /usr/lib/php/modules/mysql.so
00640000-00641000 rw-p 0000b000 08:06 52857814   /usr/lib/php/modules/mysql.so
00646000-0064c000 r-xp 00000000 08:06 52857816   /usr/lib/php/modules/pdo_mysql.so
0064c000-0064d000 rw-p 00005000 08:06 52857816   /usr/lib/php/modules/pdo_mysql.so
00655000-00657000 r-xp 00000000 08:06 52921639   /usr/lib/httpd/modules/mod_cern_meta.so
00657000-00659000 rw-p 00001000 08:06 52921639   /usr/lib/httpd/modules/mod_cern_meta.so
00659000-0066c000 r-xp 00000000 08:06 52857812   /usr/lib/php/modules/pdo.so
0066c000-0066e000 rw-p 00012000 08:06 52857812   /usr/lib/php/modules/pdo.so
00672000-00679000 r-xp 00000000 08:06 16844276   /lib/librt-2.5.so
00679000-0067a000 r--p 00006000 08:06 16844276   /lib/librt-2.5.so
0067a000-0067b000 rw-p 00007000 08:06 16844276   /lib/librt-2.5.so
0067e000-00680000 r-xp 00000000 08:06 16844269   /lib/libdl-2.5.so
00680000-00681000 r--p 00001000 08:06 16844269   /lib/libdl-2.5.so
00681000-00682000 rw-p 00002000 08:06 16844269   /lib/libdl-2.5.so
00682000-006bd000 r-xp 00000000 08:06 52671188   /usr/lib/libcurl.so.3.0.0
006bd000-006be000 rw-p 0003b000 08:06 52671188   /usr/lib/libcurl.so.3.0.0
006be000-006ee000 r-xp 00000000 08:06 52670247   /usr/lib/libidn.so.11.5.19
006ee000-006ef000 rw-p 0002f000 08:06 52670247   /usr/lib/libidn.so.11.5.19
00706000-0070c000 r-xp 00000000 08:06 52921623   /usr/lib/httpd/modules/mod_auth_digest.so
0070c000-0070e000 rw-p 00006000 08:06 52921623   /usr/lib/httpd/modules/mod_auth_digest.so
0070e000-00726000 r-xp 00000000 08:06 52857815   /usr/lib/php/modules/mysqli.so
00726000-00728000 rw-p 00018000 08:06 52857815   /usr/lib/php/modules/mysqli.so
0072e000-0084b000 r-xp 00000000 08:06 16844284   /lib/libcrypto.so.0.9.8b
0084b000-0085e000 rw-p 0011c000 08:06 16844284   /lib/libcrypto.so.0.9.8b
0085e000-00861000 rw-p 0085e000 00:00 0
00879000-0087d000 r-xp 00000000 08:06 52921645   /usr/lib/httpd/modules/mod_deflate.so
0087d000-0087f000 rw-p 00003000 08:06 52921645   /usr/lib/httpd/modules/mod_deflate.so
00881000-00882000 r-xp 00881000 00:00 0          [vdso]
00882000-008a9000 r-xp 00000000 08:06 52922693   /usr/lib/httpd/modules/mod_ssl.so
008a9000-008ab000 rw-p 00027000 08:06 52922693   /usr/lib/httpd/modules/mod_ssl.so
008ab000-008ad000 rw-p 008ab000 00:00 0
008c3000-008e2000 r-xp 00000000 08:06 16842961   /lib/libexpat.so.0.5.0
008e2000-008e4000 rw-p 0001e000 08:06 16842961   /lib/libexpat.so.0.5.0
008e4000-00916000 r-xp 00000000 08:06 52857817   /usr/lib/php/modules/gd.so
00916000-00936000 rw-p 00031000 08:06 52857817   /usr/lib/php/modules/gd.so
00936000-0094a000 rw-p 00936000 00:00 0
00959000-0099a000 r-xp 00000000 08:06 16843617   /lib/libssl.so.0.9.8b
0099a000-0099e000 rw-p 00040000 08:06 16843617   /lib/libssl.so.0.9.8b
009a0000-009a9000 r-xp 00000000 08:06 52921657   /usr/lib/httpd/modules/mod_include.so
009a9000-009ab000 rw-p 00009000 08:06 52921657   /usr/lib/httpd/modules/mod_include.so
009ab000-009d0000 r-xp 00000000 08:06 52666993   /usr/lib/libpng12.so.0.10.0
009d0000-009d1000 rw-p 00024000 08:06 52666993   /usr/lib/libpng12.so.0.10.0
009de000-009e0000 r-xp 00000000 08:06 52921675   /usr/lib/httpd/modules/mod_speling.so
009e0000-009e2000 rw-p 00001000 08:06 52921675   /usr/lib/httpd/modules/mod_speling.so
009e2000-00a03000 r-xp 00000000 08:06 52671469   /usr/lib/libjpeg.so.62.0.0
00a03000-00a04000 rw-p 00020000 08:06 52671469   /usr/lib/libjpeg.so.62.0.0
00a06000-00a0f000 r-xp 00000000 08:06 16844279   /lib/libcrypt-2.5.so
00a0f000-00a10000 r--p 00008000 08:06 16844279   /lib/libcrypt-2.5.so
00a10000-00a11000 rw-p 00009000 08:06 16844279   /lib/libcrypt-2.5.so
00a11000-00a38000 rw-p 00a11000 00:00 0
00a38000-00af1000 r-xp 00000000 08:06 52670718   /usr/lib/libaspell.so.15.1.3
00af1000-00af5000 rw-p 000b8000 08:06 52670718   /usr/lib/libaspell.so.15.1.3
00af5000-00af9000 rw-p 00af5000 00:00 0
00b12000-00b13000 r-xp 00000000 08:06 52921625   /usr/lib/httpd/modules/mod_authn_anon.so
00b13000-00b15000 rw-p 00000000 08:06 52921625   /usr/lib/httpd/modules/mod_authn_anon.so
00b2e000-00b30000 r-xp 00000000 08:06 52921634   /usr/lib/httpd/modules/mod_authz_host.so
00b30000-00b32000 rw-p 00001000 08:06 52921634   /usr/lib/httpd/modules/mod_authz_host.so
00b5c000-00b67000 r-xp 00000000 08:06 52921659   /usr/lib/httpd/modules/mod_ldap.so
00b67000-00b69000 rw-p 0000a000 08:06 52921659   /usr/lib/httpd/modules/mod_ldap.so
00be2000-00d1f000 r-xp 00000000 08:06 16844266   /lib/libc-2.5.so
00d1f000-00d21000 r--p 0013c000 08:06 16844266   /lib/libc-2.5.so
00d21000-00d22000 rw-p 0013e000 08:06 16844266   /lib/libc-2.5.so
00d22000-00d25000 rw-p 00d22000 00:00 0
00d26000-00d34000 r-xp 00000000 08:06 52921673   /usr/lib/httpd/modules/mod_rewrite.so
00d34000-00d36000 rw-p 0000d000 08:06 52921673   /usr/lib/httpd/modules/mod_rewrite.so
00d72000-00d74000 r-xp 00000000 08:06 16844287   /lib/libcom_err.so.2.1
00d74000-00d75000 rw-p 00001000 08:06 16844287   /lib/libcom_err.so.2.1
00d9d000-00dfa000 r-xp 00000000 08:06 52663637   /usr/lib/libsqlite3.so.0.8.6
00dfa000-00dfc000 rw-p 0005c000 08:06 52663637   /usr/lib/libsqlite3.so.0.8.6
00e69000-00e8e000 r-xp 00000000 08:06 16844268   /lib/libm-2.5.so
00e8e000-00e8f000 r--p 00024000 08:06 16844268   /lib/libm-2.5.so
00e8f000-00e90000 rw-p 00025000 08:06 16844268   /lib/libm-2.5.so
00ec8000-00edb000 r-xp 00000000 08:06 3866824    /usr/lib/php/modules/apc.so
00edb000-00edc000 rw-p 00013000 08:06 3866824    /usr/lib/php/modules/apc.so
00edc000-00ee1000 rw-p 00edc000 00:00 0
00f1f000-00f20000 r-xp 00000000 08:06 52671477   /usr/lib/libpspell.so.15.1.3
00f20000-00f21000 rw-p 00000000 08:06 52671477   /usr/lib/libpspell.so.15.1.3
00f22000-00f38000 r-xp 00000000 08:06 16844299   /lib/libselinux.so.1
00f38000-00f3a000 rw-p 00015000 08:06 16844299   /lib/libselinux.so.1
00f5a000-00f81000 r-xp 00000000 08:06 52669012   /usr/lib/libapr-1.so.0.2.7
00f81000-00f82000 rw-p 00026000 08:06 52669012   /usr/lib/libapr-1.so.0.2.7
00f82000-01241000 r-xp 00000000 08:06 52922692   /usr/lib/httpd/modules/libphp5.so
01241000-01260000 rw-p 002bf000 08:06 52922692   /usr/lib/httpd/modules/libphp5.so
01260000-01268000 rw-p 01260000 00:00 0
01268000-013f9000 r-xp 00000000 08:06 52857819   /usr/lib/php/modules/mbstring.so
013f9000-0140c000 rw-p 00191000 08:06 52857819   /usr/lib/php/modules/mbstring.so
02a7d000-02afa000 r-xp 00000000 08:06 52670281   /usr/lib/libfreetype.so.6.3.10
02afa000-02afd000 rw-p 0007d000 08:06 52670281   /usr/lib/libfreetype.so.6.3.10
05f33000-06013000 r-xp 00000000 08:06 52669786   /usr/lib/libstdc++.so.6.0.8
06013000-06017000 r--p 000df000 08:06 52669786   /usr/lib/libstdc++.so.6.0.8
06017000-06018000 rw-p 000e3000 08:06 52669786   /usr/lib/libstdc++.so.6.0.8
06018000-0601e000 rw-p 06018000 00:00 0
06813000-0693f000 r-xp 00000000 08:06 52666505   /usr/lib/libxml2.so.2.6.26
0693f000-06944000 rw-p 0012c000 08:06 52666505   /usr/lib/libxml2.so.2.6.26
06944000-06945000 rw-p 06944000 00:00 0
0757a000-0769d000 r-xp 00000000 08:06 52954267   /usr/lib/mysql/libmysqlclient.so.15.0.0
0769d000-076df000 rw-p 00122000 08:06 52954267   /usr/lib/mysql/libmysqlclient.so.15.0.0
076df000-076e0000 rw-p 076df000 00:00 0
09892000-09ed5000 rw-p 09892000 00:00 0
afe63000-afe77000 rw-s 00000000 00:08 888627839  /dev/zero (deleted)
afe77000-afef5000 rw-s 00000000 00:08 1021609716  /dev/zero (deleted)
afef5000-b7ef5000 rw-s 00000000 08:06 29949955   /tmp/apc.RumXNU (deleted)
b7ef5000-b7f0e000 rw-s 00000000 00:08 1021609714  /dev/zero (deleted)
b7f0e000-b7f16000 rw-p b7f0e000 00:00 0
bfd91000-bfda7000 rw-p bfd91000 00:00 0          [stack]
[Wed Dec 03 11:27:48 2008] [notice] child pid 14537 exit signal Aborted (6)
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4 ***
[Wed Dec 03 11:27:54 2008] [notice] child pid 14543 exit signal Segmentation fault (11)
[Wed Dec 03 11:27:55 2008] [notice] child pid 14542 exit signal Segmentation fault (11)

Re: Seg-faults on highly-trafficed APC site

by KannanM@gmail.com (Kannan Muthukkaruppan) :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

We ran into something similar a while back. We were using 5.2.5 at that
time. See this thread:

<
http://markmail.org/message/bhy2kbtwrljecnna#query:kannan%20muthukkaruppan%20apc+page:1+mid:ajsctublcbbjvnax+state:results
>

This issue-- the issue with improper ref count handling in
zend_assign_to_variable when the left-hand side is a reference (is_ref
set)--- was fixed in a later version of PHP. Don't recall the exact version
though.

regards,
Kannan Muthukkaruppan

On Sun, Dec 7, 2008 at 2:32 PM, SinistralEnigma <sean@...> wrote:

>
> Hi,
> I had a decent number of segfaults that appeared to be caused by APC (I am
> running APC v3.0.19 and PHP v5.1.6). Rasmus recommended that I bump up the
> space allotted to APC for caching and that made the problem go away for
> about a month but it recently came back and caused my servers to crash
> again.
>
> I have APC running on two Apache servers (behind a squid caching server)
> and
> I watched apc_INFO.php closely for the whole month, checking several times
> a
> day.  Neither of the servers were using even 50% of the space allotted to
> them, and they both had a "Cache full count" of 0 the last time I checked
> (which would have been only a matter of hours before the error occurred).
>
> The only anomaly I saw ahead of time is that the one of the servers (where
> I
> hadn't restarted httpd during the whole 4 weeks) started to have values on
> the apc_INFO panel that were negative.  For example, the hits in the
> hits/misses were a large negative number - around 1.9 billion if I recall.
> The "Request Rate" was also negative.
>
> If it matters, this is being used for a MediaWiki v1.13.2 site.
>
> Any hints would be appreciated.  I've attached a good deal of what I feel
> is
> probably helpful debugging information below.
> Thanks for your time,
>  - Sean Colombo [LyricWiki.org]
>
>
>
>
> 01260000-01268000 rw-p 01260000 00:00 0
> 01268000-013f9000 r-xp 00000000 08:06 52857819
> /usr/lib/php/modules/mbstring.so
> 013f9000-0140c000 rw-p 00191000 08:06 52857819
> /usr/lib/php/modules/mbstring.so
> 02a7d000-02afa000 r-xp 00000000 08:06 52670281
> /usr/lib/libfreetype.so.6.3.10
> 02afa000-02afd000 rw-p 0007d000 08:06 52670281
> /usr/lib/libfreetype.so.6.3.10
>
....


>
> /usr/lib/mysql/libmysqlclient.so.15.0.0
> 0769d000-076df000 rw-p 00122000 08:06 52954267
> /usr/lib/mysql/libmysqlclient.so.15.0.0
> 076df000-076e0000 rw-p 076df000 00:00 0
> 09892000-09ed5000 rw-p 09892000 00:00 0
> afe63000-afe77000 rw-s 00000000 00:08 888627839  /dev/zero (deleted)
> afe77000-afef5000 rw-s 00000000 00:08 1021609716  /dev/zero (deleted)
> afef5000-b7ef5000 rw-s 00000000 08:06 29949955   /tmp/apc.RumXNU (deleted)
> b7ef5000-b7f0e000 rw-s 00000000 00:08 1021609714  /dev/zero (deleted)
> b7f0e000-b7f16000 rw-p b7f0e000 00:00 0
> bfd91000-bfda7000 rw-p bfd91000 00:00 0          [stack]
> [Wed Dec 03 11:27:48 2008] [notice] child pid 14537 exit signal Aborted (6)
> *** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x09d4daf4
> ***
> [Wed Dec 03 11:27:54 2008] [notice] child pid 14543 exit signal
> Segmentation
> fault (11)
> [Wed Dec 03 11:27:55 2008] [notice] child pid 14542 exit signal
> Segmentation
> fault (11)
>
> --
> View this message in context:
> http://www.nabble.com/Seg-faults-on-highly-trafficed-APC-site-tp20886787p20886787.html
> Sent from the Pecl - Dev mailing list archive at Nabble.com.
>
>
> --
> PECL development discussion Mailing List (http://pecl.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Re: Seg-faults on highly-trafficed APC site

by Xuefer :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

there was a known memory corrupt issue in 5.2.5, try upgrade or downgrade php

--
PECL development discussion Mailing List (http://pecl.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Seg-faults on highly-trafficed APC site

by SinistralEnigma :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

I'm actually on 5.1.6, so I guess downgrading probably won't help too much.
 I've tried upgrading to 5.2.x before and it didn't work well... I'll have
to give that another shot.

Recently I got another crash, but with completely different error messages.
Do these look as though they're caused by the same underlying problem (old
PHP version)?


[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/normal/UtfNormalDefines.php' (dev=2054 ino=0) was on
gc-list for 2591888 seconds
[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/WebStart.php' (dev=2054 ino=0) was on gc-list for
2591586 seconds
[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/SiteConfiguration.php' (dev=2054 ino=0) was on
gc-list for 2591567 seconds
[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/ProfilerStub.php' (dev=2054 ino=0) was on gc-list for
2591551 seconds
[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/Defines.php' (dev=2054 ino=0) was on gc-list for
2591488 seconds
[Sun Dec 14 15:46:00 2008] [apc-warning] GC cache entry '/var/www/html/
lyricwiki.org/includes/DefaultSettings.php' (dev=2054 ino=0) was on gc-list
for 2591281 seconds
[Sun Dec 14 15:46:00 2008] [apc-error] apc_sma_free: could not locate
address 0x9a50000





On Tue, Dec 9, 2008 at 6:08 AM, Xuefer <xuefer@...> wrote:

> there was a known memory corrupt issue in 5.2.5, try upgrade or downgrade
> php
>
Free embeddable forum powered by Nabble Forum Help