DBD mysql driver segfault on cleanup when used together with legacy modules in apache using mysql

[Viktor Griph-4]

There is a possiblility that the mysql library has been unloaded by
the DSO loader before the cleanup function that tries to call
mysql_thread_end() is called, casusing a segmentation fault. In my
particular usecase I've been trying to use a module using a dbd
backend for mysql together with the module mod_auth_mysql. The mysql
library is dynamically linked when the mod_auth_mysql is loaded, and
is unlikned before the cleanup in mod_dbd_mysql is called, leaving a
call to an unloaded library.


Backtrace of the crash:
[Switching to Thread 0xb7d02b50 (LWP 16503)]
0xb7add81d in ?? ()
(gdb) bt
#0  0xb7add81d in ?? ()
#1  0xb7c44787 in thread_end (data=0x0) at dbd/apr_dbd_mysql.c:1260
#2  0xb7eb9fcb in apr_pool_destroy (pool=0x954a090)
    at memory/unix/apr_pools.c:2314
#3  0xb7eba22d in apr_pool_terminate () at memory/unix/apr_pools.c:605
#4  0xb7ebc16a in apr_terminate () at misc/unix/start.c:82
#5  0x08065a59 in destroy_and_exit_process (process=0x954e130,
    process_exit_value=0) at main.c:271
#6  0x08066b78 in main (argc=Cannot access memory at address 0x0
) at main.c:654

/Viktor

[Bojan Smojver]

On Wed, 2009-10-28 at 11:41 +0200, Viktor Griph wrote:

> In my
> particular usecase I've been trying to use a module using a dbd
> backend for mysql together with the module mod_auth_mysql. The mysql
> library is dynamically linked when the mod_auth_mysql is loaded, and
> is unlikned before the cleanup in mod_dbd_mysql is called, leaving a
> call to an unloaded library.

Hmm, that's weird. That cleanup should be against the same pool that DSO
was registered against.

Which version of APU is this?

--
Bojan

[Viktor Griph-4]

2009/10/28 Bojan Smojver <bojan@...>:

> On Wed, 2009-10-28 at 11:41 +0200, Viktor Griph wrote:
>
>> In my
>> particular usecase I've been trying to use a module using a dbd
>> backend for mysql together with the module mod_auth_mysql. The mysql
>> library is dynamically linked when the mod_auth_mysql is loaded, and
>> is unlikned before the cleanup in mod_dbd_mysql is called, leaving a
>> call to an unloaded library.
>
> Hmm, that's weird. That cleanup should be against the same pool that DSO
> was registered against.
>
> Which version of APU is this?
>
> --
> Bojan
>
>

$ apu-1-config --version
1.3.9

I'm using a gentoo system with apr-1.3.8 [USE="urandom -debug -doc
-older-kernels-compatibility"] and apr-util-1.3.9 [USE="gdbm mysql
-berkdb -doc -freetds -ldap -odbc -postgres -sqlite -sqlite3"].

Apache is

www-servers/apache-2.2.11-r2  USE="ssl -debug -doc -ldap (-selinux)
-sni -static -suexec -threads" APACHE2_MODULES="actions alias
auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default
authn_file authz_dbm authz_default authz_groupfile authz_host
authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate
dir disk_cache env expires ext_filter file_cache filter headers ident
imagemap include info log_config logio mem_cache mime mime_magic
negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http
rewrite setenvif speling status unique_id userdir usertrack
vhost_alias -asis -authn_alias -cern_meta -charset_lite -dumpio
-log_forensic -proxy_ftp -substitute -version" APACHE2_MPMS="-event
-itk -peruser -prefork -worker" 0 kB


And as far as I can tell it was enough to load both the mod_auth_mysql
modue and the mod_dbd module and speciy "DBDriver mysql" in the config
for apache2 -t to crash (after outputing Syntax OK).


/Viktor

[Bojan Smojver]

On Wed, 2009-10-28 at 12:40 +0200, Viktor Griph wrote:
> And as far as I can tell it was enough to load both the mod_auth_mysql
> modue and the mod_dbd module and speciy "DBDriver mysql" in the config
> for apache2 -t to crash (after outputing Syntax OK).

Right.

That other backtrace - that's not from Apache, is it?

--
Bojan

[Stefan Fritsch]

On Wednesday 28 October 2009, Viktor Griph wrote:
> There is a possiblility that the mysql library has been unloaded by
> the DSO loader before the cleanup function that tries to call
> mysql_thread_end() is called, casusing a segmentation fault. In my
> particular usecase I've been trying to use a module using a dbd
> backend for mysql together with the module mod_auth_mysql. The
>  mysql library is dynamically linked when the mod_auth_mysql is
>  loaded, and is unlikned before the cleanup in mod_dbd_mysql is
>  called, leaving a call to an unloaded library.

Not very likely in this case, but it may make sense to check it: You
do use libmysqlclient_r.so for both mod_dbd_mysql and mod_auth_mysql?
If you use libmysqlclient_r.so for one and libmysqlclient.so for the
other, you will get strange segfaults.

[Viktor Griph-4]

2009/10/28 Stefan Fritsch <sf@...>:

> On Wednesday 28 October 2009, Viktor Griph wrote:
>> There is a possiblility that the mysql library has been unloaded by
>> the DSO loader before the cleanup function that tries to call
>> mysql_thread_end() is called, casusing a segmentation fault. In my
>> particular usecase I've been trying to use a module using a dbd
>> backend for mysql together with the module mod_auth_mysql. The
>>  mysql library is dynamically linked when the mod_auth_mysql is
>>  loaded, and is unlikned before the cleanup in mod_dbd_mysql is
>>  called, leaving a call to an unloaded library.
>
> Not very likely in this case, but it may make sense to check it: You
> do use libmysqlclient_r.so for both mod_dbd_mysql and mod_auth_mysql?
> If you use libmysqlclient_r.so for one and libmysqlclient.so for the
> other, you will get strange segfaults.
>
>

Actually that was the case. Gentoo links mod_auth_mysql to
libmysqlclient, and apr-util with libmysqlclient_r. Manually
rebuilding  mod_auth_mysql linked to libmysqlclient_r solves the
problem. Thanks. I'll file a bug to Gentoo.

/Viktor

[Viktor Griph-4]

2009/10/28 Bojan Smojver <bojan@...>:
> On Wed, 2009-10-28 at 12:40 +0200, Viktor Griph wrote:
>> And as far as I can tell it was enough to load both the mod_auth_mysql
>> modue and the mod_dbd module and speciy "DBDriver mysql" in the config
>> for apache2 -t to crash (after outputing Syntax OK).
>
> Right.
>
> That other backtrace - that's not from Apache, is it?
>

It was from apache, but since the segfault was in an apr component I
though this was the rigt place to report it.

/Viktor

[Bojan Smojver]

On Wed, 2009-10-28 at 23:13 +0200, Viktor Griph wrote:
> Actually that was the case. Gentoo links mod_auth_mysql to
> libmysqlclient, and apr-util with libmysqlclient_r. Manually
> rebuilding  mod_auth_mysql linked to libmysqlclient_r solves the
> problem. Thanks.

Thanks for letting us know.

--
Bojan

[Bojan Smojver]

On Wed, 2009-10-28 at 23:15 +0200, Viktor Griph wrote:
> It was from apache, but since the segfault was in an apr component I
> though this was the rigt place to report it.

This was the right place to report it for sure. I was just trying to
figure out the sequence of events.

--
Bojan

[Stefan Fritsch]

On Wednesday 28 October 2009, Viktor Griph wrote:
> > Not very likely in this case, but it may make sense to check it:
> > You do use libmysqlclient_r.so for both mod_dbd_mysql and
> > mod_auth_mysql? If you use libmysqlclient_r.so for one and
> > libmysqlclient.so for the other, you will get strange segfaults.
>
> Actually that was the case. Gentoo links mod_auth_mysql to
> libmysqlclient, and apr-util with libmysqlclient_r. Manually
> rebuilding  mod_auth_mysql linked to libmysqlclient_r solves the
> problem. Thanks. I'll file a bug to Gentoo.

Ok. This is what happened:

start:
- linker loads mod_auth_mysql
- linker loads libmysqlclient
- linker resolves symbols in mod_auth_mysql against libmysqlclient
- linker loads mod_dbd_mysql/libaprutil1-dbd-mysql
- linker loads libmysqlclient_r
- linker resolves symbols in mod_dbd_mysql/libaprutil1-dbd-mysql
against libmysqlclient(!)

unload:
- linker unloads mod_auth_mysql and libmysqlclient
- mod_dbd_mysql crashes

This is this mysql bug: http://bugs.mysql.com/bug.php?id=32196

It would be nice if the Gentoo maintainer could add his voice there to
make the mysql people fix it.

Cheers,
Stefan