crashes when exiting octave and loading swig-wrapped code

[Soeren Sonnenburg-18]

Dear all,

I am seeing crashes when exiting octave when I use swig-wrapped code but
only when loading multiple modules (with potentially overlapping
variables/functions).

Well it seems to be some invalid delete/free occurring only when closing
octave (gdb & valgrind outputs follow):

(this happens when just *loading* multiple modules of the shogun toolbox
(http://www.shogun-toolbox.org)):

octave:1> init_shogun
octave:2>

Program received signal SIGSEGV, Segmentation fault.
malloc_consolidate (av=0x7ffff08c1a00) at malloc.c:4891
4891    malloc.c: No such file or directory.
        in malloc.c
(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7eab750 (LWP 22163)):
#0  malloc_consolidate (av=0x7ffff08c1a00) at malloc.c:4891
#1  0x00007ffff05e7e68 in _int_free (av=0x7ffff08c1a00, mem=0xc36ac0)
    at malloc.c:4782
#2  0x00007ffff05e81d6 in *__GI___libc_free (mem=0xc9f710) at
malloc.c:3625
#3  0x00007ffff14828b4 in ?? () from /usr/lib/libhdf5.so.6
#4  0x00007ffff1482924 in ?? () from /usr/lib/libhdf5.so.6
#5  0x00007ffff1485ba1 in H5FL_garbage_coll ()
from /usr/lib/libhdf5.so.6
#6  0x00007ffff1485fa8 in H5FL_term_interface ()
from /usr/lib/libhdf5.so.6
#7  0x00007ffff1392f85 in H5_term_library () from /usr/lib/libhdf5.so.6
#8  0x00007ffff05ab43d in *__GI_exit (status=0) at exit.c:75
#9  0x00007ffff73a3d5d in octave_main (argc=1, argv=0x7fffffffe428,
embedded=0)
    at octave.cc:890
#10 0x00007ffff05935c6 in __libc_start_main (main=0x400900 <main>,
argc=1,
    ubp_av=0x7fffffffe428, init=0x400920 <__libc_csu_init>,
    fini=<value optimized out>, rtld_fini=<value optimized out>,
    stack_end=0x7fffffffe418) at libc-start.c:222
#11 0x0000000000400839 in _start ()
(gdb)


initoctave:1> init_shogun
octave:2>
==22872== Invalid read of size 4
==22872==    at 0xBC3EA96: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
==22872==    by 0xC37743C: exit (exit.c:75)
==22872==    by 0x532CD5C: octave_main (octave.cc:890)
==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
==22872==  Address 0xfd364f0 is 16 bytes inside a block of size 36 free'd
==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
==22872==    by 0xC37743C: exit (exit.c:75)
==22872==    by 0x532CD5C: octave_main (octave.cc:890)
==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
==22872==
==22872== Invalid free() / delete / delete[]
==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
==22872==    by 0xC37743C: exit (exit.c:75)
==22872==    by 0x532CD5C: octave_main (octave.cc:890)
==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
==22872==  Address 0xfd364e0 is 0 bytes inside a block of size 36 free'd
==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
==22872==    by 0xC37743C: exit (exit.c:75)
==22872==    by 0x532CD5C: octave_main (octave.cc:890)
==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
==22872==
==22872==
==22872== HEAP SUMMARY:
==22872==     in use at exit: 37,844,498 bytes in 10,837 blocks
==22872==   total heap usage: 428,743 allocs, 417,946 frees, 67,396,361 bytes allocated
==22872==
==22872== LEAK SUMMARY:
==22872==    definitely lost: 116,835 bytes in 167 blocks
==22872==    indirectly lost: 31,677 bytes in 196 blocks
==22872==      possibly lost: 1,871,115 bytes in 2,937 blocks
==22872==    still reachable: 35,824,871 bytes in 7,537 blocks
==22872==         suppressed: 0 bytes in 0 blocks
==22872== Rerun with --leak-check=full to see details of leaked memory
==22872==
==22872== For counts of detected and suppressed errors, rerun with: -v


I've exchanged emails with the swig-octave maintainer (CC'ed) who
suggests:

========snip==========
There have been a number of crashes (without good resolution) related
to the dynamic loading and clearing of .oct files in octave. These are
essentially bugs in octave itself, where if you load a module, create
some variables with it, clear the module, and then try to use the
variables, it will segfault etc. Those are likely related to your
problem (eg swig-octave registers a global variable that is not
cleared before the oct file itselt is cleared). Maybe a workaround is
to have a static destructor remove all swig-related global variables?
========snip==========

Does anyone here have an idea what the culprit is?

Soeren
--
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962


_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[John W. Eaton-3]

On 21-Oct-2009, Soeren Sonnenburg wrote:

| I am seeing crashes when exiting octave when I use swig-wrapped code but
| only when loading multiple modules (with potentially overlapping
| variables/functions).
|
| Well it seems to be some invalid delete/free occurring only when closing
| octave (gdb & valgrind outputs follow):
|
| (this happens when just *loading* multiple modules of the shogun toolbox
| (http://www.shogun-toolbox.org)):
|
| octave:1> init_shogun
| octave:2>
|
| Program received signal SIGSEGV, Segmentation fault.
| malloc_consolidate (av=0x7ffff08c1a00) at malloc.c:4891
| 4891    malloc.c: No such file or directory.
|         in malloc.c
| (gdb) thread apply all bt
|
| Thread 1 (Thread 0x7ffff7eab750 (LWP 22163)):
| #0  malloc_consolidate (av=0x7ffff08c1a00) at malloc.c:4891
| #1  0x00007ffff05e7e68 in _int_free (av=0x7ffff08c1a00, mem=0xc36ac0)
|     at malloc.c:4782
| #2  0x00007ffff05e81d6 in *__GI___libc_free (mem=0xc9f710) at
| malloc.c:3625
| #3  0x00007ffff14828b4 in ?? () from /usr/lib/libhdf5.so.6
| #4  0x00007ffff1482924 in ?? () from /usr/lib/libhdf5.so.6
| #5  0x00007ffff1485ba1 in H5FL_garbage_coll ()
| from /usr/lib/libhdf5.so.6
| #6  0x00007ffff1485fa8 in H5FL_term_interface ()
| from /usr/lib/libhdf5.so.6
| #7  0x00007ffff1392f85 in H5_term_library () from /usr/lib/libhdf5.so.6
| #8  0x00007ffff05ab43d in *__GI_exit (status=0) at exit.c:75
| #9  0x00007ffff73a3d5d in octave_main (argc=1, argv=0x7fffffffe428,
| embedded=0)
|     at octave.cc:890
| #10 0x00007ffff05935c6 in __libc_start_main (main=0x400900 <main>,
| argc=1,
|     ubp_av=0x7fffffffe428, init=0x400920 <__libc_csu_init>,
|     fini=<value optimized out>, rtld_fini=<value optimized out>,
|     stack_end=0x7fffffffe418) at libc-start.c:222
| #11 0x0000000000400839 in _start ()
| (gdb)
|
|
| initoctave:1> init_shogun
| octave:2>
| ==22872== Invalid read of size 4
| ==22872==    at 0xBC3EA96: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
| ==22872==    by 0xC37743C: exit (exit.c:75)
| ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
| ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
| ==22872==  Address 0xfd364f0 is 16 bytes inside a block of size 36 free'd
| ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
| ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
| ==22872==    by 0xC37743C: exit (exit.c:75)
| ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
| ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
| ==22872==
| ==22872== Invalid free() / delete / delete[]
| ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
| ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
| ==22872==    by 0xC37743C: exit (exit.c:75)
| ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
| ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
| ==22872==  Address 0xfd364e0 is 0 bytes inside a block of size 36 free'd
| ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
| ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
| ==22872==    by 0xC37743C: exit (exit.c:75)
| ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
| ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
| ==22872==
| ==22872==
| ==22872== HEAP SUMMARY:
| ==22872==     in use at exit: 37,844,498 bytes in 10,837 blocks
| ==22872==   total heap usage: 428,743 allocs, 417,946 frees, 67,396,361 bytes allocated
| ==22872==
| ==22872== LEAK SUMMARY:
| ==22872==    definitely lost: 116,835 bytes in 167 blocks
| ==22872==    indirectly lost: 31,677 bytes in 196 blocks
| ==22872==      possibly lost: 1,871,115 bytes in 2,937 blocks
| ==22872==    still reachable: 35,824,871 bytes in 7,537 blocks
| ==22872==         suppressed: 0 bytes in 0 blocks
| ==22872== Rerun with --leak-check=full to see details of leaked memory
| ==22872==
| ==22872== For counts of detected and suppressed errors, rerun with: -v
|
|
| I've exchanged emails with the swig-octave maintainer (CC'ed) who
| suggests:
|
| ========snip==========
| There have been a number of crashes (without good resolution) related
| to the dynamic loading and clearing of .oct files in octave. These are
| essentially bugs in octave itself, where if you load a module, create
| some variables with it, clear the module, and then try to use the
| variables, it will segfault etc. Those are likely related to your
| problem (eg swig-octave registers a global variable that is not
| cleared before the oct file itselt is cleared). Maybe a workaround is
| to have a static destructor remove all swig-related global variables?
| ========snip==========
|
| Does anyone here have an idea what the culprit is?

Yes, it's a known problem that you can't load a .oct file that creates
a variable (or is it just a user-defined class?) then clear the .oct
file and expect to continue to use the variables that were created.
I think you can avoid the problem by locking the .oct file that
creates the object.

We've had some discussions about making Octave delay unloading the
file until all objects created by it are deleted.  That might be
better, but so far no one has done the work.

jwe
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Soeren Sonnenburg-18]

On Fri, 2009-10-23 at 22:05 -0400, John W. Eaton wrote:
> On 21-Oct-2009, Soeren Sonnenburg wrote:

Dear John,

thanks for the answer!

> | I am seeing crashes when exiting octave when I use swig-wrapped code but
> | only when loading multiple modules (with potentially overlapping
> | variables/functions).
> |
> | Well it seems to be some invalid delete/free occurring only when closing
> | octave (gdb & valgrind outputs follow):
> |
> | (this happens when just *loading* multiple modules of the shogun toolbox
> | (http://www.shogun-toolbox.org)):
> |[...]
> | initoctave:1> init_shogun
> | octave:2>
> | ==22872== Invalid read of size 4
> | ==22872==    at 0xBC3EA96: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
> | ==22872==    by 0xC37743C: exit (exit.c:75)
> | ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
> | ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
> | ==22872==  Address 0xfd364f0 is 16 bytes inside a block of size 36 free'd
> | ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
> | ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
> | ==22872==    by 0xC37743C: exit (exit.c:75)
> | ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
> | ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
> | ==22872==
> | ==22872== Invalid free() / delete / delete[]
> | ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
> | ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
> | ==22872==    by 0xC37743C: exit (exit.c:75)
> | ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
> | ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
> | ==22872==  Address 0xfd364e0 is 0 bytes inside a block of size 36 free'd
> | ==22872==    at 0x4C21ACE: operator delete(void*) (vg_replace_malloc.c:346)
> | ==22872==    by 0xBC3EAA8: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (in /usr/lib/libstdc++.so.6.0.13)
> | ==22872==    by 0xC37743C: exit (exit.c:75)
> | ==22872==    by 0x532CD5C: octave_main (octave.cc:890)
> | ==22872==    by 0xC35F5C5: (below main) (libc-start.c:222)
> [...]
> | ========snip==========
> | There have been a number of crashes (without good resolution) related
> | to the dynamic loading and clearing of .oct files in octave. These are
> | essentially bugs in octave itself, where if you load a module, create
> | some variables with it, clear the module, and then try to use the
> | variables, it will segfault etc. Those are likely related to your
> | problem (eg swig-octave registers a global variable that is not
> | cleared before the oct file itselt is cleared). Maybe a workaround is
> | to have a static destructor remove all swig-related global variables?
> | ========snip==========
> |
> | Does anyone here have an idea what the culprit is?
>
> Yes, it's a known problem that you can't load a .oct file that creates
> a variable (or is it just a user-defined class?) then clear the .oct
> file and expect to continue to use the variables that were created.
> I think you can avoid the problem by locking the .oct file that
> creates the object.

How could one lock the .oct file? It sounds to me that I *may* be lucky
when I reshuffle the order in which I load oct files...

> We've had some discussions about making Octave delay unloading the
> file until all objects created by it are deleted.  That might be
> better, but so far no one has done the work.

Sounds reasonable, does octave currently have any means to detect that a
variable or class is defined in a certain file?

Soeren
--
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962



_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Jaroslav Hajek-2]

On Sat, Oct 24, 2009 at 3:05 AM, John W. Eaton <jwe@...> wrote:

> On 21-Oct-2009, Soeren Sonnenburg wrote:
>
> | I am seeing crashes when exiting octave when I use swig-wrapped code but
> | only when loading multiple modules (with potentially overlapping
> | variables/functions).
> |
> | Well it seems to be some invalid delete/free occurring only when closing
> | octave (gdb & valgrind outputs follow):
> |
> | (this happens when just *loading* multiple modules of the shogun toolbox
> | (http://www.shogun-toolbox.org)):
> |

> Yes, it's a known problem that you can't load a .oct file that creates
> a variable (or is it just a user-defined class?) then clear the .oct
> file and expect to continue to use the variables that were created.
> I think you can avoid the problem by locking the .oct file that
> creates the object.
>
> We've had some discussions about making Octave delay unloading the
> file until all objects created by it are deleted.  That might be
> better, but so far no one has done the work.
>
> jwe

I inspected the code a bit, and the basic obstacle I see is the
octave_shlib::close method, which runs a hook function for each
registered function, and then forces dlclose() (or equivalent), even
though there may still be references to the shlib object.

John, can you give me a clue why this is so?

I would imagine a more straightforward implementation of octave_shlib,
being merely a reference-counted pointer to the library handle (or
zero). Then, dlclose would simply be called when the reference count
drops to zero - similar to array and all the other reference-counted
stuff.

If this were true, classes defined via .oct files could simply attach
a copy of the originating octave_shlib into the class, ensuring that
the library won't be unloaded.
This could result in multiple loads of the same file into memory, if
it was modified after loading, but I didn't find any information that
this is not possible.

Unless I'm missing something important, I'd volunteer for this job.
Any comments?

--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Soeren Sonnenburg-18]

On Tue, 2009-11-03 at 13:46 +0100, Jaroslav Hajek wrote:

> On Sat, Oct 24, 2009 at 3:05 AM, John W. Eaton <jwe@...> wrote:
> > On 21-Oct-2009, Soeren Sonnenburg wrote:
> >
> > | I am seeing crashes when exiting octave when I use swig-wrapped code but
> > | only when loading multiple modules (with potentially overlapping
> > | variables/functions).
> > |
> > | Well it seems to be some invalid delete/free occurring only when closing
> > | octave (gdb & valgrind outputs follow):
> > |
> > | (this happens when just *loading* multiple modules of the shogun toolbox
> > | (http://www.shogun-toolbox.org)):
> > |
>
> > Yes, it's a known problem that you can't load a .oct file that creates
> > a variable (or is it just a user-defined class?) then clear the .oct
> > file and expect to continue to use the variables that were created.
> > I think you can avoid the problem by locking the .oct file that
> > creates the object.
> >
> > We've had some discussions about making Octave delay unloading the
> > file until all objects created by it are deleted.  That might be
> > better, but so far no one has done the work.
> >
> > jwe
>
> I inspected the code a bit, and the basic obstacle I see is the
> octave_shlib::close method, which runs a hook function for each
> registered function, and then forces dlclose() (or equivalent), even
> though there may still be references to the shlib object.
>
> John, can you give me a clue why this is so?
>
> I would imagine a more straightforward implementation of octave_shlib,
> being merely a reference-counted pointer to the library handle (or
> zero). Then, dlclose would simply be called when the reference count
> drops to zero - similar to array and all the other reference-counted
> stuff.
>
> If this were true, classes defined via .oct files could simply attach
> a copy of the originating octave_shlib into the class, ensuring that
> the library won't be unloaded.
> This could result in multiple loads of the same file into memory, if
> it was modified after loading, but I didn't find any information that
> this is not possible.
>
> Unless I'm missing something important, I'd volunteer for this job.
> Any comments?

From my outsiders viewpoint this sounds very reasonable (and is what I
do here too in shogun). But it is John here who would need to decide...

Soeren
--
For the one fact about the future of which we can be certain is that it
will be utterly fantastic. -- Arthur C. Clarke, 1962


_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[John W. Eaton-3]

On  3-Nov-2009, Jaroslav Hajek wrote:

| On Sat, Oct 24, 2009 at 3:05 AM, John W. Eaton <jwe@...> wrote:
| > On 21-Oct-2009, Soeren Sonnenburg wrote:
| >
| > | I am seeing crashes when exiting octave when I use swig-wrapped code but
| > | only when loading multiple modules (with potentially overlapping
| > | variables/functions).
| > |
| > | Well it seems to be some invalid delete/free occurring only when closing
| > | octave (gdb & valgrind outputs follow):
| > |
| > | (this happens when just *loading* multiple modules of the shogun toolbox
| > | (http://www.shogun-toolbox.org)):
| > |
|
| > Yes, it's a known problem that you can't load a .oct file that creates
| > a variable (or is it just a user-defined class?) then clear the .oct
| > file and expect to continue to use the variables that were created.
| > I think you can avoid the problem by locking the .oct file that
| > creates the object.
| >
| > We've had some discussions about making Octave delay unloading the
| > file until all objects created by it are deleted.  That might be
| > better, but so far no one has done the work.
| >
| > jwe
|
| I inspected the code a bit, and the basic obstacle I see is the
| octave_shlib::close method, which runs a hook function for each
| registered function, and then forces dlclose() (or equivalent), even
| though there may still be references to the shlib object.
|
| John, can you give me a clue why this is so?

Let's move this discussion to the maintainers list.

jwe

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave