.oct files - keeping state across calls

[Christophe Tournery-2]

Hi all,

I am looking for a best practice advice regarding .oct files in octave  
3.2

Description:
------------
I am writing an oct file wrapper around a C library. The library  
performs block processing of audio files (similar to fft block  
processing). However the library needs to keep its internal state  
across calls to its "process" function. Therefore I am using an init  
function to initialize this internal state and return a handle to it,  
such that multiple instances of the library can be used at the same  
time. Usage in octave is in pseudo code:

        handle = mylib_init()
        for (...)
                y = mylib_process(handle, x)
                ...
        end
        mylib_destroy(handle)

The oct file (.cc) must keep track of the allocated handles and  
internal states which are C structures. One way of doing it is to get  
rid of the handle and return the internal state in octave's user  
space. However those can be a bit large and are useless to the caller  
so I'd rather keep them hidden inside the oct file, hence the handle  
which reduces complexity and improves performance by avoiding  
unnecessary copies of data.

Problem:
--------
Both mylib_init() and mylib_process() must share a list of allocated  
handles and internal states of mylib. In octave 3.0 I used a static  
std::map in my .cc file for the handles and that worked. In octave 3.2  
this fails. It seems that in 3.2 the .oct file is loaded twice, once  
for mylib_init() and once for mylib_process(), therefore the static  
variable trick does not work anymore since they have different  
addresses.

Any suggestions?

Thanks in advance
Christophe Tournery
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[dbateman]

Christophe Tournery-2 wrote:

Both mylib_init() and mylib_process() must share a list of allocated  
handles and internal states of mylib. In octave 3.0 I used a static  
std::map in my .cc file for the handles and that worked. In octave 3.2  
this fails. It seems that in 3.2 the .oct file is loaded twice, once  
for mylib_init() and once for mylib_process(), therefore the static  
variable trick does not work anymore since they have different  
addresses.

If you use the "mlock" function in the first call to the oct-file it should never be reloaded, and won't be cleared by a call to "clear all".. Does that work for you?

D.

[Christophe Tournery-2]

Thanks for the tip, I did not know mlock().

However the problem is still there. The list of handles has a  
different memory address when calling the mylib_init() function and  
when calling the mylib_process() function. Has there been major  
changes from 3.0 to 3.2 regarding loading oct files?

Do you guys encounter this design problem often or is there a very  
simple solution I do not see?

-c

On Jul 6, 2009, at 5:20 PM, dbateman wrote:

> Christophe Tournery-2 wrote:
>>
>> Both mylib_init() and mylib_process() must share a list of allocated
>> handles and internal states of mylib. In octave 3.0 I used a static
>> std::map in my .cc file for the handles and that worked. In octave  
>> 3.2
>> this fails. It seems that in 3.2 the .oct file is loaded twice, once
>> for mylib_init() and once for mylib_process(), therefore the static
>> variable trick does not work anymore since they have different
>> addresses.
>>
>
> If you use the "mlock" function in the first call to the oct-file it  
> should
> never be reloaded, and won't be cleared by a call to "clear all"..  
> Does that
> work for you?
>
> D.
>
> --
> View this message in context: http://www.nabble.com/.oct-files---keeping-state-across-calls-tp24354462p24357169.html
> Sent from the Octave - General mailing list archive at Nabble.com.
>
> _______________________________________________
> Help-octave mailing list
> Help-octave@...
> https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[dbateman2]

Christophe Tournery wrote:

> Thanks for the tip, I did not know mlock().
>
> However the problem is still there. The list of handles has a
> different memory address when calling the mylib_init() function and
> when calling the mylib_process() function. Has there been major
> changes from 3.0 to 3.2 regarding loading oct files?
>
> Do you guys encounter this design problem often or is there a very
> simple solution I do not see?
>

At this point I'm not sure there is a problem at all. Could you send a
complete simple example demonstrating the problem..

D.


--
David Bateman                                dbateman@...
35 rue Gambetta                              [phone] (Home)
92100 Boulogne-Billancourt FRANCE            [phone] (Mob)

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Christophe Tournery-2]

Attached is an example .cpp file that can be compile with mkoctfile,  
demonstrating the issue. I have included a simple makefile to compile  
it and create the necessary symlinks.

The code tries to store handles to a library in a map. The key to the  
map is passed back to octave such that the "user" can choose on which  
instance of the library a function is called.

Thanks for testing and I am curious to know if you guys think it is a  
bug in 3.2 or if it was "un-intentionally" working in 3.0.
Christophe Tournery


Testing in Octave 3.0.5:
------------------------

octave:1> h1 = mylib_init()
h1 = 0
octave:2> h2 = mylib_init()
h2 =  1
octave:3> mylib_destroy(h1)
octave:4> mylib_destroy(h2)


Testing in octave 3.2.0:
------------------------

octave:1> h1 = mylib_init()
h1 = 0
octave:2> h2 = mylib_init()
h2 =  1
octave:3> mylib_destroy(h1)
error: mylib_destroy: handle '0' does not exist!
octave:3> mylib_destroy(h2)
error: mylib_destroy: handle '1' does not exist!


On Jul 9, 2009, at 8:41 PM, David Bateman wrote:

> Christophe Tournery wrote:
>> Thanks for the tip, I did not know mlock().
>>
>> However the problem is still there. The list of handles has a  
>> different memory address when calling the mylib_init() function and  
>> when calling the mylib_process() function. Has there been major  
>> changes from 3.0 to 3.2 regarding loading oct files?
>>
>> Do you guys encounter this design problem often or is there a very  
>> simple solution I do not see?
>>
> At this point I'm not sure there is a problem at all. Could you send  
> a complete simple example demonstrating the problem..
>
> D.
>
>
> --
> David Bateman                                dbateman@...
> 35 rue Gambetta                              [phone] (Home)
> 92100 Boulogne-Billancourt FRANCE            [phone] (Mob)
>

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[John W. Eaton-3]

On 13-Jul-2009, Christophe Tournery wrote:

| Attached is an example .cpp file that can be compile with mkoctfile,  
| demonstrating the issue. I have included a simple makefile to compile  
| it and create the necessary symlinks.
|
| The code tries to store handles to a library in a map. The key to the  
| map is passed back to octave such that the "user" can choose on which  
| instance of the library a function is called.
|
| Thanks for testing and I am curious to know if you guys think it is a  
| bug in 3.2 or if it was "un-intentionally" working in 3.0.
| Christophe Tournery
|
|
| Testing in Octave 3.0.5:
| ------------------------
|
| octave:1> h1 = mylib_init()
| h1 = 0
| octave:2> h2 = mylib_init()
| h2 =  1
| octave:3> mylib_destroy(h1)
| octave:4> mylib_destroy(h2)
|
|
| Testing in octave 3.2.0:
| ------------------------
|
| octave:1> h1 = mylib_init()
| h1 = 0
| octave:2> h2 = mylib_init()
| h2 =  1
| octave:3> mylib_destroy(h1)
| error: mylib_destroy: handle '0' does not exist!
| octave:3> mylib_destroy(h2)
| error: mylib_destroy: handle '1' does not exist!

I can't duplicate this problem with Octave 3.2.0 on my system.

jwe
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Christophe Tournery-2]

On Jul 13, 2009, at 11:41 PM, John W. Eaton wrote:

> On 13-Jul-2009, Christophe Tournery wrote:
>
> | Attached is an example .cpp file that can be compile with mkoctfile,
> | demonstrating the issue. I have included a simple makefile to  
> compile
> | it and create the necessary symlinks.
> |
> | The code tries to store handles to a library in a map. The key to  
> the
> | map is passed back to octave such that the "user" can choose on  
> which
> | instance of the library a function is called.
> |
> | Thanks for testing and I am curious to know if you guys think it  
> is a
> | bug in 3.2 or if it was "un-intentionally" working in 3.0.
> | Christophe Tournery
> |
> |
> | Testing in Octave 3.0.5:
> | ------------------------
> |
> | octave:1> h1 = mylib_init()
> | h1 = 0
> | octave:2> h2 = mylib_init()
> | h2 =  1
> | octave:3> mylib_destroy(h1)
> | octave:4> mylib_destroy(h2)
> |
> |
> | Testing in octave 3.2.0:
> | ------------------------
> |
> | octave:1> h1 = mylib_init()
> | h1 = 0
> | octave:2> h2 = mylib_init()
> | h2 =  1
> | octave:3> mylib_destroy(h1)
> | error: mylib_destroy: handle '0' does not exist!
> | octave:3> mylib_destroy(h2)
> | error: mylib_destroy: handle '1' does not exist!
>
> I can't duplicate this problem with Octave 3.2.0 on my system.
>
> jwe

Could it be an issue with my build from MacPorts then? (Mac OS X  
10.5.7, macports 1.710)
I see that the Portfile configures octave with:

configure.args      --enable-shared \
                     --enable-dl \
                     --disable-static \
                     --with-hdf5 \
                     --with-fftw \
                     --with-blas="-framework Accelerate" \
                     --enable-static \
                     --enable-readline \
                     --with-zlib \
                     --with-glpk \
                     --with-curl \
                     --with-lapack \
                     --with-umfpack \
                     --with-colamd \
                     --with-ccolamd \
                     --with-cholmod \
                     --with-cxsparse

Christophe
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[John W. Eaton-3]

On 14-Jul-2009, Christophe Tournery wrote:

| Could it be an issue with my build from MacPorts then? (Mac OS X  
| 10.5.7, macports 1.710)
| I see that the Portfile configures octave with:
|
| configure.args      --enable-shared \
|                      --enable-dl \
|                      --disable-static \
|                      --with-hdf5 \
|                      --with-fftw \
|                      --with-blas="-framework Accelerate" \
|                      --enable-static \
|                      --enable-readline \
|                      --with-zlib \
|                      --with-glpk \
|                      --with-curl \
|                      --with-lapack \
|                      --with-umfpack \
|                      --with-colamd \
|                      --with-ccolamd \
|                      --with-cholmod \
|                      --with-cxsparse

I have no idea.  I don't have an OS X system, so I can't try to
reproduce the problem there.  Maybe someone else can.

Can you run Octave under gdb and set a breakpoint in the function
octave_dynamic_loader::do_load_oct and see where/why the .oct file is
loaded multiple times?  Looking at what happens in the function
octave_dyld_shlib::open might also give you some clues.

Also, can you try to use a PKG_ADD file with the lines

  autoload ("mylib_init", "mylib.oct");
  autoload ("mylib_desstroy", "mylib.oct");
  autoload ("mylib_process", "mylib.oct");

instead of using symbolic links?  Does that change anything?

Are the timestamps on your .oct files somehow in the future?

jwe
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[John W. Eaton-3]

On 14-Jul-2009, Christophe Tournery wrote:

| configure.args      --enable-shared \
|                      --enable-dl \
|                      --disable-static \
|                      --with-hdf5 \
|                      --with-fftw \
|                      --with-blas="-framework Accelerate" \
|                      --enable-static \
|                      --enable-readline \
|                      --with-zlib \
|                      --with-glpk \
|                      --with-curl \
|                      --with-lapack \
|                      --with-umfpack \
|                      --with-colamd \
|                      --with-ccolamd \
|                      --with-cholmod \
|                      --with-cxsparse

You have both --disable-static and --enable-static?  Is that intended?
The default is --disable-static.  Other than that and your --with-blas
option, I think all these are the defaults, so there is no need to
specify them explicitly.

jwe
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Christophe Tournery-2]

Thanks, I'll pass that info to the maintainers of the Portfile.
-c

On Jul 14, 2009, at 11:07 AM, John W. Eaton wrote:

> On 14-Jul-2009, Christophe Tournery wrote:
>
> | configure.args      --enable-shared \
> |                      --enable-dl \
> |                      --disable-static \
> |                      --with-hdf5 \
> |                      --with-fftw \
> |                      --with-blas="-framework Accelerate" \
> |                      --enable-static \
> |                      --enable-readline \
> |                      --with-zlib \
> |                      --with-glpk \
> |                      --with-curl \
> |                      --with-lapack \
> |                      --with-umfpack \
> |                      --with-colamd \
> |                      --with-ccolamd \
> |                      --with-cholmod \
> |                      --with-cxsparse
>
> You have both --disable-static and --enable-static?  Is that intended?
> The default is --disable-static.  Other than that and your --with-blas
> option, I think all these are the defaults, so there is no need to
> specify them explicitly.
>
> jwe

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[dbateman2]

Christophe Tournery wrote:

> Attached is an example .cpp file that can be compile with mkoctfile,
> demonstrating the issue. I have included a simple makefile to compile
> it and create the necessary symlinks.
>
> The code tries to store handles to a library in a map. The key to the
> map is passed back to octave such that the "user" can choose on which
> instance of the library a function is called.
>
> Thanks for testing and I am curious to know if you guys think it is a
> bug in 3.2 or if it was "un-intentionally" working in 3.0.
> Christophe Tournery
>
>
> Testing in Octave 3.0.5:
> ------------------------
>
> octave:1> h1 = mylib_init()
> h1 = 0
> octave:2> h2 = mylib_init()
> h2 =  1
> octave:3> mylib_destroy(h1)
> octave:4> mylib_destroy(h2)
>
>
> Testing in octave 3.2.0:
> ------------------------
>
> octave:1> h1 = mylib_init()
> h1 = 0
> octave:2> h2 = mylib_init()
> h2 =  1
> octave:3> mylib_destroy(h1)
> error: mylib_destroy: handle '0' does not exist!
> octave:3> mylib_destroy(h2)
> error: mylib_destroy: handle '1' does not exist!

I don't have a 3.2 installed but I can't generate this behavior with the
latest tip or 3.0.x.. Could someone else try on 3.2.1?

Cheers
David

--
David Bateman                                dbateman@...
35 rue Gambetta                              [phone] (Home)
92100 Boulogne-Billancourt FRANCE            [phone] (Mob)

_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave

[Christophe Tournery-2]

On Jul 14, 2009, at 11:02 AM, John W. Eaton wrote:

> On 14-Jul-2009, Christophe Tournery wrote:
>
> | Could it be an issue with my build from MacPorts then? (Mac OS X
> | 10.5.7, macports 1.710)
> | I see that the Portfile configures octave with:
> |
> | configure.args      --enable-shared \
> |                      --enable-dl \
> |                      --disable-static \
> |                      --with-hdf5 \
> |                      --with-fftw \
> |                      --with-blas="-framework Accelerate" \
> |                      --enable-static \
> |                      --enable-readline \
> |                      --with-zlib \
> |                      --with-glpk \
> |                      --with-curl \
> |                      --with-lapack \
> |                      --with-umfpack \
> |                      --with-colamd \
> |                      --with-ccolamd \
> |                      --with-cholmod \
> |                      --with-cxsparse
>
> I have no idea.  I don't have an OS X system, so I can't try to
> reproduce the problem there.  Maybe someone else can.
>
> Can you run Octave under gdb and set a breakpoint in the function
> octave_dynamic_loader::do_load_oct and see where/why the .oct file is
> loaded multiple times?  Looking at what happens in the function
> octave_dyld_shlib::open might also give you some clues.

Thanks for the pointers to the functions where to set break points.
Now that I recompiled octave 3.2.0 myself (and not from macports), I  
don't have the issue anymore and everything seems fine in the  
octave_dynamic_loader::do_load_oct function.

So I am closing this thread for now and will try to see if the  
configure args used by macports are at fault.

> Also, can you try to use a PKG_ADD file with the lines
>
>  autoload ("mylib_init", "mylib.oct");
>  autoload ("mylib_desstroy", "mylib.oct");
>  autoload ("mylib_process", "mylib.oct");
>
> instead of using symbolic links?  Does that change anything?

Just as mentioned above, PKG_ADD works on octave 3.2.0 that I compiled  
myself, not on the one compile from macports.

> Are the timestamps on your .oct files somehow in the future?

No they have a correct date and time.

> jwe

Thanks for your support,
-c
_______________________________________________
Help-octave mailing list
Help-octave@...
https://www-old.cae.wisc.edu/mailman/listinfo/help-octave