boost::python and threads

[Paul Scruby]

I am having some problems using boost::python with boost::thread.  I'm using
threads because I want to run some tasks in the background when I'm using
the Python's interactive shell.  However, when I use get_override() to call
a Python method from  another boost::thread it crashes internally.  For
example:

    #include <boost/python.hpp>
    #include <boost/thread/thread.hpp>
    #include <boost/thread/xtime.hpp>

    using namespace boost::python;

    class Ticker
        :    public wrapper<Ticker>
    {
    private:
        bool run_;
        volatile bool * running_;
        boost::thread * thread_;
        boost::xtime xt_;
    public:
        Ticker() : running_(&run_) { *running_ = false; }

        void operator()()
        {
            while (*running_)
            {
                boost::xtime_get(&xt_, boost::TIME_UTC);
                ++xt_.sec;
                boost::thread::sleep(xt_);
                onTick();
            }
        }

        void run()
        {
            if (*running_ == false)
            {
                *running_ = true;
                thread_ = new boost::thread(*this);
            }
        }

        void stop()
        {
            if (*running_ == true)
            {
                *running_ = false;
                thread_->join();
                delete thread_;
            }
        }

        virtual void onTick() { get_override("onTick")(); }
        void default_onTick() {}
    };

    BOOST_PYTHON_MODULE(tick)
    {
        class_<Ticker, boost::noncopyable> ("Ticker")
            .def("run", &Ticker::run)
            .def("stop", &Ticker::stop)
            .def("onTick", &Ticker::default_onTick);
    }

Here is a test script that which will crash when you import it into Python's
interactive shell.

    from tick import Ticker

    class MyTicker(Ticker):
        def onTick(self):
        print "Each second"

    myticker = MyTicker()
    myticker.run()

I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler on
Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008 on
Windows XP.

The call-stack in dbx on Solaris:

    >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
PyErr_Restore at 0xfef38fa1
    0xfef38fa1: PyErr_Restore+0x0031:       movl     0x00000028(%edi),%ecx
    Current function is boost::python::override::operator()
       99           detail::method_result x(

    (dbx) where
    current thread: t@2
      [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
0x80652fc), at 0xfef38fa1
      [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
      [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
      [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
      [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
0xfef2bf02
      [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
    =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
"override.hpp"
      [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
      [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
      [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
56 in "thread.hpp"
      [11] thread_proxy(0x810a288), at 0xfea78ce4
      [12] _thr_setup(0xfe670200), at 0xfee159b9
      [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
0x80652fc, 0x80f1220), at 0xfee15ca0

The call-stack in Visual Studio 2008:

    python26.dll!1e013595()
    [Frames below may be incorrect and/or missing, no symbols loaded for
python26.dll]
    python26.dll!1e09ee7d()
  > tick.pyd!boost::python::override::operator()()  Line 103 + 0x16 bytes
C++
    00f3fd64()
    tick.pyd!Ticker::operator()()  Line 27 + 0xe bytes C++
    tick.pyd!boost::detail::thread_data<Ticker>::run()  Line 57 C++
    tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
param=0x00245f30)  Line 168 C++
    msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes C
    msvcr90d.dll!_threadstartex(void * ptd=0x00d46938)  Line 331 C
    kernel32.dll!7c80b729()


Have a missed a trick using the wrapper, or does boost::python not support
threading?

Many thanks,

Paul



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Thomas Berg-3]

Hi,

On Fri, Jul 3, 2009 at 1:15 PM, Paul Scruby<paul@...> wrote:
> I am having some problems using boost::python with boost::thread.  I'm using
> threads because I want to run some tasks in the background when I'm using
> the Python's interactive shell.  However, when I use get_override() to call
> a Python method from  another boost::thread it crashes internally.  For
> example:
>

>
> Have a missed a trick using the wrapper, or does boost::python not support
> threading?
>
> Many thanks,
>
> Paul

Short answer: boost::python does not support threading. There is more
information here, and a few interesting links (with partial
solutions): http://www.boost.org/doc/libs/1_39_0/libs/python/todo.html#full-threading-support

Hope this helps,
Thomas
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[William Ladwig]

It looks to me like you have a garbage collection problem going on.  If you create a wrapped c++ object in python, then python is going to own the object and will destroy it when its reference count goes to 0.  In your python example script at the bottom, you call the Ticker's run() function, which from the python point of view, returns quickly and the script ends.  Python has no idea that you spawned off a new thread from the C++ side, so when the script ends, python destroys the object and now you have a problem.  One way that you can check to see if this is what is going on is to add this to the bottom of the test script and see if the crashes go away:

# Warning, you'll need to kill this script manually
import time
while True:
    time.sleep(1)

Generally when I want to fire off a new C++ thread, I hold any objects that the thread needs in an auto_ptr (see held type for class wrappers), take ownership of them in C++ (see the FAQ in the documentation) and start the thread from the C++ side.  Or, you can also create C++ wrappers which accept shared_ptr arguments, while holding your classes in shared_ptrs, and this should handle the reference counting as well.  Unfortunately, this may require some interface changes to what you have already written (or possibly some clever wrapping).  

Hope this helps,
Bill


________________________________________
From: cplusplus-sig-bounces+wladwig=wdtinc.com@... [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul Scruby [paul@...]
Sent: Friday, July 03, 2009 6:15 AM
To: cplusplus-sig@...
Subject: [C++-sig] boost::python and threads

I am having some problems using boost::python with boost::thread.  I'm using
threads because I want to run some tasks in the background when I'm using
the Python's interactive shell.  However, when I use get_override() to call
a Python method from  another boost::thread it crashes internally.  For
example:

    #include <boost/python.hpp>
    #include <boost/thread/thread.hpp>
    #include <boost/thread/xtime.hpp>

    using namespace boost::python;

    class Ticker
        :    public wrapper<Ticker>
    {
    private:
        bool run_;
        volatile bool * running_;
        boost::thread * thread_;
        boost::xtime xt_;
    public:
        Ticker() : running_(&run_) { *running_ = false; }

        void operator()()
        {
            while (*running_)
            {
                boost::xtime_get(&xt_, boost::TIME_UTC);
                ++xt_.sec;
                boost::thread::sleep(xt_);
                onTick();
            }
        }

        void run()
        {
            if (*running_ == false)
            {
                *running_ = true;
                thread_ = new boost::thread(*this);
            }
        }

        void stop()
        {
            if (*running_ == true)
            {
                *running_ = false;
                thread_->join();
                delete thread_;
            }
        }

        virtual void onTick() { get_override("onTick")(); }
        void default_onTick() {}
    };

    BOOST_PYTHON_MODULE(tick)
    {
        class_<Ticker, boost::noncopyable> ("Ticker")
            .def("run", &Ticker::run)
            .def("stop", &Ticker::stop)
            .def("onTick", &Ticker::default_onTick);
    }

Here is a test script that which will crash when you import it into Python's
interactive shell.

    from tick import Ticker

    class MyTicker(Ticker):
        def onTick(self):
        print "Each second"

    myticker = MyTicker()
    myticker.run()

I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler on
Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008 on
Windows XP.

The call-stack in dbx on Solaris:

    >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
PyErr_Restore at 0xfef38fa1
    0xfef38fa1: PyErr_Restore+0x0031:       movl     0x00000028(%edi),%ecx
    Current function is boost::python::override::operator()
       99           detail::method_result x(

    (dbx) where
    current thread: t@2
      [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
0x80652fc), at 0xfef38fa1
      [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
      [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
      [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
      [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
0xfef2bf02
      [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
    =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
"override.hpp"
      [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
      [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
      [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
56 in "thread.hpp"
      [11] thread_proxy(0x810a288), at 0xfea78ce4
      [12] _thr_setup(0xfe670200), at 0xfee159b9
      [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
0x80652fc, 0x80f1220), at 0xfee15ca0

The call-stack in Visual Studio 2008:

    python26.dll!1e013595()
    [Frames below may be incorrect and/or missing, no symbols loaded for
python26.dll]
    python26.dll!1e09ee7d()
  > tick.pyd!boost::python::override::operator()()  Line 103 + 0x16 bytes
C++
    00f3fd64()
    tick.pyd!Ticker::operator()()  Line 27 + 0xe bytes C++
    tick.pyd!boost::detail::thread_data<Ticker>::run()  Line 57 C++
    tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
param=0x00245f30)  Line 168 C++
    msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes C
    msvcr90d.dll!_threadstartex(void * ptd=0x00d46938)  Line 331 C
    kernel32.dll!7c80b729()


Have a missed a trick using the wrapper, or does boost::python not support
threading?

Many thanks,

Paul



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[William Ladwig]

Whoops, I think this problem is a little uglier than I thought, since you overrode the onTick() function in python with a call to print, which needs access to the interpreter in your new thread.  See the link Thomas posted for dealing with the GIL (along with worrying about any possible garbage collection issues).  Now I remember why I kept my C++ threads isolated from Python stuff....sorry, it's been a while....

Bill
________________________________________
From: cplusplus-sig-bounces+wladwig=wdtinc.com@... [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of William Ladwig [wladwig@...]
Sent: Saturday, July 04, 2009 1:34 PM
To: Development of Python/C++ integration
Subject: Re: [C++-sig] boost::python and threads

It looks to me like you have a garbage collection problem going on.  If you create a wrapped c++ object in python, then python is going to own the object and will destroy it when its reference count goes to 0.  In your python example script at the bottom, you call the Ticker's run() function, which from the python point of view, returns quickly and the script ends.  Python has no idea that you spawned off a new thread from the C++ side, so when the script ends, python destroys the object and now you have a problem.  One way that you can check to see if this is what is going on is to add this to the bottom of the test script and see if the crashes go away:

# Warning, you'll need to kill this script manually
import time
while True:
    time.sleep(1)

Generally when I want to fire off a new C++ thread, I hold any objects that the thread needs in an auto_ptr (see held type for class wrappers), take ownership of them in C++ (see the FAQ in the documentation) and start the thread from the C++ side.  Or, you can also create C++ wrappers which accept shared_ptr arguments, while holding your classes in shared_ptrs, and this should handle the reference counting as well.  Unfortunately, this may require some interface changes to what you have already written (or possibly some clever wrapping).

Hope this helps,
Bill


________________________________________
From: cplusplus-sig-bounces+wladwig=wdtinc.com@... [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul Scruby [paul@...]
Sent: Friday, July 03, 2009 6:15 AM
To: cplusplus-sig@...
Subject: [C++-sig] boost::python and threads

I am having some problems using boost::python with boost::thread.  I'm using
threads because I want to run some tasks in the background when I'm using
the Python's interactive shell.  However, when I use get_override() to call
a Python method from  another boost::thread it crashes internally.  For
example:

    #include <boost/python.hpp>
    #include <boost/thread/thread.hpp>
    #include <boost/thread/xtime.hpp>

    using namespace boost::python;

    class Ticker
        :    public wrapper<Ticker>
    {
    private:
        bool run_;
        volatile bool * running_;
        boost::thread * thread_;
        boost::xtime xt_;
    public:
        Ticker() : running_(&run_) { *running_ = false; }

        void operator()()
        {
            while (*running_)
            {
                boost::xtime_get(&xt_, boost::TIME_UTC);
                ++xt_.sec;
                boost::thread::sleep(xt_);
                onTick();
            }
        }

        void run()
        {
            if (*running_ == false)
            {
                *running_ = true;
                thread_ = new boost::thread(*this);
            }
        }

        void stop()
        {
            if (*running_ == true)
            {
                *running_ = false;
                thread_->join();
                delete thread_;
            }
        }

        virtual void onTick() { get_override("onTick")(); }
        void default_onTick() {}
    };

    BOOST_PYTHON_MODULE(tick)
    {
        class_<Ticker, boost::noncopyable> ("Ticker")
            .def("run", &Ticker::run)
            .def("stop", &Ticker::stop)
            .def("onTick", &Ticker::default_onTick);
    }

Here is a test script that which will crash when you import it into Python's
interactive shell.

    from tick import Ticker

    class MyTicker(Ticker):
        def onTick(self):
        print "Each second"

    myticker = MyTicker()
    myticker.run()

I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler on
Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008 on
Windows XP.

The call-stack in dbx on Solaris:

    >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
PyErr_Restore at 0xfef38fa1
    0xfef38fa1: PyErr_Restore+0x0031:       movl     0x00000028(%edi),%ecx
    Current function is boost::python::override::operator()
       99           detail::method_result x(

    (dbx) where
    current thread: t@2
      [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
0x80652fc), at 0xfef38fa1
      [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
      [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
      [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
      [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
0xfef2bf02
      [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
    =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
"override.hpp"
      [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
      [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
      [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
56 in "thread.hpp"
      [11] thread_proxy(0x810a288), at 0xfea78ce4
      [12] _thr_setup(0xfe670200), at 0xfee159b9
      [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
0x80652fc, 0x80f1220), at 0xfee15ca0

The call-stack in Visual Studio 2008:

    python26.dll!1e013595()
    [Frames below may be incorrect and/or missing, no symbols loaded for
python26.dll]
    python26.dll!1e09ee7d()
  > tick.pyd!boost::python::override::operator()()  Line 103 + 0x16 bytes
C++
    00f3fd64()
    tick.pyd!Ticker::operator()()  Line 27 + 0xe bytes C++
    tick.pyd!boost::detail::thread_data<Ticker>::run()  Line 57 C++
    tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
param=0x00245f30)  Line 168 C++
    msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes C
    msvcr90d.dll!_threadstartex(void * ptd=0x00d46938)  Line 331 C
    kernel32.dll!7c80b729()


Have a missed a trick using the wrapper, or does boost::python not support
threading?

Many thanks,

Paul



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Renato Araujo]

Hi Paul

In my bindings I had a problem like this, to solve I created a simple
class like that:

class thread_locker
{
thread_locker()
{
    if (thread_support::enabled())
        m_gstate = PyGILState_Ensure();
}

~thread_locker()
{
    if (thread_support::enabled())
        PyGILState_Release(m_gstate);
}
};

then in my wrapper virtual implementation I did this:

...
void wrapper::virtual_func(..)
{
thread_locker lock;
.. my code ..
}
....

this solve my problems with call of virtual functions in thread enviroment.

BR




On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:

> Whoops, I think this problem is a little uglier than I thought, since you overrode the onTick() function in python with a call to print, which needs access to the interpreter in your new thread.  See the link Thomas posted for dealing with the GIL (along with worrying about any possible garbage collection issues).  Now I remember why I kept my C++ threads isolated from Python stuff....sorry, it's been a while....
>
> Bill
> ________________________________________
> From: cplusplus-sig-bounces+wladwig=wdtinc.com@... [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of William Ladwig [wladwig@...]
> Sent: Saturday, July 04, 2009 1:34 PM
> To: Development of Python/C++ integration
> Subject: Re: [C++-sig] boost::python and threads
>
> It looks to me like you have a garbage collection problem going on.  If you create a wrapped c++ object in python, then python is going to own the object and will destroy it when its reference count goes to 0.  In your python example script at the bottom, you call the Ticker's run() function, which from the python point of view, returns quickly and the script ends.  Python has no idea that you spawned off a new thread from the C++ side, so when the script ends, python destroys the object and now you have a problem.  One way that you can check to see if this is what is going on is to add this to the bottom of the test script and see if the crashes go away:
>
> # Warning, you'll need to kill this script manually
> import time
> while True:
>    time.sleep(1)
>
> Generally when I want to fire off a new C++ thread, I hold any objects that the thread needs in an auto_ptr (see held type for class wrappers), take ownership of them in C++ (see the FAQ in the documentation) and start the thread from the C++ side.  Or, you can also create C++ wrappers which accept shared_ptr arguments, while holding your classes in shared_ptrs, and this should handle the reference counting as well.  Unfortunately, this may require some interface changes to what you have already written (or possibly some clever wrapping).
>
> Hope this helps,
> Bill
>
>
> ________________________________________
> From: cplusplus-sig-bounces+wladwig=wdtinc.com@... [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul Scruby [paul@...]
> Sent: Friday, July 03, 2009 6:15 AM
> To: cplusplus-sig@...
> Subject: [C++-sig] boost::python and threads
>
> I am having some problems using boost::python with boost::thread.  I'm using
> threads because I want to run some tasks in the background when I'm using
> the Python's interactive shell.  However, when I use get_override() to call
> a Python method from  another boost::thread it crashes internally.  For
> example:
>
>    #include <boost/python.hpp>
>    #include <boost/thread/thread.hpp>
>    #include <boost/thread/xtime.hpp>
>
>    using namespace boost::python;
>
>    class Ticker
>        :    public wrapper<Ticker>
>    {
>    private:
>        bool run_;
>        volatile bool * running_;
>        boost::thread * thread_;
>        boost::xtime xt_;
>    public:
>        Ticker() : running_(&run_) { *running_ = false; }
>
>        void operator()()
>        {
>            while (*running_)
>            {
>                boost::xtime_get(&xt_, boost::TIME_UTC);
>                ++xt_.sec;
>                boost::thread::sleep(xt_);
>                onTick();
>            }
>        }
>
>        void run()
>        {
>            if (*running_ == false)
>            {
>                *running_ = true;
>                thread_ = new boost::thread(*this);
>            }
>        }
>
>        void stop()
>        {
>            if (*running_ == true)
>            {
>                *running_ = false;
>                thread_->join();
>                delete thread_;
>            }
>        }
>
>        virtual void onTick() { get_override("onTick")(); }
>        void default_onTick() {}
>    };
>
>    BOOST_PYTHON_MODULE(tick)
>    {
>        class_<Ticker, boost::noncopyable> ("Ticker")
>            .def("run", &Ticker::run)
>            .def("stop", &Ticker::stop)
>            .def("onTick", &Ticker::default_onTick);
>    }
>
> Here is a test script that which will crash when you import it into Python's
> interactive shell.
>
>    from tick import Ticker
>
>    class MyTicker(Ticker):
>        def onTick(self):
>        print "Each second"
>
>    myticker = MyTicker()
>    myticker.run()
>
> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler on
> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008 on
> Windows XP.
>
> The call-stack in dbx on Solaris:
>
>    >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
> PyErr_Restore at 0xfef38fa1
>    0xfef38fa1: PyErr_Restore+0x0031:       movl     0x00000028(%edi),%ecx
>    Current function is boost::python::override::operator()
>       99           detail::method_result x(
>
>    (dbx) where
>    current thread: t@2
>      [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
> 0x80652fc), at 0xfef38fa1
>      [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>      [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>      [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>      [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
> 0xfef2bf02
>      [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>    =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
> "override.hpp"
>      [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>      [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>      [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
> 56 in "thread.hpp"
>      [11] thread_proxy(0x810a288), at 0xfea78ce4
>      [12] _thr_setup(0xfe670200), at 0xfee159b9
>      [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
> 0x80652fc, 0x80f1220), at 0xfee15ca0
>
> The call-stack in Visual Studio 2008:
>
>    python26.dll!1e013595()
>    [Frames below may be incorrect and/or missing, no symbols loaded for
> python26.dll]
>    python26.dll!1e09ee7d()
>  > tick.pyd!boost::python::override::operator()()  Line 103 + 0x16 bytes
> C++
>    00f3fd64()
>    tick.pyd!Ticker::operator()()  Line 27 + 0xe bytes C++
>    tick.pyd!boost::detail::thread_data<Ticker>::run()  Line 57 C++
>    tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
> param=0x00245f30)  Line 168 C++
>    msvcr90d.dll!_callthreadstartex()  Line 348 + 0xf bytes C
>    msvcr90d.dll!_threadstartex(void * ptd=0x00d46938)  Line 331 C
>    kernel32.dll!7c80b729()
>
>
> Have a missed a trick using the wrapper, or does boost::python not support
> threading?
>
> Many thanks,
>
> Paul
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Paul Scruby]

Hiya,

That's fantastic, all I needed to do was to put PyGILState_Ensure(); before
my virtual function calls into python from another thread and my program no
longer crashes.  Problem solved, isn't boost::python great!

Many thanks,

Paul


"Renato Araujo" <renatox@...> wrote in message
news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
Hi Paul

In my bindings I had a problem like this, to solve I created a simple
class like that:

class thread_locker
{
thread_locker()
{
    if (thread_support::enabled())
        m_gstate = PyGILState_Ensure();
}

~thread_locker()
{
    if (thread_support::enabled())
        PyGILState_Release(m_gstate);
}
};

then in my wrapper virtual implementation I did this:

...
void wrapper::virtual_func(..)
{
thread_locker lock;
.. my code ..
}
....

this solve my problems with call of virtual functions in thread enviroment.

BR




On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:

> Whoops, I think this problem is a little uglier than I thought, since you
> overrode the onTick() function in python with a call to print, which needs
> access to the interpreter in your new thread. See the link Thomas posted
> for dealing with the GIL (along with worrying about any possible garbage
> collection issues). Now I remember why I kept my C++ threads isolated from
> Python stuff....sorry, it's been a while....
>
> Bill
> ________________________________________
> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of William
> Ladwig [wladwig@...]
> Sent: Saturday, July 04, 2009 1:34 PM
> To: Development of Python/C++ integration
> Subject: Re: [C++-sig] boost::python and threads
>
> It looks to me like you have a garbage collection problem going on. If you
> create a wrapped c++ object in python, then python is going to own the
> object and will destroy it when its reference count goes to 0. In your
> python example script at the bottom, you call the Ticker's run() function,
> which from the python point of view, returns quickly and the script ends.
> Python has no idea that you spawned off a new thread from the C++ side, so
> when the script ends, python destroys the object and now you have a
> problem. One way that you can check to see if this is what is going on is
> to add this to the bottom of the test script and see if the crashes go
> away:
>
> # Warning, you'll need to kill this script manually
> import time
> while True:
> time.sleep(1)
>
> Generally when I want to fire off a new C++ thread, I hold any objects
> that the thread needs in an auto_ptr (see held type for class wrappers),
> take ownership of them in C++ (see the FAQ in the documentation) and start
> the thread from the C++ side. Or, you can also create C++ wrappers which
> accept shared_ptr arguments, while holding your classes in shared_ptrs,
> and this should handle the reference counting as well. Unfortunately, this
> may require some interface changes to what you have already written (or
> possibly some clever wrapping).
>
> Hope this helps,
> Bill
>
>
> ________________________________________
> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
> Scruby [paul@...]
> Sent: Friday, July 03, 2009 6:15 AM
> To: cplusplus-sig@...
> Subject: [C++-sig] boost::python and threads
>
> I am having some problems using boost::python with boost::thread. I'm
> using
> threads because I want to run some tasks in the background when I'm using
> the Python's interactive shell. However, when I use get_override() to call
> a Python method from another boost::thread it crashes internally. For
> example:
>
> #include <boost/python.hpp>
> #include <boost/thread/thread.hpp>
> #include <boost/thread/xtime.hpp>
>
> using namespace boost::python;
>
> class Ticker
> : public wrapper<Ticker>
> {
> private:
> bool run_;
> volatile bool * running_;
> boost::thread * thread_;
> boost::xtime xt_;
> public:
> Ticker() : running_(&run_) { *running_ = false; }
>
> void operator()()
> {
> while (*running_)
> {
> boost::xtime_get(&xt_, boost::TIME_UTC);
> ++xt_.sec;
> boost::thread::sleep(xt_);
> onTick();
> }
> }
>
> void run()
> {
> if (*running_ == false)
> {
> *running_ = true;
> thread_ = new boost::thread(*this);
> }
> }
>
> void stop()
> {
> if (*running_ == true)
> {
> *running_ = false;
> thread_->join();
> delete thread_;
> }
> }
>
> virtual void onTick() { get_override("onTick")(); }
> void default_onTick() {}
> };
>
> BOOST_PYTHON_MODULE(tick)
> {
> class_<Ticker, boost::noncopyable> ("Ticker")
> .def("run", &Ticker::run)
> .def("stop", &Ticker::stop)
> .def("onTick", &Ticker::default_onTick);
> }
>
> Here is a test script that which will crash when you import it into
> Python's
> interactive shell.
>
> from tick import Ticker
>
> class MyTicker(Ticker):
> def onTick(self):
> print "Each second"
>
> myticker = MyTicker()
> myticker.run()
>
> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler on
> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008 on
> Windows XP.
>
> The call-stack in dbx on Solaris:
>
> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
> PyErr_Restore at 0xfef38fa1
> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
> Current function is boost::python::override::operator()
> 99 detail::method_result x(
>
> (dbx) where
> current thread: t@2
> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
> 0x80652fc), at 0xfef38fa1
> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
> 0xfef2bf02
> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
> "override.hpp"
> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
> 56 in "thread.hpp"
> [11] thread_proxy(0x810a288), at 0xfea78ce4
> [12] _thr_setup(0xfe670200), at 0xfee159b9
> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
> 0x80652fc, 0x80f1220), at 0xfee15ca0
>
> The call-stack in Visual Studio 2008:
>
> python26.dll!1e013595()
> [Frames below may be incorrect and/or missing, no symbols loaded for
> python26.dll]
> python26.dll!1e09ee7d()
> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
> C++
> 00f3fd64()
> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
> param=0x00245f30) Line 168 C++
> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
> kernel32.dll!7c80b729()
>
>
> Have a missed a trick using the wrapper, or does boost::python not support
> threading?
>
> Many thanks,
>
> Paul
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Paul Scruby]

Hello again,

Sorry, I spoke too soon.  The good news is that wrapping my virtual method
calls into Python between PyGILState_Ensure() and PyGILState_Release() fixed
the crash under Python2.6.2 on Windows, but not when I tested it again under
Python 2.4.4 on Solaris.  Under Python 2.4.4 on SolarisSolaris it now calls
and exits the virtual method in Python sucessfully, but then crashes when
C++ trys to release the global interpretor lock.

The call-stack with the Sun C++ 5.9 compiler under Solaris is
    t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid
at 0xfec453ed
    0xfec453ed: sem_invalid+0x0013: movzwl   0x00000006(%eax),%eax
    Current function is Ticker::operator()
    28                           PyGILState_Release(state_);
    (dbx) where
    current thread: t@2
    [1] sem_invalid(0x0), at 0xfec453ed
    [2] _sem_post(0x0), at 0xfec454c4
    [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
0xfe77ef38, 0xfef441b5), at 0xfef492dc
    [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
    [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
0x0, 0x80a3140), at 0xfef43eba
    [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
  =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
    [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
in "thread.hpp"
    [9] thread_proxy(0x810a288), at 0xfea78ce4
    [10] _thr_setup(0xfe670200), at 0xfee159b9
    [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
0xfe77ef14), at 0xfee15ca0

Do you think it's worth repeating this test using gcc/linux, or do you think
that this is just a limitation of using Python with threads?

Thanks again,

Paul


t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid at
0xfec453ed
0xfec453ed: sem_invalid+0x0013: movzwl   0x00000006(%eax),%eax
Current function is Ticker::operator()
   28                           PyGILState_Release(state_);
(dbx) where
current thread: t@2
  [1] sem_invalid(0x0), at 0xfec453ed
  [2] _sem_post(0x0), at 0xfec454c4
  [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
0xfe77ef38, 0xfef441b5), at 0xfef492dc
  [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
  [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
0x0, 0x80a3140), at 0xfef43eba
  [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
=>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
  [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
"thread.hpp"
  [9] thread_proxy(0x810a288), at 0xfea78ce4
  [10] _thr_setup(0xfe670200), at 0xfee159b9
  [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
0xfe77ef14), at 0xfee15ca0



"Paul Scruby" <paul@...> wrote in message
news:h2sgic$ad4$1@......

> Hiya,
>
> That's fantastic, all I needed to do was to put PyGILState_Ensure();
> before my virtual function calls into python from another thread and my
> program no longer crashes.  Problem solved, isn't boost::python great!
>
> Many thanks,
>
> Paul
>
>
> "Renato Araujo" <renatox@...> wrote in message
> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
> Hi Paul
>
> In my bindings I had a problem like this, to solve I created a simple
> class like that:
>
> class thread_locker
> {
> thread_locker()
> {
>    if (thread_support::enabled())
>        m_gstate = PyGILState_Ensure();
> }
>
> ~thread_locker()
> {
>    if (thread_support::enabled())
>        PyGILState_Release(m_gstate);
> }
> };
>
> then in my wrapper virtual implementation I did this:
>
> ...
> void wrapper::virtual_func(..)
> {
> thread_locker lock;
> .. my code ..
> }
> ....
>
> this solve my problems with call of virtual functions in thread
> enviroment.
>
> BR
>
>
>
>
> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:
>> Whoops, I think this problem is a little uglier than I thought, since you
>> overrode the onTick() function in python with a call to print, which
>> needs access to the interpreter in your new thread. See the link Thomas
>> posted for dealing with the GIL (along with worrying about any possible
>> garbage collection issues). Now I remember why I kept my C++ threads
>> isolated from Python stuff....sorry, it's been a while....
>>
>> Bill
>> ________________________________________
>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of
>> William Ladwig [wladwig@...]
>> Sent: Saturday, July 04, 2009 1:34 PM
>> To: Development of Python/C++ integration
>> Subject: Re: [C++-sig] boost::python and threads
>>
>> It looks to me like you have a garbage collection problem going on. If
>> you create a wrapped c++ object in python, then python is going to own
>> the object and will destroy it when its reference count goes to 0. In
>> your python example script at the bottom, you call the Ticker's run()
>> function, which from the python point of view, returns quickly and the
>> script ends. Python has no idea that you spawned off a new thread from
>> the C++ side, so when the script ends, python destroys the object and now
>> you have a problem. One way that you can check to see if this is what is
>> going on is to add this to the bottom of the test script and see if the
>> crashes go away:
>>
>> # Warning, you'll need to kill this script manually
>> import time
>> while True:
>> time.sleep(1)
>>
>> Generally when I want to fire off a new C++ thread, I hold any objects
>> that the thread needs in an auto_ptr (see held type for class wrappers),
>> take ownership of them in C++ (see the FAQ in the documentation) and
>> start the thread from the C++ side. Or, you can also create C++ wrappers
>> which accept shared_ptr arguments, while holding your classes in
>> shared_ptrs, and this should handle the reference counting as well.
>> Unfortunately, this may require some interface changes to what you have
>> already written (or possibly some clever wrapping).
>>
>> Hope this helps,
>> Bill
>>
>>
>> ________________________________________
>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
>> Scruby [paul@...]
>> Sent: Friday, July 03, 2009 6:15 AM
>> To: cplusplus-sig@...
>> Subject: [C++-sig] boost::python and threads
>>
>> I am having some problems using boost::python with boost::thread. I'm
>> using
>> threads because I want to run some tasks in the background when I'm using
>> the Python's interactive shell. However, when I use get_override() to
>> call
>> a Python method from another boost::thread it crashes internally. For
>> example:
>>
>> #include <boost/python.hpp>
>> #include <boost/thread/thread.hpp>
>> #include <boost/thread/xtime.hpp>
>>
>> using namespace boost::python;
>>
>> class Ticker
>> : public wrapper<Ticker>
>> {
>> private:
>> bool run_;
>> volatile bool * running_;
>> boost::thread * thread_;
>> boost::xtime xt_;
>> public:
>> Ticker() : running_(&run_) { *running_ = false; }
>>
>> void operator()()
>> {
>> while (*running_)
>> {
>> boost::xtime_get(&xt_, boost::TIME_UTC);
>> ++xt_.sec;
>> boost::thread::sleep(xt_);
>> onTick();
>> }
>> }
>>
>> void run()
>> {
>> if (*running_ == false)
>> {
>> *running_ = true;
>> thread_ = new boost::thread(*this);
>> }
>> }
>>
>> void stop()
>> {
>> if (*running_ == true)
>> {
>> *running_ = false;
>> thread_->join();
>> delete thread_;
>> }
>> }
>>
>> virtual void onTick() { get_override("onTick")(); }
>> void default_onTick() {}
>> };
>>
>> BOOST_PYTHON_MODULE(tick)
>> {
>> class_<Ticker, boost::noncopyable> ("Ticker")
>> .def("run", &Ticker::run)
>> .def("stop", &Ticker::stop)
>> .def("onTick", &Ticker::default_onTick);
>> }
>>
>> Here is a test script that which will crash when you import it into
>> Python's
>> interactive shell.
>>
>> from tick import Ticker
>>
>> class MyTicker(Ticker):
>> def onTick(self):
>> print "Each second"
>>
>> myticker = MyTicker()
>> myticker.run()
>>
>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>> on
>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>> on
>> Windows XP.
>>
>> The call-stack in dbx on Solaris:
>>
>> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
>> PyErr_Restore at 0xfef38fa1
>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>> Current function is boost::python::override::operator()
>> 99 detail::method_result x(
>>
>> (dbx) where
>> current thread: t@2
>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>> 0x80652fc), at 0xfef38fa1
>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>> 0xfef2bf02
>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>> "override.hpp"
>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>> 56 in "thread.hpp"
>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>
>> The call-stack in Visual Studio 2008:
>>
>> python26.dll!1e013595()
>> [Frames below may be incorrect and/or missing, no symbols loaded for
>> python26.dll]
>> python26.dll!1e09ee7d()
>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>> C++
>> 00f3fd64()
>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>> param=0x00245f30) Line 168 C++
>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>> kernel32.dll!7c80b729()
>>
>>
>> Have a missed a trick using the wrapper, or does boost::python not
>> support
>> threading?
>>
>> Many thanks,
>>
>> Paul
>>
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig@...
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig@...
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig@...
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Renato Araujo Oliveira Filho

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Renato Araujo]

I'm using gcc/linux and python >= 2.4 and works fine for me.



On Mon, Jul 6, 2009 at 7:39 AM, Paul Scruby<paul@...> wrote:

> Hello again,
>
> Sorry, I spoke too soon.  The good news is that wrapping my virtual method
> calls into Python between PyGILState_Ensure() and PyGILState_Release() fixed
> the crash under Python2.6.2 on Windows, but not when I tested it again under
> Python 2.4.4 on Solaris.  Under Python 2.4.4 on SolarisSolaris it now calls
> and exits the virtual method in Python sucessfully, but then crashes when
> C++ trys to release the global interpretor lock.
>
> The call-stack with the Sun C++ 5.9 compiler under Solaris is
>    t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid
> at 0xfec453ed
>    0xfec453ed: sem_invalid+0x0013: movzwl   0x00000006(%eax),%eax
>    Current function is Ticker::operator()
>    28                           PyGILState_Release(state_);
>    (dbx) where
>    current thread: t@2
>    [1] sem_invalid(0x0), at 0xfec453ed
>    [2] _sem_post(0x0), at 0xfec454c4
>    [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>    [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>    [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
> 0x0, 0x80a3140), at 0xfef43eba
>    [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>  =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>    [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
> in "thread.hpp"
>    [9] thread_proxy(0x810a288), at 0xfea78ce4
>    [10] _thr_setup(0xfe670200), at 0xfee159b9
>    [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
> 0xfe77ef14), at 0xfee15ca0
>
> Do you think it's worth repeating this test using gcc/linux, or do you think
> that this is just a limitation of using Python with threads?
>
> Thanks again,
>
> Paul
>
>
> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid at
> 0xfec453ed
> 0xfec453ed: sem_invalid+0x0013: movzwl   0x00000006(%eax),%eax
> Current function is Ticker::operator()
>   28                           PyGILState_Release(state_);
> (dbx) where
> current thread: t@2
>  [1] sem_invalid(0x0), at 0xfec453ed
>  [2] _sem_post(0x0), at 0xfec454c4
>  [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>  [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>  [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
> 0x0, 0x80a3140), at 0xfef43eba
>  [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>  [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
> "thread.hpp"
>  [9] thread_proxy(0x810a288), at 0xfea78ce4
>  [10] _thr_setup(0xfe670200), at 0xfee159b9
>  [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
> 0xfe77ef14), at 0xfee15ca0
>
>
>
> "Paul Scruby" <paul@...> wrote in message
> news:h2sgic$ad4$1@......
>> Hiya,
>>
>> That's fantastic, all I needed to do was to put PyGILState_Ensure();
>> before my virtual function calls into python from another thread and my
>> program no longer crashes.  Problem solved, isn't boost::python great!
>>
>> Many thanks,
>>
>> Paul
>>
>>
>> "Renato Araujo" <renatox@...> wrote in message
>> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
>> Hi Paul
>>
>> In my bindings I had a problem like this, to solve I created a simple
>> class like that:
>>
>> class thread_locker
>> {
>> thread_locker()
>> {
>>    if (thread_support::enabled())
>>        m_gstate = PyGILState_Ensure();
>> }
>>
>> ~thread_locker()
>> {
>>    if (thread_support::enabled())
>>        PyGILState_Release(m_gstate);
>> }
>> };
>>
>> then in my wrapper virtual implementation I did this:
>>
>> ...
>> void wrapper::virtual_func(..)
>> {
>> thread_locker lock;
>> .. my code ..
>> }
>> ....
>>
>> this solve my problems with call of virtual functions in thread
>> enviroment.
>>
>> BR
>>
>>
>>
>>
>> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:
>>> Whoops, I think this problem is a little uglier than I thought, since you
>>> overrode the onTick() function in python with a call to print, which
>>> needs access to the interpreter in your new thread. See the link Thomas
>>> posted for dealing with the GIL (along with worrying about any possible
>>> garbage collection issues). Now I remember why I kept my C++ threads
>>> isolated from Python stuff....sorry, it's been a while....
>>>
>>> Bill
>>> ________________________________________
>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of
>>> William Ladwig [wladwig@...]
>>> Sent: Saturday, July 04, 2009 1:34 PM
>>> To: Development of Python/C++ integration
>>> Subject: Re: [C++-sig] boost::python and threads
>>>
>>> It looks to me like you have a garbage collection problem going on. If
>>> you create a wrapped c++ object in python, then python is going to own
>>> the object and will destroy it when its reference count goes to 0. In
>>> your python example script at the bottom, you call the Ticker's run()
>>> function, which from the python point of view, returns quickly and the
>>> script ends. Python has no idea that you spawned off a new thread from
>>> the C++ side, so when the script ends, python destroys the object and now
>>> you have a problem. One way that you can check to see if this is what is
>>> going on is to add this to the bottom of the test script and see if the
>>> crashes go away:
>>>
>>> # Warning, you'll need to kill this script manually
>>> import time
>>> while True:
>>> time.sleep(1)
>>>
>>> Generally when I want to fire off a new C++ thread, I hold any objects
>>> that the thread needs in an auto_ptr (see held type for class wrappers),
>>> take ownership of them in C++ (see the FAQ in the documentation) and
>>> start the thread from the C++ side. Or, you can also create C++ wrappers
>>> which accept shared_ptr arguments, while holding your classes in
>>> shared_ptrs, and this should handle the reference counting as well.
>>> Unfortunately, this may require some interface changes to what you have
>>> already written (or possibly some clever wrapping).
>>>
>>> Hope this helps,
>>> Bill
>>>
>>>
>>> ________________________________________
>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
>>> Scruby [paul@...]
>>> Sent: Friday, July 03, 2009 6:15 AM
>>> To: cplusplus-sig@...
>>> Subject: [C++-sig] boost::python and threads
>>>
>>> I am having some problems using boost::python with boost::thread. I'm
>>> using
>>> threads because I want to run some tasks in the background when I'm using
>>> the Python's interactive shell. However, when I use get_override() to
>>> call
>>> a Python method from another boost::thread it crashes internally. For
>>> example:
>>>
>>> #include <boost/python.hpp>
>>> #include <boost/thread/thread.hpp>
>>> #include <boost/thread/xtime.hpp>
>>>
>>> using namespace boost::python;
>>>
>>> class Ticker
>>> : public wrapper<Ticker>
>>> {
>>> private:
>>> bool run_;
>>> volatile bool * running_;
>>> boost::thread * thread_;
>>> boost::xtime xt_;
>>> public:
>>> Ticker() : running_(&run_) { *running_ = false; }
>>>
>>> void operator()()
>>> {
>>> while (*running_)
>>> {
>>> boost::xtime_get(&xt_, boost::TIME_UTC);
>>> ++xt_.sec;
>>> boost::thread::sleep(xt_);
>>> onTick();
>>> }
>>> }
>>>
>>> void run()
>>> {
>>> if (*running_ == false)
>>> {
>>> *running_ = true;
>>> thread_ = new boost::thread(*this);
>>> }
>>> }
>>>
>>> void stop()
>>> {
>>> if (*running_ == true)
>>> {
>>> *running_ = false;
>>> thread_->join();
>>> delete thread_;
>>> }
>>> }
>>>
>>> virtual void onTick() { get_override("onTick")(); }
>>> void default_onTick() {}
>>> };
>>>
>>> BOOST_PYTHON_MODULE(tick)
>>> {
>>> class_<Ticker, boost::noncopyable> ("Ticker")
>>> .def("run", &Ticker::run)
>>> .def("stop", &Ticker::stop)
>>> .def("onTick", &Ticker::default_onTick);
>>> }
>>>
>>> Here is a test script that which will crash when you import it into
>>> Python's
>>> interactive shell.
>>>
>>> from tick import Ticker
>>>
>>> class MyTicker(Ticker):
>>> def onTick(self):
>>> print "Each second"
>>>
>>> myticker = MyTicker()
>>> myticker.run()
>>>
>>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>>> on
>>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>>> on
>>> Windows XP.
>>>
>>> The call-stack in dbx on Solaris:
>>>
>>> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
>>> PyErr_Restore at 0xfef38fa1
>>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>>> Current function is boost::python::override::operator()
>>> 99 detail::method_result x(
>>>
>>> (dbx) where
>>> current thread: t@2
>>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>>> 0x80652fc), at 0xfef38fa1
>>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>>> 0xfef2bf02
>>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>>> "override.hpp"
>>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>>> 56 in "thread.hpp"
>>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>>
>>> The call-stack in Visual Studio 2008:
>>>
>>> python26.dll!1e013595()
>>> [Frames below may be incorrect and/or missing, no symbols loaded for
>>> python26.dll]
>>> python26.dll!1e09ee7d()
>>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>>> C++
>>> 00f3fd64()
>>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>>> param=0x00245f30) Line 168 C++
>>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>>> kernel32.dll!7c80b729()
>>>
>>>
>>> Have a missed a trick using the wrapper, or does boost::python not
>>> support
>>> threading?
>>>
>>> Many thanks,
>>>
>>> Paul
>>>
>>>
>>>
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>
>>
>>
>>
>> --
>> Renato Araujo Oliveira Filho
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Paul Scruby]

Hi Renato,

Okay, I have installed Python 2.4.2 with gcc 4.1.2 on Linux and it's
crashing in the same place as on Solaris.

  Program received signal SIGSEGV, Segmentation fault.
  [Switching to Thread 0x41074940 (LWP 12004)]
  0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
  (gdb) where
  #0  0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
  #1  0x000000308b2b78e9 in PyThread_release_lock ()
     from /usr/lib64/libpython2.4.so.1.0
  #2  0x00002b46531d9f82 in Ticker::operator() (this=0xbd0f478) at
ticker.cc:28
  #3  0x00002b46531d9fb0 in boost::detail::thread_data<Ticker>::run (
      this=0xbd0f370)
      at /opt/atm/include/boost-1_39/boost/thread/detail/thread.hpp:56
  #4  0x00002b46533ee14b in thread_proxy ()
     from /opt/atm/lib64/libboost_thread-gcc41-mt-1_39.so.1.39.0
  #5  0x0000003077406367 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000030768d2f7d in clone () from /lib64/libc.so.6

With the global interpret lock added my code now looks like this...

  #include <boost/python.hpp>
  #include <boost/thread/thread.hpp>
  #include <boost/thread/xtime.hpp>
  using namespace boost::python;

  class Ticker
      :    public wrapper<Ticker>
  {
  private:
      bool run_;
      volatile bool * running_;
      boost::thread * thread_;
      boost::xtime xt_;
      PyGILState_STATE state_;
  public:
      Ticker() :running_(&run_) { *running_ = false; }

      void operator()()
      {
          while (*running_)
          {
              boost::xtime_get(&xt_, boost::TIME_UTC);
              ++xt_.sec;
              boost::thread::sleep(xt_);
              state_ = PyGILState_Ensure();
              onTick();
              PyGILState_Release(state_);
          }
      }

      void run()
      {
          if (*running_ == false)
          {
              *running_ = true;
              thread_ = new boost::thread(*this);
          }
      }

      void stop()
      {
          if (*running_ == true)
          {
              *running_ = false;
              thread_->join();
              delete thread_;
          }
      }

      virtual void onTick() { get_override("onTick")(); }
      void default_onTick() {}
  };

  BOOST_PYTHON_MODULE(tick)
  {
      class_<Ticker, boost::noncopyable> ("Ticker")
          .def("run", &Ticker::run)
          .def("stop", &Ticker::stop)
          .def("onTick", &Ticker::default_onTick);
  }


Thanks again,

Paul


"Renato Araujo" <renatox@...> wrote in message
news:95291a80907060619u5bff0dcey56947d1ac848cc8e@......
I'm using gcc/linux and python >= 2.4 and works fine for me.



On Mon, Jul 6, 2009 at 7:39 AM, Paul Scruby<paul@...> wrote:

> Hello again,
>
> Sorry, I spoke too soon. The good news is that wrapping my virtual method
> calls into Python between PyGILState_Ensure() and PyGILState_Release()
> fixed
> the crash under Python2.6.2 on Windows, but not when I tested it again
> under
> Python 2.4.4 on Solaris. Under Python 2.4.4 on SolarisSolaris it now calls
> and exits the virtual method in Python sucessfully, but then crashes when
> C++ trys to release the global interpretor lock.
>
> The call-stack with the Sun C++ 5.9 compiler under Solaris is
> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid
> at 0xfec453ed
> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
> Current function is Ticker::operator()
> 28 PyGILState_Release(state_);
> (dbx) where
> current thread: t@2
> [1] sem_invalid(0x0), at 0xfec453ed
> [2] _sem_post(0x0), at 0xfec454c4
> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
> 0x0, 0x80a3140), at 0xfef43eba
> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
> in "thread.hpp"
> [9] thread_proxy(0x810a288), at 0xfea78ce4
> [10] _thr_setup(0xfe670200), at 0xfee159b9
> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
> 0xfe77ef14), at 0xfee15ca0
>
> Do you think it's worth repeating this test using gcc/linux, or do you
> think
> that this is just a limitation of using Python with threads?
>
> Thanks again,
>
> Paul
>
>
> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid at
> 0xfec453ed
> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
> Current function is Ticker::operator()
> 28 PyGILState_Release(state_);
> (dbx) where
> current thread: t@2
> [1] sem_invalid(0x0), at 0xfec453ed
> [2] _sem_post(0x0), at 0xfec454c4
> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
> 0x0, 0x80a3140), at 0xfef43eba
> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
> "thread.hpp"
> [9] thread_proxy(0x810a288), at 0xfea78ce4
> [10] _thr_setup(0xfe670200), at 0xfee159b9
> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
> 0xfe77ef14), at 0xfee15ca0
>
>
>
> "Paul Scruby" <paul@...> wrote in message
> news:h2sgic$ad4$1@......
>> Hiya,
>>
>> That's fantastic, all I needed to do was to put PyGILState_Ensure();
>> before my virtual function calls into python from another thread and my
>> program no longer crashes. Problem solved, isn't boost::python great!
>>
>> Many thanks,
>>
>> Paul
>>
>>
>> "Renato Araujo" <renatox@...> wrote in message
>> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
>> Hi Paul
>>
>> In my bindings I had a problem like this, to solve I created a simple
>> class like that:
>>
>> class thread_locker
>> {
>> thread_locker()
>> {
>> if (thread_support::enabled())
>> m_gstate = PyGILState_Ensure();
>> }
>>
>> ~thread_locker()
>> {
>> if (thread_support::enabled())
>> PyGILState_Release(m_gstate);
>> }
>> };
>>
>> then in my wrapper virtual implementation I did this:
>>
>> ...
>> void wrapper::virtual_func(..)
>> {
>> thread_locker lock;
>> .. my code ..
>> }
>> ....
>>
>> this solve my problems with call of virtual functions in thread
>> enviroment.
>>
>> BR
>>
>>
>>
>>
>> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:
>>> Whoops, I think this problem is a little uglier than I thought, since
>>> you
>>> overrode the onTick() function in python with a call to print, which
>>> needs access to the interpreter in your new thread. See the link Thomas
>>> posted for dealing with the GIL (along with worrying about any possible
>>> garbage collection issues). Now I remember why I kept my C++ threads
>>> isolated from Python stuff....sorry, it's been a while....
>>>
>>> Bill
>>> ________________________________________
>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of
>>> William Ladwig [wladwig@...]
>>> Sent: Saturday, July 04, 2009 1:34 PM
>>> To: Development of Python/C++ integration
>>> Subject: Re: [C++-sig] boost::python and threads
>>>
>>> It looks to me like you have a garbage collection problem going on. If
>>> you create a wrapped c++ object in python, then python is going to own
>>> the object and will destroy it when its reference count goes to 0. In
>>> your python example script at the bottom, you call the Ticker's run()
>>> function, which from the python point of view, returns quickly and the
>>> script ends. Python has no idea that you spawned off a new thread from
>>> the C++ side, so when the script ends, python destroys the object and
>>> now
>>> you have a problem. One way that you can check to see if this is what is
>>> going on is to add this to the bottom of the test script and see if the
>>> crashes go away:
>>>
>>> # Warning, you'll need to kill this script manually
>>> import time
>>> while True:
>>> time.sleep(1)
>>>
>>> Generally when I want to fire off a new C++ thread, I hold any objects
>>> that the thread needs in an auto_ptr (see held type for class wrappers),
>>> take ownership of them in C++ (see the FAQ in the documentation) and
>>> start the thread from the C++ side. Or, you can also create C++ wrappers
>>> which accept shared_ptr arguments, while holding your classes in
>>> shared_ptrs, and this should handle the reference counting as well.
>>> Unfortunately, this may require some interface changes to what you have
>>> already written (or possibly some clever wrapping).
>>>
>>> Hope this helps,
>>> Bill
>>>
>>>
>>> ________________________________________
>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
>>> Scruby [paul@...]
>>> Sent: Friday, July 03, 2009 6:15 AM
>>> To: cplusplus-sig@...
>>> Subject: [C++-sig] boost::python and threads
>>>
>>> I am having some problems using boost::python with boost::thread. I'm
>>> using
>>> threads because I want to run some tasks in the background when I'm
>>> using
>>> the Python's interactive shell. However, when I use get_override() to
>>> call
>>> a Python method from another boost::thread it crashes internally. For
>>> example:
>>>
>>> #include <boost/python.hpp>
>>> #include <boost/thread/thread.hpp>
>>> #include <boost/thread/xtime.hpp>
>>>
>>> using namespace boost::python;
>>>
>>> class Ticker
>>> : public wrapper<Ticker>
>>> {
>>> private:
>>> bool run_;
>>> volatile bool * running_;
>>> boost::thread * thread_;
>>> boost::xtime xt_;
>>> public:
>>> Ticker() : running_(&run_) { *running_ = false; }
>>>
>>> void operator()()
>>> {
>>> while (*running_)
>>> {
>>> boost::xtime_get(&xt_, boost::TIME_UTC);
>>> ++xt_.sec;
>>> boost::thread::sleep(xt_);
>>> onTick();
>>> }
>>> }
>>>
>>> void run()
>>> {
>>> if (*running_ == false)
>>> {
>>> *running_ = true;
>>> thread_ = new boost::thread(*this);
>>> }
>>> }
>>>
>>> void stop()
>>> {
>>> if (*running_ == true)
>>> {
>>> *running_ = false;
>>> thread_->join();
>>> delete thread_;
>>> }
>>> }
>>>
>>> virtual void onTick() { get_override("onTick")(); }
>>> void default_onTick() {}
>>> };
>>>
>>> BOOST_PYTHON_MODULE(tick)
>>> {
>>> class_<Ticker, boost::noncopyable> ("Ticker")
>>> .def("run", &Ticker::run)
>>> .def("stop", &Ticker::stop)
>>> .def("onTick", &Ticker::default_onTick);
>>> }
>>>
>>> Here is a test script that which will crash when you import it into
>>> Python's
>>> interactive shell.
>>>
>>> from tick import Ticker
>>>
>>> class MyTicker(Ticker):
>>> def onTick(self):
>>> print "Each second"
>>>
>>> myticker = MyTicker()
>>> myticker.run()
>>>
>>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>>> on
>>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>>> on
>>> Windows XP.
>>>
>>> The call-stack in dbx on Solaris:
>>>
>>> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
>>> PyErr_Restore at 0xfef38fa1
>>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>>> Current function is boost::python::override::operator()
>>> 99 detail::method_result x(
>>>
>>> (dbx) where
>>> current thread: t@2
>>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>>> 0x80652fc), at 0xfef38fa1
>>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>>> 0xfef2bf02
>>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>>> "override.hpp"
>>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>>> 56 in "thread.hpp"
>>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>>
>>> The call-stack in Visual Studio 2008:
>>>
>>> python26.dll!1e013595()
>>> [Frames below may be incorrect and/or missing, no symbols loaded for
>>> python26.dll]
>>> python26.dll!1e09ee7d()
>>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>>> C++
>>> 00f3fd64()
>>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>>> param=0x00245f30) Line 168 C++
>>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>>> kernel32.dll!7c80b729()
>>>
>>>
>>> Have a missed a trick using the wrapper, or does boost::python not
>>> support
>>> threading?
>>>
>>> Many thanks,
>>>
>>> Paul
>>>
>>>
>>>
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> Cplusplus-sig@...
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>
>>
>>
>>
>> --
>> Renato Araujo Oliveira Filho
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Renato Araujo]

Ok I made 2 modifications and I got this working here.

Try this.

51,54c50
<      virtual void onTick() {
<          if (object o = get_override("onTick"))
<              o();
<       }
---
>      virtual void onTick() { get_override("onTick")(); }
60d55
<      PyEval_InitThreads();
65a61
>

BR




On Wed, Jul 8, 2009 at 12:48 PM, Paul Scruby<paul@...> wrote:

> Hi Renato,
>
> Okay, I have installed Python 2.4.2 with gcc 4.1.2 on Linux and it's
> crashing in the same place as on Solaris.
>
>  Program received signal SIGSEGV, Segmentation fault.
>  [Switching to Thread 0x41074940 (LWP 12004)]
>  0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
>  (gdb) where
>  #0  0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
>  #1  0x000000308b2b78e9 in PyThread_release_lock ()
>     from /usr/lib64/libpython2.4.so.1.0
>  #2  0x00002b46531d9f82 in Ticker::operator() (this=0xbd0f478) at
> ticker.cc:28
>  #3  0x00002b46531d9fb0 in boost::detail::thread_data<Ticker>::run (
>      this=0xbd0f370)
>      at /opt/atm/include/boost-1_39/boost/thread/detail/thread.hpp:56
>  #4  0x00002b46533ee14b in thread_proxy ()
>     from /opt/atm/lib64/libboost_thread-gcc41-mt-1_39.so.1.39.0
>  #5  0x0000003077406367 in start_thread () from /lib64/libpthread.so.0
>  #6  0x00000030768d2f7d in clone () from /lib64/libc.so.6
>
> With the global interpret lock added my code now looks like this...
>
>  #include <boost/python.hpp>
>  #include <boost/thread/thread.hpp>
>  #include <boost/thread/xtime.hpp>
>  using namespace boost::python;
>
>  class Ticker
>      :    public wrapper<Ticker>
>  {
>  private:
>      bool run_;
>      volatile bool * running_;
>      boost::thread * thread_;
>      boost::xtime xt_;
>      PyGILState_STATE state_;
>  public:
>      Ticker() :running_(&run_) { *running_ = false; }
>
>      void operator()()
>      {
>          while (*running_)
>          {
>              boost::xtime_get(&xt_, boost::TIME_UTC);
>              ++xt_.sec;
>              boost::thread::sleep(xt_);
>              state_ = PyGILState_Ensure();
>              onTick();
>              PyGILState_Release(state_);
>          }
>      }
>
>      void run()
>      {
>          if (*running_ == false)
>          {
>              *running_ = true;
>              thread_ = new boost::thread(*this);
>          }
>      }
>
>      void stop()
>      {
>          if (*running_ == true)
>          {
>              *running_ = false;
>              thread_->join();
>              delete thread_;
>          }
>      }
>
>      virtual void onTick() { get_override("onTick")(); }
>      void default_onTick() {}
>  };
>
>  BOOST_PYTHON_MODULE(tick)
>  {
>      class_<Ticker, boost::noncopyable> ("Ticker")
>          .def("run", &Ticker::run)
>          .def("stop", &Ticker::stop)
>          .def("onTick", &Ticker::default_onTick);
>  }
>
>
> Thanks again,
>
> Paul
>
>
> "Renato Araujo" <renatox@...> wrote in message
> news:95291a80907060619u5bff0dcey56947d1ac848cc8e@......
> I'm using gcc/linux and python >= 2.4 and works fine for me.
>
>
>
> On Mon, Jul 6, 2009 at 7:39 AM, Paul Scruby<paul@...> wrote:
>> Hello again,
>>
>> Sorry, I spoke too soon. The good news is that wrapping my virtual method
>> calls into Python between PyGILState_Ensure() and PyGILState_Release()
>> fixed
>> the crash under Python2.6.2 on Windows, but not when I tested it again
>> under
>> Python 2.4.4 on Solaris. Under Python 2.4.4 on SolarisSolaris it now calls
>> and exits the virtual method in Python sucessfully, but then crashes when
>> C++ trys to release the global interpretor lock.
>>
>> The call-stack with the Sun C++ 5.9 compiler under Solaris is
>> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid
>> at 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t@2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
>> in "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>> Do you think it's worth repeating this test using gcc/linux, or do you
>> think
>> that this is just a limitation of using Python with threads?
>>
>> Thanks again,
>>
>> Paul
>>
>>
>> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid at
>> 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t@2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
>> "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>>
>>
>> "Paul Scruby" <paul@...> wrote in message
>> news:h2sgic$ad4$1@......
>>> Hiya,
>>>
>>> That's fantastic, all I needed to do was to put PyGILState_Ensure();
>>> before my virtual function calls into python from another thread and my
>>> program no longer crashes. Problem solved, isn't boost::python great!
>>>
>>> Many thanks,
>>>
>>> Paul
>>>
>>>
>>> "Renato Araujo" <renatox@...> wrote in message
>>> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
>>> Hi Paul
>>>
>>> In my bindings I had a problem like this, to solve I created a simple
>>> class like that:
>>>
>>> class thread_locker
>>> {
>>> thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> m_gstate = PyGILState_Ensure();
>>> }
>>>
>>> ~thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> PyGILState_Release(m_gstate);
>>> }
>>> };
>>>
>>> then in my wrapper virtual implementation I did this:
>>>
>>> ...
>>> void wrapper::virtual_func(..)
>>> {
>>> thread_locker lock;
>>> .. my code ..
>>> }
>>> ....
>>>
>>> this solve my problems with call of virtual functions in thread
>>> enviroment.
>>>
>>> BR
>>>
>>>
>>>
>>>
>>> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...> wrote:
>>>> Whoops, I think this problem is a little uglier than I thought, since
>>>> you
>>>> overrode the onTick() function in python with a call to print, which
>>>> needs access to the interpreter in your new thread. See the link Thomas
>>>> posted for dealing with the GIL (along with worrying about any possible
>>>> garbage collection issues). Now I remember why I kept my C++ threads
>>>> isolated from Python stuff....sorry, it's been a while....
>>>>
>>>> Bill
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of
>>>> William Ladwig [wladwig@...]
>>>> Sent: Saturday, July 04, 2009 1:34 PM
>>>> To: Development of Python/C++ integration
>>>> Subject: Re: [C++-sig] boost::python and threads
>>>>
>>>> It looks to me like you have a garbage collection problem going on. If
>>>> you create a wrapped c++ object in python, then python is going to own
>>>> the object and will destroy it when its reference count goes to 0. In
>>>> your python example script at the bottom, you call the Ticker's run()
>>>> function, which from the python point of view, returns quickly and the
>>>> script ends. Python has no idea that you spawned off a new thread from
>>>> the C++ side, so when the script ends, python destroys the object and
>>>> now
>>>> you have a problem. One way that you can check to see if this is what is
>>>> going on is to add this to the bottom of the test script and see if the
>>>> crashes go away:
>>>>
>>>> # Warning, you'll need to kill this script manually
>>>> import time
>>>> while True:
>>>> time.sleep(1)
>>>>
>>>> Generally when I want to fire off a new C++ thread, I hold any objects
>>>> that the thread needs in an auto_ptr (see held type for class wrappers),
>>>> take ownership of them in C++ (see the FAQ in the documentation) and
>>>> start the thread from the C++ side. Or, you can also create C++ wrappers
>>>> which accept shared_ptr arguments, while holding your classes in
>>>> shared_ptrs, and this should handle the reference counting as well.
>>>> Unfortunately, this may require some interface changes to what you have
>>>> already written (or possibly some clever wrapping).
>>>>
>>>> Hope this helps,
>>>> Bill
>>>>
>>>>
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
>>>> Scruby [paul@...]
>>>> Sent: Friday, July 03, 2009 6:15 AM
>>>> To: cplusplus-sig@...
>>>> Subject: [C++-sig] boost::python and threads
>>>>
>>>> I am having some problems using boost::python with boost::thread. I'm
>>>> using
>>>> threads because I want to run some tasks in the background when I'm
>>>> using
>>>> the Python's interactive shell. However, when I use get_override() to
>>>> call
>>>> a Python method from another boost::thread it crashes internally. For
>>>> example:
>>>>
>>>> #include <boost/python.hpp>
>>>> #include <boost/thread/thread.hpp>
>>>> #include <boost/thread/xtime.hpp>
>>>>
>>>> using namespace boost::python;
>>>>
>>>> class Ticker
>>>> : public wrapper<Ticker>
>>>> {
>>>> private:
>>>> bool run_;
>>>> volatile bool * running_;
>>>> boost::thread * thread_;
>>>> boost::xtime xt_;
>>>> public:
>>>> Ticker() : running_(&run_) { *running_ = false; }
>>>>
>>>> void operator()()
>>>> {
>>>> while (*running_)
>>>> {
>>>> boost::xtime_get(&xt_, boost::TIME_UTC);
>>>> ++xt_.sec;
>>>> boost::thread::sleep(xt_);
>>>> onTick();
>>>> }
>>>> }
>>>>
>>>> void run()
>>>> {
>>>> if (*running_ == false)
>>>> {
>>>> *running_ = true;
>>>> thread_ = new boost::thread(*this);
>>>> }
>>>> }
>>>>
>>>> void stop()
>>>> {
>>>> if (*running_ == true)
>>>> {
>>>> *running_ = false;
>>>> thread_->join();
>>>> delete thread_;
>>>> }
>>>> }
>>>>
>>>> virtual void onTick() { get_override("onTick")(); }
>>>> void default_onTick() {}
>>>> };
>>>>
>>>> BOOST_PYTHON_MODULE(tick)
>>>> {
>>>> class_<Ticker, boost::noncopyable> ("Ticker")
>>>> .def("run", &Ticker::run)
>>>> .def("stop", &Ticker::stop)
>>>> .def("onTick", &Ticker::default_onTick);
>>>> }
>>>>
>>>> Here is a test script that which will crash when you import it into
>>>> Python's
>>>> interactive shell.
>>>>
>>>> from tick import Ticker
>>>>
>>>> class MyTicker(Ticker):
>>>> def onTick(self):
>>>> print "Each second"
>>>>
>>>> myticker = MyTicker()
>>>> myticker.run()
>>>>
>>>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>>>> on
>>>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>>>> on
>>>> Windows XP.
>>>>
>>>> The call-stack in dbx on Solaris:
>>>>
>>>> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
>>>> PyErr_Restore at 0xfef38fa1
>>>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>>>> Current function is boost::python::override::operator()
>>>> 99 detail::method_result x(
>>>>
>>>> (dbx) where
>>>> current thread: t@2
>>>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>>>> 0x80652fc), at 0xfef38fa1
>>>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>>>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>>>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>>>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>>>> 0xfef2bf02
>>>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>>>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>>>> "override.hpp"
>>>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>>>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>>>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>>>> 56 in "thread.hpp"
>>>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>>>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>>>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>>>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>>>
>>>> The call-stack in Visual Studio 2008:
>>>>
>>>> python26.dll!1e013595()
>>>> [Frames below may be incorrect and/or missing, no symbols loaded for
>>>> python26.dll]
>>>> python26.dll!1e09ee7d()
>>>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>>>> C++
>>>> 00f3fd64()
>>>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>>>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>>>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>>>> param=0x00245f30) Line 168 C++
>>>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>>>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>>>> kernel32.dll!7c80b729()
>>>>
>>>>
>>>> Have a missed a trick using the wrapper, or does boost::python not
>>>> support
>>>> threading?
>>>>
>>>> Many thanks,
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>>
>>>
>>>
>>>
>>> --
>>> Renato Araujo Oliveira Filho
>>
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig@...
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Renato Araujo Oliveira Filho
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Paul Scruby]

Hi Renato,

It's all working now, tested on gcc, sun C++ and visual studio.

Thanks again,

Paul


"Renato Araujo" <renatox@...> wrote in message
news:95291a80907081023w5c1959d4u9bb7d5a4258d9240@......
Ok I made 2 modifications and I got this working here.

Try this.

51,54c50
<      virtual void onTick() {
<          if (object o = get_override("onTick"))
<              o();
<       }
---
>      virtual void onTick() { get_override("onTick")(); }
60d55
<      PyEval_InitThreads();
65a61
>

BR




On Wed, Jul 8, 2009 at 12:48 PM, Paul Scruby<paul@...> wrote:

> Hi Renato,
>
> Okay, I have installed Python 2.4.2 with gcc 4.1.2 on Linux and it's
> crashing in the same place as on Solaris.
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x41074940 (LWP 12004)]
> 0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
> (gdb) where
> #0 0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
> #1 0x000000308b2b78e9 in PyThread_release_lock ()
> from /usr/lib64/libpython2.4.so.1.0
> #2 0x00002b46531d9f82 in Ticker::operator() (this=0xbd0f478) at
> ticker.cc:28
> #3 0x00002b46531d9fb0 in boost::detail::thread_data<Ticker>::run (
> this=0xbd0f370)
> at /opt/atm/include/boost-1_39/boost/thread/detail/thread.hpp:56
> #4 0x00002b46533ee14b in thread_proxy ()
> from /opt/atm/lib64/libboost_thread-gcc41-mt-1_39.so.1.39.0
> #5 0x0000003077406367 in start_thread () from /lib64/libpthread.so.0
> #6 0x00000030768d2f7d in clone () from /lib64/libc.so.6
>
> With the global interpret lock added my code now looks like this...
>
> #include <boost/python.hpp>
> #include <boost/thread/thread.hpp>
> #include <boost/thread/xtime.hpp>
> using namespace boost::python;
>
> class Ticker
> : public wrapper<Ticker>
> {
> private:
> bool run_;
> volatile bool * running_;
> boost::thread * thread_;
> boost::xtime xt_;
> PyGILState_STATE state_;
> public:
> Ticker() :running_(&run_) { *running_ = false; }
>
> void operator()()
> {
> while (*running_)
> {
> boost::xtime_get(&xt_, boost::TIME_UTC);
> ++xt_.sec;
> boost::thread::sleep(xt_);
> state_ = PyGILState_Ensure();
> onTick();
> PyGILState_Release(state_);
> }
> }
>
> void run()
> {
> if (*running_ == false)
> {
> *running_ = true;
> thread_ = new boost::thread(*this);
> }
> }
>
> void stop()
> {
> if (*running_ == true)
> {
> *running_ = false;
> thread_->join();
> delete thread_;
> }
> }
>
> virtual void onTick() { get_override("onTick")(); }
> void default_onTick() {}
> };
>
> BOOST_PYTHON_MODULE(tick)
> {
> class_<Ticker, boost::noncopyable> ("Ticker")
> .def("run", &Ticker::run)
> .def("stop", &Ticker::stop)
> .def("onTick", &Ticker::default_onTick);
> }
>
>
> Thanks again,
>
> Paul
>
>
> "Renato Araujo" <renatox@...> wrote in message
> news:95291a80907060619u5bff0dcey56947d1ac848cc8e@......
> I'm using gcc/linux and python >= 2.4 and works fine for me.
>
>
>
> On Mon, Jul 6, 2009 at 7:39 AM, Paul Scruby<paul@...> wrote:
>> Hello again,
>>
>> Sorry, I spoke too soon. The good news is that wrapping my virtual method
>> calls into Python between PyGILState_Ensure() and PyGILState_Release()
>> fixed
>> the crash under Python2.6.2 on Windows, but not when I tested it again
>> under
>> Python 2.4.4 on Solaris. Under Python 2.4.4 on SolarisSolaris it now
>> calls
>> and exits the virtual method in Python sucessfully, but then crashes when
>> C++ trys to release the global interpretor lock.
>>
>> The call-stack with the Sun C++ 5.9 compiler under Solaris is
>> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid
>> at 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t@2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
>> in "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>> Do you think it's worth repeating this test using gcc/linux, or do you
>> think
>> that this is just a limitation of using Python with threads?
>>
>> Thanks again,
>>
>> Paul
>>
>>
>> t@2 (l@2) signal SEGV (no mapping at the fault address) in sem_invalid at
>> 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t@2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
>> "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>>
>>
>> "Paul Scruby" <paul@...> wrote in message
>> news:h2sgic$ad4$1@......
>>> Hiya,
>>>
>>> That's fantastic, all I needed to do was to put PyGILState_Ensure();
>>> before my virtual function calls into python from another thread and my
>>> program no longer crashes. Problem solved, isn't boost::python great!
>>>
>>> Many thanks,
>>>
>>> Paul
>>>
>>>
>>> "Renato Araujo" <renatox@...> wrote in message
>>> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91@......
>>> Hi Paul
>>>
>>> In my bindings I had a problem like this, to solve I created a simple
>>> class like that:
>>>
>>> class thread_locker
>>> {
>>> thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> m_gstate = PyGILState_Ensure();
>>> }
>>>
>>> ~thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> PyGILState_Release(m_gstate);
>>> }
>>> };
>>>
>>> then in my wrapper virtual implementation I did this:
>>>
>>> ...
>>> void wrapper::virtual_func(..)
>>> {
>>> thread_locker lock;
>>> .. my code ..
>>> }
>>> ....
>>>
>>> this solve my problems with call of virtual functions in thread
>>> enviroment.
>>>
>>> BR
>>>
>>>
>>>
>>>
>>> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig@...>
>>> wrote:
>>>> Whoops, I think this problem is a little uglier than I thought, since
>>>> you
>>>> overrode the onTick() function in python with a call to print, which
>>>> needs access to the interpreter in your new thread. See the link Thomas
>>>> posted for dealing with the GIL (along with worrying about any possible
>>>> garbage collection issues). Now I remember why I kept my C++ threads
>>>> isolated from Python stuff....sorry, it's been a while....
>>>>
>>>> Bill
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of
>>>> William Ladwig [wladwig@...]
>>>> Sent: Saturday, July 04, 2009 1:34 PM
>>>> To: Development of Python/C++ integration
>>>> Subject: Re: [C++-sig] boost::python and threads
>>>>
>>>> It looks to me like you have a garbage collection problem going on. If
>>>> you create a wrapped c++ object in python, then python is going to own
>>>> the object and will destroy it when its reference count goes to 0. In
>>>> your python example script at the bottom, you call the Ticker's run()
>>>> function, which from the python point of view, returns quickly and the
>>>> script ends. Python has no idea that you spawned off a new thread from
>>>> the C++ side, so when the script ends, python destroys the object and
>>>> now
>>>> you have a problem. One way that you can check to see if this is what
>>>> is
>>>> going on is to add this to the bottom of the test script and see if the
>>>> crashes go away:
>>>>
>>>> # Warning, you'll need to kill this script manually
>>>> import time
>>>> while True:
>>>> time.sleep(1)
>>>>
>>>> Generally when I want to fire off a new C++ thread, I hold any objects
>>>> that the thread needs in an auto_ptr (see held type for class
>>>> wrappers),
>>>> take ownership of them in C++ (see the FAQ in the documentation) and
>>>> start the thread from the C++ side. Or, you can also create C++
>>>> wrappers
>>>> which accept shared_ptr arguments, while holding your classes in
>>>> shared_ptrs, and this should handle the reference counting as well.
>>>> Unfortunately, this may require some interface changes to what you have
>>>> already written (or possibly some clever wrapping).
>>>>
>>>> Hope this helps,
>>>> Bill
>>>>
>>>>
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com@...
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com@...] On Behalf Of Paul
>>>> Scruby [paul@...]
>>>> Sent: Friday, July 03, 2009 6:15 AM
>>>> To: cplusplus-sig@...
>>>> Subject: [C++-sig] boost::python and threads
>>>>
>>>> I am having some problems using boost::python with boost::thread. I'm
>>>> using
>>>> threads because I want to run some tasks in the background when I'm
>>>> using
>>>> the Python's interactive shell. However, when I use get_override() to
>>>> call
>>>> a Python method from another boost::thread it crashes internally. For
>>>> example:
>>>>
>>>> #include <boost/python.hpp>
>>>> #include <boost/thread/thread.hpp>
>>>> #include <boost/thread/xtime.hpp>
>>>>
>>>> using namespace boost::python;
>>>>
>>>> class Ticker
>>>> : public wrapper<Ticker>
>>>> {
>>>> private:
>>>> bool run_;
>>>> volatile bool * running_;
>>>> boost::thread * thread_;
>>>> boost::xtime xt_;
>>>> public:
>>>> Ticker() : running_(&run_) { *running_ = false; }
>>>>
>>>> void operator()()
>>>> {
>>>> while (*running_)
>>>> {
>>>> boost::xtime_get(&xt_, boost::TIME_UTC);
>>>> ++xt_.sec;
>>>> boost::thread::sleep(xt_);
>>>> onTick();
>>>> }
>>>> }
>>>>
>>>> void run()
>>>> {
>>>> if (*running_ == false)
>>>> {
>>>> *running_ = true;
>>>> thread_ = new boost::thread(*this);
>>>> }
>>>> }
>>>>
>>>> void stop()
>>>> {
>>>> if (*running_ == true)
>>>> {
>>>> *running_ = false;
>>>> thread_->join();
>>>> delete thread_;
>>>> }
>>>> }
>>>>
>>>> virtual void onTick() { get_override("onTick")(); }
>>>> void default_onTick() {}
>>>> };
>>>>
>>>> BOOST_PYTHON_MODULE(tick)
>>>> {
>>>> class_<Ticker, boost::noncopyable> ("Ticker")
>>>> .def("run", &Ticker::run)
>>>> .def("stop", &Ticker::stop)
>>>> .def("onTick", &Ticker::default_onTick);
>>>> }
>>>>
>>>> Here is a test script that which will crash when you import it into
>>>> Python's
>>>> interactive shell.
>>>>
>>>> from tick import Ticker
>>>>
>>>> class MyTicker(Ticker):
>>>> def onTick(self):
>>>> print "Each second"
>>>>
>>>> myticker = MyTicker()
>>>> myticker.run()
>>>>
>>>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>>>> on
>>>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>>>> on
>>>> Windows XP.
>>>>
>>>> The call-stack in dbx on Solaris:
>>>>
>>>> >>> t@2 (l@2) signal SEGV (no mapping at the fault address) in
>>>> PyErr_Restore at 0xfef38fa1
>>>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>>>> Current function is boost::python::override::operator()
>>>> 99 detail::method_result x(
>>>>
>>>> (dbx) where
>>>> current thread: t@2
>>>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>>>> 0x80652fc), at 0xfef38fa1
>>>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>>>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>>>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>>>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>>>> 0xfef2bf02
>>>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>>>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99
>>>> in
>>>> "override.hpp"
>>>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>>>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>>>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>>>> 56 in "thread.hpp"
>>>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>>>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>>>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>>>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>>>
>>>> The call-stack in Visual Studio 2008:
>>>>
>>>> python26.dll!1e013595()
>>>> [Frames below may be incorrect and/or missing, no symbols loaded for
>>>> python26.dll]
>>>> python26.dll!1e09ee7d()
>>>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>>>> C++
>>>> 00f3fd64()
>>>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>>>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>>>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>>>> param=0x00245f30) Line 168 C++
>>>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>>>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>>>> kernel32.dll!7c80b729()
>>>>
>>>>
>>>> Have a missed a trick using the wrapper, or does boost::python not
>>>> support
>>>> threading?
>>>>
>>>> Many thanks,
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig@...
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>>
>>>
>>>
>>>
>>> --
>>> Renato Araujo Oliveira Filho
>>
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig@...
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Renato Araujo Oliveira Filho
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

--
Renato Araujo Oliveira Filho



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig