Segfault using concurrent threads

[Paul Balomiri]

Hi,

I encountered a similar problem to
http://lists.osgeo.org/pipermail/geos-devel/2009-July/004280.html , which is
at the exact same execution point.

I am using shapely 1.0.13. As the lib is linked with PyDLL into
python, the GIL (Global Interpreter Log) is used to schedule library
entrance ( = no python code (in any thread) runs concurrently),
so my first guess was, that an init error or an error in handling the
global Context might cause the segfault.

The backtrace I got from gdb is


[Switching to Thread 0x7fffe7f076f0 (LWP 6598)]
geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
os=@0x7fffeff11f50) at WKBWriter.cpp:77
77              switch (g.getGeometryTypeId()) {
Current language:  auto; currently c++
(gdb) where
#0  geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
os=@0x7fffeff11f50) at WKBWriter.cpp:77
#1  0x00007fffe6883549 in GEOSGeomToWKB_buf_r (extHandle=0xa5beb0,
g=0xd8005270, size=0x14407f0) at geos_ts_c.cpp:948
#2  0x00007fffe6aa53b8 in ffi_call_unix64 () at
/build/buildd/python2.6-2.6.2/Modules/_ctypes/libffi/src/x86/unix64.S:75
Python stacktrace:

[Python Stack]

#46 0x0000000000417ae9 in _start () at ../sysdeps/x86_64/elf/start.S:113

A full stacktrace is included as an attachment. I have augmented the
stacktrace with the python
stackframe where the stack frame is in python code.

I was wondering why GEOSGeomToWKB_buf is called, but
GEOSGeomToWKB_buf_r appears on the stacktrace.

I do use threads in the application, but  only the main thread has
calling points to shapely.
Also the application uses SQLAlchemy with the user defined Geometry
type from shapely. The gluecode is part of mapfish.

Perhaps you have some ideas about why the segfault shows up. I'm not
familiar with the way
shapely handles GEOS c pointers. In gdb the geos geometry object
g=@0xd8005270 points to an adress

trying to access the geometry object reveals

(gdb) p g
$1 = (const geos::geom::Geometry &) @0xd8005270: <error reading variable>


regards,
Paul


_______________________________________________
Community mailing list
Community@...
http://lists.gispython.org/mailman/listinfo/community

[Sean Gillies-2]

On Oct 3, 2009, at 7:52 PM, Paul Balomiri wrote:

> Hi,
>
> I encountered a similar problem to
> http://lists.osgeo.org/pipermail/geos-devel/2009-July/004280.html ,  
> which is
> at the exact same execution point.
>
> I am using shapely 1.0.13. As the lib is linked with PyDLL into
> python, the GIL (Global Interpreter Log) is used to schedule library
> entrance ( = no python code (in any thread) runs concurrently),
> so my first guess was, that an init error or an error in handling the
> global Context might cause the segfault.
>
> The backtrace I got from gdb is
>
>
> [Switching to Thread 0x7fffe7f076f0 (LWP 6598)]
> geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
> os=@0x7fffeff11f50) at WKBWriter.cpp:77
> 77              switch (g.getGeometryTypeId()) {
> Current language:  auto; currently c++
> (gdb) where
> #0  geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
> os=@0x7fffeff11f50) at WKBWriter.cpp:77
> #1  0x00007fffe6883549 in GEOSGeomToWKB_buf_r (extHandle=0xa5beb0,
> g=0xd8005270, size=0x14407f0) at geos_ts_c.cpp:948
> #2  0x00007fffe6aa53b8 in ffi_call_unix64 () at
> /build/buildd/python2.6-2.6.2/Modules/_ctypes/libffi/src/x86/
> unix64.S:75
> Python stacktrace:
>
> [Python Stack]
>
> #46 0x0000000000417ae9 in _start () at ../sysdeps/x86_64/elf/start.S:
> 113
>
> A full stacktrace is included as an attachment. I have augmented the
> stacktrace with the python
> stackframe where the stack frame is in python code.
>
> I was wondering why GEOSGeomToWKB_buf is called, but
> GEOSGeomToWKB_buf_r appears on the stacktrace.
>
> I do use threads in the application, but  only the main thread has
> calling points to shapely.
> Also the application uses SQLAlchemy with the user defined Geometry
> type from shapely. The gluecode is part of mapfish.
>
> Perhaps you have some ideas about why the segfault shows up. I'm not
> familiar with the way
> shapely handles GEOS c pointers. In gdb the geos geometry object
> g=@0xd8005270 points to an adress
>
> trying to access the geometry object reveals
>
> (gdb) p g
> $1 = (const geos::geom::Geometry &) @0xd8005270: <error reading  
> variable>
>
>
> regards,
> Paul

Hi Paul,

I'm sorry you're experiencing trouble with Shapely. If you can write a  
Python program that doesn't depend on MapFish or SQLAlchemy or PostGIS  
(I don't have that stack running right now) that reliably crashes,  
I'll get right on it. Maybe it's just a matter of writing out WKB from  
multiple threads at once.

If you're using GEOS 3.1, the non-reentrant API is just a thin layer  
over the reentrant one, so EOSGeomToWKB_buf_r will always be called.

Cheers,

--
Sean

_______________________________________________
Community mailing list
Community@...
http://lists.gispython.org/mailman/listinfo/community

[Paul Balomiri]

Hi,
Now having gone deeper into the matter i found out a bug,
and can further get into details about when a segfault still occurs.
The relevant fact i did not mention last time is that i'm running
shapely on a 64 bit CPU.

1)  http://trac.gispython.org/lab/browser/Shapely/tags/rel-1.0.13/shapely/geometry/base.py
ay line 352 you use

size = c_int()

and later on pass it as size size arg to GEOS' GEOSGeomToWKB_buf.
This is wrong on 64 bit processors.
Please use size = c_size_t() (8 byte long on 64 bit)

2) I could find what may be a corelated symptom, and easily to check:
the __geom__ type is int. This causes segfaults when the actual ponter type has
a value >2^32-1. I could not follow the code to the point where __geom__ is set,
but am searching for it. Perhaps you can give me a hint ?


2009/10/5 Sean Gillies <sean.gillies@...>:

> On Oct 3, 2009, at 7:52 PM, Paul Balomiri wrote:
>
>> Hi,
>>
>> I encountered a similar problem to
>> http://lists.osgeo.org/pipermail/geos-devel/2009-July/004280.html ,
>> which is
>> at the exact same execution point.
>>
>> I am using shapely 1.0.13. As the lib is linked with PyDLL into
>> python, the GIL (Global Interpreter Log) is used to schedule library
>> entrance ( = no python code (in any thread) runs concurrently),
>> so my first guess was, that an init error or an error in handling the
>> global Context might cause the segfault.
>>
>> The backtrace I got from gdb is
>>
>>
>> [Switching to Thread 0x7fffe7f076f0 (LWP 6598)]
>> geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>> 77              switch (g.getGeometryTypeId()) {
>> Current language:  auto; currently c++
>> (gdb) where
>> #0  geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>> #1  0x00007fffe6883549 in GEOSGeomToWKB_buf_r (extHandle=0xa5beb0,
>> g=0xd8005270, size=0x14407f0) at geos_ts_c.cpp:948
>> #2  0x00007fffe6aa53b8 in ffi_call_unix64 () at
>> /build/buildd/python2.6-2.6.2/Modules/_ctypes/libffi/src/x86/
>> unix64.S:75
>> Python stacktrace:
>>
>> [Python Stack]
>>
>> #46 0x0000000000417ae9 in _start () at ../sysdeps/x86_64/elf/start.S:
>> 113
>>
>> A full stacktrace is included as an attachment. I have augmented the
>> stacktrace with the python
>> stackframe where the stack frame is in python code.
>>
>> I was wondering why GEOSGeomToWKB_buf is called, but
>> GEOSGeomToWKB_buf_r appears on the stacktrace.
>>
>> I do use threads in the application, but  only the main thread has
>> calling points to shapely.
>> Also the application uses SQLAlchemy with the user defined Geometry
>> type from shapely. The gluecode is part of mapfish.
>>
>> Perhaps you have some ideas about why the segfault shows up. I'm not
>> familiar with the way
>> shapely handles GEOS c pointers. In gdb the geos geometry object
>> g=@0xd8005270 points to an adress
>>
>> trying to access the geometry object reveals
>>
>> (gdb) p g
>> $1 = (const geos::geom::Geometry &) @0xd8005270: <error reading
>> variable>
>>
>>
>> regards,
>> Paul
>
> Hi Paul,
>
> I'm sorry you're experiencing trouble with Shapely. If you can write a
> Python program that doesn't depend on MapFish or SQLAlchemy or PostGIS
> (I don't have that stack running right now) that reliably crashes,
> I'll get right on it. Maybe it's just a matter of writing out WKB from
> multiple threads at once.
>
> If you're using GEOS 3.1, the non-reentrant API is just a thin layer
> over the reentrant one, so EOSGeomToWKB_buf_r will always be called.
>
> Cheers,
>
> --
> Sean
>
> _______________________________________________
> Community mailing list
> Community@...
> http://lists.gispython.org/mailman/listinfo/community
>

--
paulbalomiri@...
_______________________________________________
Community mailing list
Community@...
http://lists.gispython.org/mailman/listinfo/community

[Paul Balomiri]

Hi,

I tried to find out where exactly BaseGeometry.__geom__ becomes int type.

and here is what I found out:

>>> p=Point(1.0,2.3)
pointaddr=10373776
>>> p
<shapely.geometry.point.Point object at 0x7f7f7e8ebed0>
>>> p.__geom__.__class__
<type 'int'>


OK, now going into the constructor of Point I could pinpoint
Line #53 in point.py

Creating a sequence there yielda an int instead of a c_void_p:

>>> cs = lgeos.GEOSCoordSeq_create(1, 2)
>>> cs.__class__
<type 'int'>

This seems strance as in
http://trac.gispython.org/lab/browser/Shapely/tags/rel-1.0.13/shapely/ctypes_declarations.py#L26
tou correctly defined

 lgeos.GEOSCoordSeq_create.restype = ctypes.c_void_p

So the only thing coming to my mind is that the c_type_declarations
might somehow not be included ?

Do you have any hints where to go from here ?

Cheers Paul

2009/10/5 Paul Balomiri <paulbalomiri@...>:

> Hi,
> Now having gone deeper into the matter i found out a bug,
> and can further get into details about when a segfault still occurs.
> The relevant fact i did not mention last time is that i'm running
> shapely on a 64 bit CPU.
>
> 1)  http://trac.gispython.org/lab/browser/Shapely/tags/rel-1.0.13/shapely/geometry/base.py
> ay line 352 you use
>
> size = c_int()
>
> and later on pass it as size size arg to GEOS' GEOSGeomToWKB_buf.
> This is wrong on 64 bit processors.
> Please use size = c_size_t() (8 byte long on 64 bit)
>
> 2) I could find what may be a corelated symptom, and easily to check:
> the __geom__ type is int. This causes segfaults when the actual ponter type has
> a value >2^32-1. I could not follow the code to the point where __geom__ is set,
> but am searching for it. Perhaps you can give me a hint ?
>
>
> 2009/10/5 Sean Gillies <sean.gillies@...>:
>> On Oct 3, 2009, at 7:52 PM, Paul Balomiri wrote:
>>
>>> Hi,
>>>
>>> I encountered a similar problem to
>>> http://lists.osgeo.org/pipermail/geos-devel/2009-July/004280.html ,
>>> which is
>>> at the exact same execution point.
>>>
>>> I am using shapely 1.0.13. As the lib is linked with PyDLL into
>>> python, the GIL (Global Interpreter Log) is used to schedule library
>>> entrance ( = no python code (in any thread) runs concurrently),
>>> so my first guess was, that an init error or an error in handling the
>>> global Context might cause the segfault.
>>>
>>> The backtrace I got from gdb is
>>>
>>>
>>> [Switching to Thread 0x7fffe7f076f0 (LWP 6598)]
>>> geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>>> 77              switch (g.getGeometryTypeId()) {
>>> Current language:  auto; currently c++
>>> (gdb) where
>>> #0  geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>>> #1  0x00007fffe6883549 in GEOSGeomToWKB_buf_r (extHandle=0xa5beb0,
>>> g=0xd8005270, size=0x14407f0) at geos_ts_c.cpp:948
>>> #2  0x00007fffe6aa53b8 in ffi_call_unix64 () at
>>> /build/buildd/python2.6-2.6.2/Modules/_ctypes/libffi/src/x86/
>>> unix64.S:75
>>> Python stacktrace:
>>>
>>> [Python Stack]
>>>
>>> #46 0x0000000000417ae9 in _start () at ../sysdeps/x86_64/elf/start.S:
>>> 113
>>>
>>> A full stacktrace is included as an attachment. I have augmented the
>>> stacktrace with the python
>>> stackframe where the stack frame is in python code.
>>>
>>> I was wondering why GEOSGeomToWKB_buf is called, but
>>> GEOSGeomToWKB_buf_r appears on the stacktrace.
>>>
>>> I do use threads in the application, but  only the main thread has
>>> calling points to shapely.
>>> Also the application uses SQLAlchemy with the user defined Geometry
>>> type from shapely. The gluecode is part of mapfish.
>>>
>>> Perhaps you have some ideas about why the segfault shows up. I'm not
>>> familiar with the way
>>> shapely handles GEOS c pointers. In gdb the geos geometry object
>>> g=@0xd8005270 points to an adress
>>>
>>> trying to access the geometry object reveals
>>>
>>> (gdb) p g
>>> $1 = (const geos::geom::Geometry &) @0xd8005270: <error reading
>>> variable>
>>>
>>>
>>> regards,
>>> Paul
>>
>> Hi Paul,
>>
>> I'm sorry you're experiencing trouble with Shapely. If you can write a
>> Python program that doesn't depend on MapFish or SQLAlchemy or PostGIS
>> (I don't have that stack running right now) that reliably crashes,
>> I'll get right on it. Maybe it's just a matter of writing out WKB from
>> multiple threads at once.
>>
>> If you're using GEOS 3.1, the non-reentrant API is just a thin layer
>> over the reentrant one, so EOSGeomToWKB_buf_r will always be called.
>>
>> Cheers,
>>
>> --
>> Sean
>>
>> _______________________________________________
>> Community mailing list
>> Community@...
>> http://lists.gispython.org/mailman/listinfo/community
>>
>
>
>
> --
> paulbalomiri@...
>

--
paulbalomiri@...
_______________________________________________
Community mailing list
Community@...
http://lists.gispython.org/mailman/listinfo/community

[Paul Balomiri]

This thread is continued  using the subject
  Segmentation fault for 64 Bit platforms in to_wkb function
here: http://lists.gispython.org/pipermail/community/2009-October/002281.html

2009/10/5 Paul Balomiri <paulbalomiri@...>:

> Hi,
>
> I tried to find out where exactly BaseGeometry.__geom__ becomes int type.
>
> and here is what I found out:
>
>>>> p=Point(1.0,2.3)
> pointaddr=10373776
>>>> p
> <shapely.geometry.point.Point object at 0x7f7f7e8ebed0>
>>>> p.__geom__.__class__
> <type 'int'>
>
>
> OK, now going into the constructor of Point I could pinpoint
> Line #53 in point.py
>
> Creating a sequence there yielda an int instead of a c_void_p:
>
>>>> cs = lgeos.GEOSCoordSeq_create(1, 2)
>>>> cs.__class__
> <type 'int'>
>
> This seems strance as in
> http://trac.gispython.org/lab/browser/Shapely/tags/rel-1.0.13/shapely/ctypes_declarations.py#L26
> tou correctly defined
>
>  lgeos.GEOSCoordSeq_create.restype = ctypes.c_void_p
>
> So the only thing coming to my mind is that the c_type_declarations
> might somehow not be included ?
>
> Do you have any hints where to go from here ?
>
> Cheers Paul
>
> 2009/10/5 Paul Balomiri <paulbalomiri@...>:
>> Hi,
>> Now having gone deeper into the matter i found out a bug,
>> and can further get into details about when a segfault still occurs.
>> The relevant fact i did not mention last time is that i'm running
>> shapely on a 64 bit CPU.
>>
>> 1)  http://trac.gispython.org/lab/browser/Shapely/tags/rel-1.0.13/shapely/geometry/base.py
>> ay line 352 you use
>>
>> size = c_int()
>>
>> and later on pass it as size size arg to GEOS' GEOSGeomToWKB_buf.
>> This is wrong on 64 bit processors.
>> Please use size = c_size_t() (8 byte long on 64 bit)
>>
>> 2) I could find what may be a corelated symptom, and easily to check:
>> the __geom__ type is int. This causes segfaults when the actual ponter type has
>> a value >2^32-1. I could not follow the code to the point where __geom__ is set,
>> but am searching for it. Perhaps you can give me a hint ?
>>
>>
>> 2009/10/5 Sean Gillies <sean.gillies@...>:
>>> On Oct 3, 2009, at 7:52 PM, Paul Balomiri wrote:
>>>
>>>> Hi,
>>>>
>>>> I encountered a similar problem to
>>>> http://lists.osgeo.org/pipermail/geos-devel/2009-July/004280.html ,
>>>> which is
>>>> at the exact same execution point.
>>>>
>>>> I am using shapely 1.0.13. As the lib is linked with PyDLL into
>>>> python, the GIL (Global Interpreter Log) is used to schedule library
>>>> entrance ( = no python code (in any thread) runs concurrently),
>>>> so my first guess was, that an init error or an error in handling the
>>>> global Context might cause the segfault.
>>>>
>>>> The backtrace I got from gdb is
>>>>
>>>>
>>>> [Switching to Thread 0x7fffe7f076f0 (LWP 6598)]
>>>> geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>>>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>>>> 77              switch (g.getGeometryTypeId()) {
>>>> Current language:  auto; currently c++
>>>> (gdb) where
>>>> #0  geos::io::WKBWriter::write (this=0x7fffeff120c0, g=@0xd8005270,
>>>> os=@0x7fffeff11f50) at WKBWriter.cpp:77
>>>> #1  0x00007fffe6883549 in GEOSGeomToWKB_buf_r (extHandle=0xa5beb0,
>>>> g=0xd8005270, size=0x14407f0) at geos_ts_c.cpp:948
>>>> #2  0x00007fffe6aa53b8 in ffi_call_unix64 () at
>>>> /build/buildd/python2.6-2.6.2/Modules/_ctypes/libffi/src/x86/
>>>> unix64.S:75
>>>> Python stacktrace:
>>>>
>>>> [Python Stack]
>>>>
>>>> #46 0x0000000000417ae9 in _start () at ../sysdeps/x86_64/elf/start.S:
>>>> 113
>>>>
>>>> A full stacktrace is included as an attachment. I have augmented the
>>>> stacktrace with the python
>>>> stackframe where the stack frame is in python code.
>>>>
>>>> I was wondering why GEOSGeomToWKB_buf is called, but
>>>> GEOSGeomToWKB_buf_r appears on the stacktrace.
>>>>
>>>> I do use threads in the application, but  only the main thread has
>>>> calling points to shapely.
>>>> Also the application uses SQLAlchemy with the user defined Geometry
>>>> type from shapely. The gluecode is part of mapfish.
>>>>
>>>> Perhaps you have some ideas about why the segfault shows up. I'm not
>>>> familiar with the way
>>>> shapely handles GEOS c pointers. In gdb the geos geometry object
>>>> g=@0xd8005270 points to an adress
>>>>
>>>> trying to access the geometry object reveals
>>>>
>>>> (gdb) p g
>>>> $1 = (const geos::geom::Geometry &) @0xd8005270: <error reading
>>>> variable>
>>>>
>>>>
>>>> regards,
>>>> Paul
>>>
>>> Hi Paul,
>>>
>>> I'm sorry you're experiencing trouble with Shapely. If you can write a
>>> Python program that doesn't depend on MapFish or SQLAlchemy or PostGIS
>>> (I don't have that stack running right now) that reliably crashes,
>>> I'll get right on it. Maybe it's just a matter of writing out WKB from
>>> multiple threads at once.
>>>
>>> If you're using GEOS 3.1, the non-reentrant API is just a thin layer
>>> over the reentrant one, so EOSGeomToWKB_buf_r will always be called.
>>>
>>> Cheers,
>>>
>>> --
>>> Sean
>>>
>>> _______________________________________________
>>> Community mailing list
>>> Community@...
>>> http://lists.gispython.org/mailman/listinfo/community
>>>
>>
>>
>>
>> --
>> paulbalomiri@...
>>
>
>
>
> --
> paulbalomiri@...
>

--
paulbalomiri@...
_______________________________________________
Community mailing list
Community@...
http://lists.gispython.org/mailman/listinfo/community