Deadlock in NSLog

[David Chisnall]

Hi,

I've recently encountered a slightly strange problem with NSLog.  It  
seems to be doing a lot of initialisation stuff the first time it's  
called.  When NSLog is called first, simultaneously, from two  
threads, one of them gets deadlocked in GSPrivateDefaultLocale(), the  
other returns happily.

It appears that GNUstep is using the ObjC runtime mutex, which tries  
to emulate a recursive mutex using a non-recursive mutex.  It looked  
like there was a potential for deadlock in here when I looked at the  
code a few months ago.  Since GNUstep depends on pthreads anyway, it  
might be better to use the pthread functions directly, rather than  
going through a buggy abstraction layer.

David


_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Ayers-2]

Am Freitag, den 01.08.2008, 12:06 +0100 schrieb David Chisnall:
> It appears that GNUstep is using the ObjC runtime mutex, which tries  
> to emulate a recursive mutex using a non-recursive mutex.  It looked  
> like there was a potential for deadlock in here when I looked at the  
> code a few months ago.  Since GNUstep depends on pthreads anyway, it  
> might be better to use the pthread functions directly, rather than  
> going through a buggy abstraction layer.

I don't believe that bypassing the objc abstraction layer is a good
idea.

GNUstep and GNU ObjC have been ported to platforms that may not be
supported be pthreads.  In particular  I remember that FreeBSD at on
point used a different threading library that claimed POSIX/pthread
compatibility.

I would instead try to create a libobjc test case a report a bug against
libobjc to get it fixed there.  Now with all this ObjC 2.0 activity I
would believe that someone would have get GCC libobjc up to par anyway
on these platforms to be able test it there anyway.

Cheers,
David




_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Chisnall]

On 4 Aug 2008, at 10:12, David Ayers wrote:

> Am Freitag, den 01.08.2008, 12:06 +0100 schrieb David Chisnall:
>> It appears that GNUstep is using the ObjC runtime mutex, which tries
>> to emulate a recursive mutex using a non-recursive mutex.  It looked
>> like there was a potential for deadlock in here when I looked at the
>> code a few months ago.  Since GNUstep depends on pthreads anyway, it
>> might be better to use the pthread functions directly, rather than
>> going through a buggy abstraction layer.
>
> I don't believe that bypassing the objc abstraction layer is a good
> idea.
>
> GNUstep and GNU ObjC have been ported to platforms that may not be
> supported be pthreads.  In particular  I remember that FreeBSD at on
> point used a different threading library that claimed POSIX/pthread
> compatibility.
>
> I would instead try to create a libobjc test case a report a bug  
> against
> libobjc to get it fixed there.  Now with all this ObjC 2.0 activity I
> would believe that someone would have get GCC libobjc up to par anyway
> on these platforms to be able test it there anyway.
>
> Cheers,
> David

On FreeBSD, libobjc has always used POSIX threads.  The abstraction  
layer includes Mach, Irix, and Solaris back ends.  All of these  
platforms now have a POSIX-compliant threading library (and have got  
about a decade).  There are no platforms on which GNUstep runs that  
do no either support POSIX threads or get POSIX threads through the  
same POSIX-compatibility framework needed for the rest of GNUstep.

This is one of the reasons why the new runtime uses pthreads  
directly.  The abstraction layer in the GNU runtime is almost as big  
as the entire codebase for the new runtime.

David

P.S. Andrew is very busy on other things at the moment, and I believe  
I am the only person currently actively hacking on GNU libobjc.  You  
can find the experimental version in the Étoilé repository.

_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[Nicola Pero-4]

>>
> On FreeBSD, libobjc has always used POSIX threads.  The abstraction  
> layer includes Mach, Irix, and Solaris back ends.  All of these  
> platforms now have a POSIX-compliant threading library (and have got  
> about a decade).  There are no platforms on which GNUstep runs that  
> do no either support POSIX threads or get POSIX threads through the  
> same POSIX-compatibility framework needed for the rest of GNUstep.
>
> This is one of the reasons why the new runtime uses pthreads  
> directly.  The abstraction layer in the GNU runtime is almost as big  
> as the entire codebase for the new runtime.

Possibly ... but the threading layer is all implemented and very  
stable and actively used - and provides nice flexibility in porting
the Objective-C runtime to different machines.  Any special reason for  
removing this flexibility and future-proofing - other than
that you might have found a bug in the code ?

Btw, when you compile the GNU Objective-C runtime as part of GCC, it  
automatically uses GCC's own internal abstraction layer for threads.
In that case, the abstraction layer is a single trivial file that just  
remaps all functions to GCC's own internal abstraction layer ones. ;-)

If you compile the GNU Objective-C runtime standalone, yes you do get  
a bunch of different threading implementations; I think the two
that are more likely to be in use are the posix one and the win32  
one.  If we removed the abstraction layer, how would we deal with
the win32 one ?  It's nice to be able to use native Windows threads on  
Windows.  Were you thinking of using some posix-layer on top
of the win32 threads ?

Anyway, my favourite reason for having the abstraction layer is the  
ability to disable threads on a platform without changing any
of the frameworks.  If you configure the abstraction layer to use thr-
single, you can then compile and use everything on top of it
in single-thread mode with no threading library installed.  It also  
sounds quite useful if someone is porting Objective-C stuff to
some future architecture or machine where threads are not available,  
or there is no threading library - or they just want to get
stuff working single-threaded first and deal with thread complications  
later. :-)

Anyway given that the abstraction layer is all done for the most  
useful thread implementations (single, posix and win32) I'm not sure why
we'd want to remove it. ;-)

Thanks


_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Ayers-2]

Am Montag, den 04.08.2008, 10:58 +0100 schrieb David Chisnall:

> On 4 Aug 2008, at 10:12, David Ayers wrote:
>
> > Am Freitag, den 01.08.2008, 12:06 +0100 schrieb David Chisnall:
> >> It appears that GNUstep is using the ObjC runtime mutex, which tries
> >> to emulate a recursive mutex using a non-recursive mutex.  It looked
> >> like there was a potential for deadlock in here when I looked at the
> >> code a few months ago.  Since GNUstep depends on pthreads anyway, it
> >> might be better to use the pthread functions directly, rather than
> >> going through a buggy abstraction layer.
> >
> > I don't believe that bypassing the objc abstraction layer is a good
> > idea.
> >
> > GNUstep and GNU ObjC have been ported to platforms that may not be
> > supported be pthreads.  In particular  I remember that FreeBSD at on
> > point used a different threading library that claimed POSIX/pthread
> > compatibility.
> >
> > I would instead try to create a libobjc test case a report a bug  
> > against
> > libobjc to get it fixed there.  Now with all this ObjC 2.0 activity I
> > would believe that someone would have get GCC libobjc up to par anyway
> > on these platforms to be able test it there anyway.
> >
> On FreeBSD, libobjc has always used POSIX threads.  The abstraction  
> layer includes Mach, Irix, and Solaris back ends.  All of these  
> platforms now have a POSIX-compliant threading library (and have got  
> about a decade).  There are no platforms on which GNUstep runs that  
> do no either support POSIX threads or get POSIX threads through the  
> same POSIX-compatibility framework needed for the rest of GNUstep.

Just to make my position clear.  I personally have no issue if libobjc
required a working POSIX threads implementation and the legacy threading
support is removed from libobjc [in fact it may already have been
deactivated].  But I do believe this should be fixed in libobjc and not
worked around in -base.

Wether specific platform supports POSIX threads or not is irrelevent
(and I seem to be misremebering the FreeBSD case, maybe it was NetBSD
and pth?).  What is relevent is which specific threading library libobjc
was configured to use when it was built.  This decission is not
something that GNUstep code can influence (well save if you have
multiple libobjc's installed each configured differently).

What you are proposing is to simply assume that libobjc was configured
with a library called libptheads and have -base link against it directly
[something which have assumed in the past and possibly still assume in
some cases since I haven't checked recently].  Yet this is error prone
in the sense that it will work in most cases and fail subtly on some
platforms by potentially linking two threading libraries.

I also have no issue if there where some configure option as a stop-gap
until the fixed libobjc is widely distributed.  But never the less... it
should be fixed in "mainline" libobjc first.

Cheers,
David





_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Chisnall]

On 4 Aug 2008, at 12:46, David Ayers wrote:

> Just to make my position clear.  I personally have no issue if libobjc
> required a working POSIX threads implementation and the legacy  
> threading
> support is removed from libobjc [in fact it may already have been
> deactivated].  But I do believe this should be fixed in libobjc and  
> not
> worked around in -base.
>
> Wether specific platform supports POSIX threads or not is irrelevent
> (and I seem to be misremebering the FreeBSD case, maybe it was NetBSD
> and pth?).  What is relevent is which specific threading library  
> libobjc
> was configured to use when it was built.  This decission is not
> something that GNUstep code can influence (well save if you have
> multiple libobjc's installed each configured differently).
>
> What you are proposing is to simply assume that libobjc was configured
> with a library called libptheads and have -base link against it  
> directly
> [something which have assumed in the past and possibly still assume in
> some cases since I haven't checked recently].  Yet this is error prone
> in the sense that it will work in most cases and fail subtly on some
> platforms by potentially linking two threading libraries.
>
> I also have no issue if there where some configure option as a stop-
> gap
> until the fixed libobjc is widely distributed.  But never the  
> less... it
> should be fixed in "mainline" libobjc first.

I don't care whether libobjc uses its own threading implementation or  
not, however there is no reason for GNUstep to be using an  
inefficient and potentially (in this case, definitely) buggy wrapper  
around pthreads, rather than using pthreads directly.  The threading  
abstraction in libobjc implements the minimal functionality required  
for libobjc, not the minimal functionality required in general, or  
required for GNUstep.

NSRecursiveLock is implemented on top of objc_mutex, which emulates a  
recursive mutex on top of a non-recursive pthread (or other platform-
specific) mutex.  Quite how this makes more sense than using a  
recursive pthread mutex is beyond me.  On platforms without native  
pthreads support, there are pthread-compatibility libraries that are  
a lot better tested for general-purpose use than the libobjc code.

libobjc has to support more platforms than GNUstep.  For example, it  
supports VxWorks and Windows directly.  In order to use GNUstep on  
these platforms, you need a POSIX API layer, such as cygwin or mingw,  
which implements its own pthread wrapper.  If you do this, then some  
code will be using the cygwin / mingw / whatever wrapper code around  
the native APIs, and some will be using the libobjc wrapper around  
the native APIs.

Since GNUstep depends on POSIX for a lot of -base, I see no reason  
why it can't use POSIX functions.

David


_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Ayers-2]

Am Montag, den 04.08.2008, 12:56 +0100 schrieb David Chisnall:

> I don't care whether libobjc uses its own threading implementation or  
> not, however there is no reason for GNUstep to be using an  
> inefficient and potentially (in this case, definitely) buggy wrapper  
> around pthreads, rather than using pthreads directly.  The threading  
> abstraction in libobjc implements the minimal functionality required  
> for libobjc, not the minimal functionality required in general, or  
> required for GNUstep.

That assumes that ObjC code using GNUstep also does not use the libobjc
API directly for certain features that OpenStep/Cocoa did/does not
export.  I don't believe that is a safe assumption.  In fact I would be
very surprised if code that uses threading in meaingfull ways does not
rely on some of those features.

> NSRecursiveLock is implemented on top of objc_mutex, which emulates a  
> recursive mutex on top of a non-recursive pthread (or other platform-
> specific) mutex.  

If this wrapper is broken, then please file a bug (or even fix it since
this seems to be the pthread implementation which you are refering
to).  

> Quite how this makes more sense than using a  
> recursive pthread mutex is beyond me.  

Because libobjc wraps the threading API for a reason.  I don't claim to
know all the reasons.  I'm weary of ignoring them since debugging those
issues is painful.  So if libpthread (note I'm not stating POSIX... but
a specific implementation that -base would link to).

> On platforms without native  
> pthreads support, there are pthread-compatibility libraries that are  
> a lot better tested for general-purpose use than the libobjc code.

I cannot asses that evaluation, but I do clearly see a benifit in fixing
libobjc rather than working around this in -base.  If those
pthread-compatibility libraries are so much better, then libobjc should
be using them.  I have no issue with that.  In fact I think it would be
great if libobjc could be simplified in this fasion.

> libobjc has to support more platforms than GNUstep.  For example, it  
> supports VxWorks and Windows directly.  In order to use GNUstep on  
> these platforms, you need a POSIX API layer, such as cygwin or mingw,  
> which implements its own pthread wrapper.  If you do this, then some  
> code will be using the cygwin / mingw / whatever wrapper code around  
> the native APIs, and some will be using the libobjc wrapper around  
> the native APIs.
>
> Since GNUstep depends on POSIX for a lot of -base, I see no reason  
> why it can't use POSIX functions.

Well, there is a lot of code in -base that you would have adapt to
remove the dependency of libobjc's threading implementation.  In fact
I'm not even sure if the NSWillBecomeMutlithread hook can reliably be
called via the libobjc runtime if -base was configured with a different
threading library than libobjc.  (Of course it would happen to work if
the threading libraries happend to be identical).

But maybe you can explain why you do not seem to see an merit in fixing
the libobjc wrapper.

Cheers,
David




_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Chisnall]

On 4 Aug 2008, at 13:29, David Ayers wrote:

> Am Montag, den 04.08.2008, 12:56 +0100 schrieb David Chisnall:
>
>> I don't care whether libobjc uses its own threading implementation or
>> not, however there is no reason for GNUstep to be using an
>> inefficient and potentially (in this case, definitely) buggy wrapper
>> around pthreads, rather than using pthreads directly.  The threading
>> abstraction in libobjc implements the minimal functionality required
>> for libobjc, not the minimal functionality required in general, or
>> required for GNUstep.
>
> That assumes that ObjC code using GNUstep also does not use the  
> libobjc
> API directly for certain features that OpenStep/Cocoa did/does not
> export.  I don't believe that is a safe assumption.  In fact I  
> would be
> very surprised if code that uses threading in meaingfull ways does not
> rely on some of those features.

I've come across a lot of code in Objective-C applications that uses  
POSIX thread calls directly (this is even what Apple's Cocoa docs  
recommend if NSThread and friends are not adequate for your  
purposes), but none that calls GNU runtime threading functions other  
than GNUstep.  Note that these are GNU runtime specific - the NeXT,  
Apple, and Étoilé runtimes all use POSIX threads directly.  I am not  
sure what happens with NSLock if you try to compile it with the NeXT/
Apple runtime.  Possibly it just doesn't work.  It will need  
rewriting before it can support the Étoilé runtime (and, thus, much  
of Objective-C 2) anyway.

>> NSRecursiveLock is implemented on top of objc_mutex, which emulates a
>> recursive mutex on top of a non-recursive pthread (or other platform-
>> specific) mutex.
>
> If this wrapper is broken, then please file a bug (or even fix it  
> since
> this seems to be the pthread implementation which you are refering
> to).

The wrapper works in the specific usage patterns for libobjc.  It is  
not a general wrapper.

>> Quite how this makes more sense than using a
>> recursive pthread mutex is beyond me.
>
> Because libobjc wraps the threading API for a reason.  I don't  
> claim to
> know all the reasons.  I'm weary of ignoring them since debugging  
> those
> issues is painful.  So if libpthread (note I'm not stating POSIX...  
> but
> a specific implementation that -base would link to).

It wraps threading APIs because it is old, and when it was written  
there was a Solaris threading API, an HPUX threading API, and IRIX  
threading API, but no standard threading API.

It continues to wrap it because it, like GCC, supports non-POSIX  
platforms such as Windows, OS/2, and VxWorks where POSIX is not a  
native API.

>> On platforms without native
>> pthreads support, there are pthread-compatibility libraries that are
>> a lot better tested for general-purpose use than the libobjc code.
>
> I cannot asses that evaluation, but I do clearly see a benifit in  
> fixing
> libobjc rather than working around this in -base.  If those
> pthread-compatibility libraries are so much better, then libobjc  
> should
> be using them.  I have no issue with that.  In fact I think it  
> would be
> great if libobjc could be simplified in this fasion.

For libobjc to use them would introduce a dependency on POSIX into  
libobjc.  You can use libobjc and GCC to compile Objective-C code  
that does not use GNUstep and that uses the platform APIs directly.  
Adding a dependency on POSIX would be counter to the goals of GNU  
libobjc.

>> libobjc has to support more platforms than GNUstep.  For example, it
>> supports VxWorks and Windows directly.  In order to use GNUstep on
>> these platforms, you need a POSIX API layer, such as cygwin or mingw,
>> which implements its own pthread wrapper.  If you do this, then some
>> code will be using the cygwin / mingw / whatever wrapper code around
>> the native APIs, and some will be using the libobjc wrapper around
>> the native APIs.
>>
>> Since GNUstep depends on POSIX for a lot of -base, I see no reason
>> why it can't use POSIX functions.
>
> Well, there is a lot of code in -base that you would have adapt to
> remove the dependency of libobjc's threading implementation.  In fact
> I'm not even sure if the NSWillBecomeMutlithread hook can reliably be
> called via the libobjc runtime if -base was configured with a  
> different
> threading library than libobjc.  (Of course it would happen to work if
> the threading libraries happend to be identical).

NSWillBecomeMultithreaded should not be depended upon in any case.  
Recent versions of Cocoa always run in multithreaded mode due to  
difficulties in safely handling the notification and of  
interoperating with code that uses pthreads directly (which Apple  
have been encouraging for a while, although less so with the NSThread  
extensions in 10.5).

On OS X, NSWillBecomeMultithreaded is delivered if, and only if, new  
threads are created with NSThread.  It is the responsibility of  
NSThread to send this notification, and has nothing at all to do with  
whether the thread APIs are called via a wrapper or not (I believe  
early versions of OS X used Mach locks rather than POSIX ones (which  
were slightly slower) but this doesn't matter unless you expect to be  
able to lock a mutex with pthread_mutex_lock and unlock it with -
[NSLock unlock], which neither Cocoa nor GNUstep support).

> But maybe you can explain why you do not seem to see an merit in  
> fixing
> the libobjc wrapper.

Because it is not broken as used by libobjc.  It was never intended  
as a general-purpose threading abstraction layer, just as one that  
allowed the specific use cases required by libobjc.  Fixing it in the  
general case would require a lot of work which has already been done  
operating system / libc maintainers.

David

_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev

[David Ayers-2]

Am Montag, den 04.08.2008, 14:00 +0100 schrieb David Chisnall:
> On 4 Aug 2008, at 13:29, David Ayers wrote:
>

> I've come across a lot of code in Objective-C applications that uses  
> POSIX thread calls directly (this is even what Apple's Cocoa docs  
> recommend if NSThread and friends are not adequate for your  
> purposes)

You are aware that Apple can do this since it only supports libpthread
for it's runtime.

> , but none that calls GNU runtime threading functions other  
> than GNUstep.  Note that these are GNU runtime specific - the NeXT,  
> Apple, and Étoilé runtimes all use POSIX threads directly.  I am not  
> sure what happens with NSLock if you try to compile it with the NeXT/
> Apple runtime.  Possibly it just doesn't work.  It will need  
> rewriting before it can support the Étoilé runtime (and, thus, much  
> of Objective-C 2) anyway.

AFAIK the apple-gnu-* combos are not supported.

> >> NSRecursiveLock is implemented on top of objc_mutex, which emulates a
> >> recursive mutex on top of a non-recursive pthread (or other platform-
> >> specific) mutex.
> >
> > If this wrapper is broken, then please file a bug (or even fix it  
> > since
> > this seems to be the pthread implementation which you are refering
> > to).
>
> The wrapper works in the specific usage patterns for libobjc.  It is  
> not a general wrapper.

OK... so this is the real topic...

Is the threading API of libobjc a publicly exported user interface or an
internal libobjc API?

> It wraps threading APIs because it is old, and when it was written  
> there was a Solaris threading API, an HPUX threading API, and IRIX  
> threading API, but no standard threading API.
>
> It continues to wrap it because it, like GCC, supports non-POSIX  
> platforms such as Windows, OS/2, and VxWorks where POSIX is not a  
> native API.

[I'll refrain from replying to this for now as the answer to the above
question is what will define everything that follows.]

> >> On platforms without native
> >> pthreads support, there are pthread-compatibility libraries that are
> >> a lot better tested for general-purpose use than the libobjc code.
> >
> > I cannot asses that evaluation, but I do clearly see a benifit in  
> > fixing
> > libobjc rather than working around this in -base.  If those
> > pthread-compatibility libraries are so much better, then libobjc  
> > should
> > be using them.  I have no issue with that.  In fact I think it  
> > would be
> > great if libobjc could be simplified in this fasion.
>
> For libobjc to use them would introduce a dependency on POSIX into  
> libobjc.  You can use libobjc and GCC to compile Objective-C code  
> that does not use GNUstep and that uses the platform APIs directly.  
> Adding a dependency on POSIX would be counter to the goals of GNU  
> libobjc.

I don't understand how this would be adding a new dependency... libobjc
already depends on libpthread /if/ it was configured to use libpthread.
Since all you care about is libpthread, all you need to fix it for is
libpthread and have all others use the historic code.  Of course a FIXME
would be nice.

> > Well, there is a lot of code in -base that you would have adapt to
> > remove the dependency of libobjc's threading implementation.  In fact
> > I'm not even sure if the NSWillBecomeMutlithread hook can reliably be
> > called via the libobjc runtime if -base was configured with a  
> > different
> > threading library than libobjc.  (Of course it would happen to work if
> > the threading libraries happend to be identical).
>
> NSWillBecomeMultithreaded should not be depended upon in any case.  

Well that's part of the point of discussion mentioned above.  Currently
we rely on:

 objc_set_thread_callback
 objc_thread_set_data
 objc_thread_get_data

Would you consider this part of the public ObjC API?  Do you see them
usefull for something other than a public API?  (I'm not trying to sound
sarcastic... I really wonder why you don't believe they were intended
for something internal.)

> Recent versions of Cocoa always run in multithreaded mode due to  
> difficulties in safely handling the notification and of  
> interoperating with code that uses pthreads directly (which Apple  
> have been encouraging for a while, although less so with the NSThread  
> extensions in 10.5).

I'm not sure how this is relevant.

> On OS X, NSWillBecomeMultithreaded is delivered if, and only if, new  
> threads are created with NSThread.  It is the responsibility of  
> NSThread to send this notification, and has nothing at all to do with  
> whether the thread APIs are called via a wrapper or not (I believe  
> early versions of OS X used Mach locks rather than POSIX ones (which  
> were slightly slower) but this doesn't matter unless you expect to be  
> able to lock a mutex with pthread_mutex_lock and unlock it with -
> [NSLock unlock], which neither Cocoa nor GNUstep support).

It is true that NSWillBecomeMultithreaded is only intended for the
OpenStep/Cocoa API.  But what I was trying to say is that if the runtime
has been linked to a different threading library then -base that the
reliability of that notification is questionable.

> > But maybe you can explain why you do not seem to see an merit in  
> > fixing
> > the libobjc wrapper.
>
> Because it is not broken as used by libobjc.  It was never intended  
> as a general-purpose threading abstraction layer, just as one that  
> allowed the specific use cases required by libobjc.  Fixing it in the  
> general case would require a lot of work which has already been done  
> operating system / libc maintainers.

OK... so let's center around "It was never intended as a general-purpose
threading abstracont layers" as this is where we disagree.  I believe
the threading API is exactly that:

A general-purpose threading abstraction layer for libobjc

If I am wrong and that is not the case, then I totally agree with you,
that GNUstep has no business whatsoever using any of the API.

Cheers,
David




_______________________________________________
Gnustep-dev mailing list
Gnustep-dev@...
http://lists.gnu.org/mailman/listinfo/gnustep-dev