semaphores between processes

[Andrew Gallatin]

Hi,

We're designing some software which has to lock access to
shared memory pages between several processes, and has to
run on Linux, Solaris, and FreeBSD.  We were planning to
have the lock be a pthread_mutex_t residing in the
shared memory page.  This works well on Linux and Solaris,
but FreeBSD (at least 7-stable) does not support
PTHREAD_PROCESS_SHARED mutexes.

We then moved on to posix semaphores.  Using sem_wait/sem_post
with the sem_t residing in a shared page seems to work on
all 3 platforms.  However, the FreeBSD (7-stable) man page
for sem_init(3) has this scary text regarding the pshared
value:

      The sem_init() function initializes the unnamed semaphore pointed
to by
      sem to have the value value.  A non-zero value for pshared specifies a
      shared semaphore that can be used by multiple processes, which this
      implementation is not capable of.

Is this text obsolete?  Or is my test just "getting lucky"?

Is there recommended way to do this?

Thanks,

Drew
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Thu, 22 Oct 2009, Andrew Gallatin wrote:

> Hi,
>
> We're designing some software which has to lock access to
> shared memory pages between several processes, and has to
> run on Linux, Solaris, and FreeBSD.  We were planning to
> have the lock be a pthread_mutex_t residing in the
> shared memory page.  This works well on Linux and Solaris,
> but FreeBSD (at least 7-stable) does not support
> PTHREAD_PROCESS_SHARED mutexes.
>
> We then moved on to posix semaphores.  Using sem_wait/sem_post
> with the sem_t residing in a shared page seems to work on
> all 3 platforms.  However, the FreeBSD (7-stable) man page
> for sem_init(3) has this scary text regarding the pshared
> value:
>
>     The sem_init() function initializes the unnamed semaphore pointed to by
>     sem to have the value value.  A non-zero value for pshared specifies a
>     shared semaphore that can be used by multiple processes, which this
>     implementation is not capable of.
>
> Is this text obsolete?  Or is my test just "getting lucky"?

I think you're getting lucky.

> Is there recommended way to do this?

I believe the only way to do this is with SYSV semaphores
(semop, semget, semctl).  Unfortunately, these are not as
easy to use, IMHO.

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Vlad GALU-2]

On Thu, Oct 22, 2009 at 11:08 PM, Andrew Gallatin <gallatin@...> wrote:

> Hi,
>
> We're designing some software which has to lock access to
> shared memory pages between several processes, and has to
> run on Linux, Solaris, and FreeBSD.  We were planning to
> have the lock be a pthread_mutex_t residing in the
> shared memory page.  This works well on Linux and Solaris,
> but FreeBSD (at least 7-stable) does not support
> PTHREAD_PROCESS_SHARED mutexes.
>
> We then moved on to posix semaphores.  Using sem_wait/sem_post
> with the sem_t residing in a shared page seems to work on
> all 3 platforms.  However, the FreeBSD (7-stable) man page
> for sem_init(3) has this scary text regarding the pshared
> value:
>
>     The sem_init() function initializes the unnamed semaphore pointed to by
>     sem to have the value value.  A non-zero value for pshared specifies a
>     shared semaphore that can be used by multiple processes, which this
>     implementation is not capable of.
>
> Is this text obsolete?  Or is my test just "getting lucky"?
>
> Is there recommended way to do this?
>
> Thanks,
>
> Drew

Hi Andrew,

This works in Linux is because Linux defines sem_t as a struct type(or
union, IIRC), while our sem_t is a pointer type (with more state kept
in the pointed struct). SYSV semaphores seems the way to go...
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Andrew Gallatin]

Daniel Eischen wrote:

> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>
>> Hi,
>>
>> We're designing some software which has to lock access to
>> shared memory pages between several processes, and has to
>> run on Linux, Solaris, and FreeBSD.  We were planning to
>> have the lock be a pthread_mutex_t residing in the
>> shared memory page.  This works well on Linux and Solaris,
>> but FreeBSD (at least 7-stable) does not support
>> PTHREAD_PROCESS_SHARED mutexes.
>>
>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>> with the sem_t residing in a shared page seems to work on
>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>> for sem_init(3) has this scary text regarding the pshared
>> value:
>>
>>     The sem_init() function initializes the unnamed semaphore pointed
>> to by
>>     sem to have the value value.  A non-zero value for pshared
>> specifies a
>>     shared semaphore that can be used by multiple processes, which this
>>     implementation is not capable of.
>>
>> Is this text obsolete?  Or is my test just "getting lucky"?
>
> I think you're getting lucky.

Yes, after playing with the code some, I now see that. :(

>> Is there recommended way to do this?
>
> I believe the only way to do this is with SYSV semaphores
> (semop, semget, semctl).  Unfortunately, these are not as
> easy to use, IMHO.

Yes, they are pretty ugly, and we were hoping to avoid them.
Are there any plans to support either PTHREAD_PROCESS_SHARED
mutexes, or pshared posix semaphores in FreeBSD?


Thanks,

Drew

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Thu, 22 Oct 2009, Andrew Gallatin wrote:

> Daniel Eischen wrote:
>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>
>>> Hi,
>>>
>>> We're designing some software which has to lock access to
>>> shared memory pages between several processes, and has to
>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>> have the lock be a pthread_mutex_t residing in the
>>> shared memory page.  This works well on Linux and Solaris,
>>> but FreeBSD (at least 7-stable) does not support
>>> PTHREAD_PROCESS_SHARED mutexes.
>>>
>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>> with the sem_t residing in a shared page seems to work on
>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>> for sem_init(3) has this scary text regarding the pshared
>>> value:
>>>
>>>     The sem_init() function initializes the unnamed semaphore pointed to
>>> by
>>>     sem to have the value value.  A non-zero value for pshared specifies a
>>>     shared semaphore that can be used by multiple processes, which this
>>>     implementation is not capable of.
>>>
>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>
>> I think you're getting lucky.
>
> Yes, after playing with the code some, I now see that. :(
>
>>> Is there recommended way to do this?
>>
>> I believe the only way to do this is with SYSV semaphores
>> (semop, semget, semctl).  Unfortunately, these are not as
>> easy to use, IMHO.
>
> Yes, they are pretty ugly, and we were hoping to avoid them.
> Are there any plans to support either PTHREAD_PROCESS_SHARED
> mutexes, or pshared posix semaphores in FreeBSD?

It's planned, just not (yet) being actively worked on.
It's a API change mostly, and then adding in all the
compat hooks so we don't break ABI.

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Jilles Tjoelker]

On Thu, Oct 22, 2009 at 04:08:11PM -0400, Andrew Gallatin wrote:
> We then moved on to posix semaphores.  Using sem_wait/sem_post
> with the sem_t residing in a shared page seems to work on
> all 3 platforms.  However, the FreeBSD (7-stable) man page
> for sem_init(3) has this scary text regarding the pshared
> value:

>       The sem_init() function initializes the unnamed semaphore pointed
> to by
>       sem to have the value value.  A non-zero value for pshared specifies a
>       shared semaphore that can be used by multiple processes, which this
>       implementation is not capable of.

> Is this text obsolete?  Or is my test just "getting lucky"?

They work, but only if the processes are related and do not exec and the
parent process initializes the semaphores before forking. This is
because sem_t is a pointer to a malloc'ed structure. For process-shared
semaphores this really only contains an identifier of the kernel
semaphore. This also means process-shared semaphores are slower than
in-process semaphores (libthr implements those using atomics and umtx so
that system calls are only needed if a thread needs to sleep or be
awakened).

This is documented in comments in the source code, but not in man pages
or other documentation.

> Is there recommended way to do this?

Apart from sysv semaphores, perhaps posix named semaphores (sem_open()
etc). These require a 'kldload sem' on older versions though.

--
Jilles Tjoelker
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[jhb-3]

On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:

> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>
> > Daniel Eischen wrote:
> >> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
> >>
> >>> Hi,
> >>>
> >>> We're designing some software which has to lock access to
> >>> shared memory pages between several processes, and has to
> >>> run on Linux, Solaris, and FreeBSD.  We were planning to
> >>> have the lock be a pthread_mutex_t residing in the
> >>> shared memory page.  This works well on Linux and Solaris,
> >>> but FreeBSD (at least 7-stable) does not support
> >>> PTHREAD_PROCESS_SHARED mutexes.
> >>>
> >>> We then moved on to posix semaphores.  Using sem_wait/sem_post
> >>> with the sem_t residing in a shared page seems to work on
> >>> all 3 platforms.  However, the FreeBSD (7-stable) man page
> >>> for sem_init(3) has this scary text regarding the pshared
> >>> value:
> >>>
> >>>     The sem_init() function initializes the unnamed semaphore pointed to
> >>> by
> >>>     sem to have the value value.  A non-zero value for pshared specifies

a

> >>>     shared semaphore that can be used by multiple processes, which this
> >>>     implementation is not capable of.
> >>>
> >>> Is this text obsolete?  Or is my test just "getting lucky"?
> >>
> >> I think you're getting lucky.
> >
> > Yes, after playing with the code some, I now see that. :(
> >
> >>> Is there recommended way to do this?
> >>
> >> I believe the only way to do this is with SYSV semaphores
> >> (semop, semget, semctl).  Unfortunately, these are not as
> >> easy to use, IMHO.
> >
> > Yes, they are pretty ugly, and we were hoping to avoid them.
> > Are there any plans to support either PTHREAD_PROCESS_SHARED
> > mutexes, or pshared posix semaphores in FreeBSD?
>
> It's planned, just not (yet) being actively worked on.
> It's a API change mostly, and then adding in all the
> compat hooks so we don't break ABI.

There are also an alternate set of patches on threads@ to allow just shared
semaphores I think w/o the changes to the pthread types.  I can't recall
exactly what they did, but I think rrs@ was playing with using umtx directly
to implement some sort of process-shared primitive.

--
John Baldwin
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Fri, 23 Oct 2009, John Baldwin wrote:

> On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>
>>> Daniel Eischen wrote:
>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We're designing some software which has to lock access to
>>>>> shared memory pages between several processes, and has to
>>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>>>> have the lock be a pthread_mutex_t residing in the
>>>>> shared memory page.  This works well on Linux and Solaris,
>>>>> but FreeBSD (at least 7-stable) does not support
>>>>> PTHREAD_PROCESS_SHARED mutexes.
>>>>>
>>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>>>> with the sem_t residing in a shared page seems to work on
>>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>>>> for sem_init(3) has this scary text regarding the pshared
>>>>> value:
>>>>>
>>>>>     The sem_init() function initializes the unnamed semaphore pointed to
>>>>> by
>>>>>     sem to have the value value.  A non-zero value for pshared specifies
> a
>>>>>     shared semaphore that can be used by multiple processes, which this
>>>>>     implementation is not capable of.
>>>>>
>>>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>>>
>>>> I think you're getting lucky.
>>>
>>> Yes, after playing with the code some, I now see that. :(
>>>
>>>>> Is there recommended way to do this?
>>>>
>>>> I believe the only way to do this is with SYSV semaphores
>>>> (semop, semget, semctl).  Unfortunately, these are not as
>>>> easy to use, IMHO.
>>>
>>> Yes, they are pretty ugly, and we were hoping to avoid them.
>>> Are there any plans to support either PTHREAD_PROCESS_SHARED
>>> mutexes, or pshared posix semaphores in FreeBSD?
>>
>> It's planned, just not (yet) being actively worked on.
>> It's a API change mostly, and then adding in all the
>> compat hooks so we don't break ABI.
>
> There are also an alternate set of patches on threads@ to allow just shared
> semaphores I think w/o the changes to the pthread types.  I can't recall
> exactly what they did, but I think rrs@ was playing with using umtx directly
> to implement some sort of process-shared primitive.

That's really not the way to go.  The structs really need
to become public.

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[jhb-3]

On Friday 23 October 2009 10:56:06 am Daniel Eischen wrote:

> On Fri, 23 Oct 2009, John Baldwin wrote:
>
> > On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
> >> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
> >>
> >>> Daniel Eischen wrote:
> >>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> We're designing some software which has to lock access to
> >>>>> shared memory pages between several processes, and has to
> >>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
> >>>>> have the lock be a pthread_mutex_t residing in the
> >>>>> shared memory page.  This works well on Linux and Solaris,
> >>>>> but FreeBSD (at least 7-stable) does not support
> >>>>> PTHREAD_PROCESS_SHARED mutexes.
> >>>>>
> >>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
> >>>>> with the sem_t residing in a shared page seems to work on
> >>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
> >>>>> for sem_init(3) has this scary text regarding the pshared
> >>>>> value:
> >>>>>
> >>>>>     The sem_init() function initializes the unnamed semaphore pointed to
> >>>>> by
> >>>>>     sem to have the value value.  A non-zero value for pshared specifies
> > a
> >>>>>     shared semaphore that can be used by multiple processes, which this
> >>>>>     implementation is not capable of.
> >>>>>
> >>>>> Is this text obsolete?  Or is my test just "getting lucky"?
> >>>>
> >>>> I think you're getting lucky.
> >>>
> >>> Yes, after playing with the code some, I now see that. :(
> >>>
> >>>>> Is there recommended way to do this?
> >>>>
> >>>> I believe the only way to do this is with SYSV semaphores
> >>>> (semop, semget, semctl).  Unfortunately, these are not as
> >>>> easy to use, IMHO.
> >>>
> >>> Yes, they are pretty ugly, and we were hoping to avoid them.
> >>> Are there any plans to support either PTHREAD_PROCESS_SHARED
> >>> mutexes, or pshared posix semaphores in FreeBSD?
> >>
> >> It's planned, just not (yet) being actively worked on.
> >> It's a API change mostly, and then adding in all the
> >> compat hooks so we don't break ABI.
> >
> > There are also an alternate set of patches on threads@ to allow just shared
> > semaphores I think w/o the changes to the pthread types.  I can't recall
> > exactly what they did, but I think rrs@ was playing with using umtx directly
> > to implement some sort of process-shared primitive.
>
> That's really not the way to go.  The structs really need
> to become public.

I was mostly suggesting it as a way to use something sooner since I expect it
will be a long while before anyone does the pthreads work.

--
John Baldwin
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Andrew Gallatin]

Daniel Eischen wrote:

> On Fri, 23 Oct 2009, John Baldwin wrote:
>
>> On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>
>>>> Daniel Eischen wrote:
>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We're designing some software which has to lock access to
>>>>>> shared memory pages between several processes, and has to
>>>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>>>>> have the lock be a pthread_mutex_t residing in the
>>>>>> shared memory page.  This works well on Linux and Solaris,
>>>>>> but FreeBSD (at least 7-stable) does not support
>>>>>> PTHREAD_PROCESS_SHARED mutexes.
>>>>>>
>>>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>>>>> with the sem_t residing in a shared page seems to work on
>>>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>>>>> for sem_init(3) has this scary text regarding the pshared
>>>>>> value:
>>>>>>
>>>>>>     The sem_init() function initializes the unnamed semaphore
>>>>>> pointed to
>>>>>> by
>>>>>>     sem to have the value value.  A non-zero value for pshared
>>>>>> specifies
>> a
>>>>>>     shared semaphore that can be used by multiple processes, which
>>>>>> this
>>>>>>     implementation is not capable of.
>>>>>>
>>>>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>>>>
>>>>> I think you're getting lucky.
>>>>
>>>> Yes, after playing with the code some, I now see that. :(
>>>>
>>>>>> Is there recommended way to do this?
>>>>>
>>>>> I believe the only way to do this is with SYSV semaphores
>>>>> (semop, semget, semctl).  Unfortunately, these are not as
>>>>> easy to use, IMHO.
>>>>
>>>> Yes, they are pretty ugly, and we were hoping to avoid them.
>>>> Are there any plans to support either PTHREAD_PROCESS_SHARED
>>>> mutexes, or pshared posix semaphores in FreeBSD?
>>>
>>> It's planned, just not (yet) being actively worked on.
>>> It's a API change mostly, and then adding in all the
>>> compat hooks so we don't break ABI.
>>
>> There are also an alternate set of patches on threads@ to allow just
>> shared
>> semaphores I think w/o the changes to the pthread types.  I can't recall
>> exactly what they did, but I think rrs@ was playing with using umtx
>> directly
>> to implement some sort of process-shared primitive.
>
> That's really not the way to go.  The structs really need
> to become public.
>

It would be great if they were, but that discussion was 6 months
ago, and nothing seems to have happened.  Plus we need to support
at least 7.X and probably 6, so any changes here might not even
help us.

What is wrong  with just using umtx directly?  It seems to do
exactly what we need.

Thanks,
Drew
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Fri, 23 Oct 2009, Andrew Gallatin wrote:

> Daniel Eischen wrote:
>> On Fri, 23 Oct 2009, John Baldwin wrote:
>>
>>> On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>
>>>>> Daniel Eischen wrote:
>>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We're designing some software which has to lock access to
>>>>>>> shared memory pages between several processes, and has to
>>>>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>>>>>> have the lock be a pthread_mutex_t residing in the
>>>>>>> shared memory page.  This works well on Linux and Solaris,
>>>>>>> but FreeBSD (at least 7-stable) does not support
>>>>>>> PTHREAD_PROCESS_SHARED mutexes.
>>>>>>>
>>>>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>>>>>> with the sem_t residing in a shared page seems to work on
>>>>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>>>>>> for sem_init(3) has this scary text regarding the pshared
>>>>>>> value:
>>>>>>>
>>>>>>>     The sem_init() function initializes the unnamed semaphore pointed
>>>>>>> to
>>>>>>> by
>>>>>>>     sem to have the value value.  A non-zero value for pshared
>>>>>>> specifies
>>> a
>>>>>>>     shared semaphore that can be used by multiple processes, which
>>>>>>> this
>>>>>>>     implementation is not capable of.
>>>>>>>
>>>>>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>>>>>
>>>>>> I think you're getting lucky.
>>>>>
>>>>> Yes, after playing with the code some, I now see that. :(
>>>>>
>>>>>>> Is there recommended way to do this?
>>>>>>
>>>>>> I believe the only way to do this is with SYSV semaphores
>>>>>> (semop, semget, semctl).  Unfortunately, these are not as
>>>>>> easy to use, IMHO.
>>>>>
>>>>> Yes, they are pretty ugly, and we were hoping to avoid them.
>>>>> Are there any plans to support either PTHREAD_PROCESS_SHARED
>>>>> mutexes, or pshared posix semaphores in FreeBSD?
>>>>
>>>> It's planned, just not (yet) being actively worked on.
>>>> It's a API change mostly, and then adding in all the
>>>> compat hooks so we don't break ABI.
>>>
>>> There are also an alternate set of patches on threads@ to allow just
>>> shared
>>> semaphores I think w/o the changes to the pthread types.  I can't recall
>>> exactly what they did, but I think rrs@ was playing with using umtx
>>> directly
>>> to implement some sort of process-shared primitive.
>>
>> That's really not the way to go.  The structs really need
>> to become public.
>>
>
> It would be great if they were, but that discussion was 6 months
> ago, and nothing seems to have happened.  Plus we need to support
> at least 7.X and probably 6, so any changes here might not even
> help us.
>
> What is wrong  with just using umtx directly?  It seems to do
> exactly what we need.

Because you can't do anything more than use umtx directly,
like check for mutex types and return appropriate error
codes.  Just look at other implementations - Solaris,
Linux, all have their pthread_*_t as public structs.

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Andrew Gallatin]

Daniel Eischen wrote:

> On Fri, 23 Oct 2009, Andrew Gallatin wrote:
>
>> Daniel Eischen wrote:
>>> On Fri, 23 Oct 2009, John Baldwin wrote:
>>>
>>>> On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>
>>>>>> Daniel Eischen wrote:
>>>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We're designing some software which has to lock access to
>>>>>>>> shared memory pages between several processes, and has to
>>>>>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>>>>>>> have the lock be a pthread_mutex_t residing in the
>>>>>>>> shared memory page.  This works well on Linux and Solaris,
>>>>>>>> but FreeBSD (at least 7-stable) does not support
>>>>>>>> PTHREAD_PROCESS_SHARED mutexes.
>>>>>>>>
>>>>>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>>>>>>> with the sem_t residing in a shared page seems to work on
>>>>>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>>>>>>> for sem_init(3) has this scary text regarding the pshared
>>>>>>>> value:
>>>>>>>>
>>>>>>>>     The sem_init() function initializes the unnamed semaphore
>>>>>>>> pointed to
>>>>>>>> by
>>>>>>>>     sem to have the value value.  A non-zero value for pshared
>>>>>>>> specifies
>>>> a
>>>>>>>>     shared semaphore that can be used by multiple processes,
>>>>>>>> which this
>>>>>>>>     implementation is not capable of.
>>>>>>>>
>>>>>>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>>>>>>
>>>>>>> I think you're getting lucky.
>>>>>>
>>>>>> Yes, after playing with the code some, I now see that. :(
>>>>>>
>>>>>>>> Is there recommended way to do this?
>>>>>>>
>>>>>>> I believe the only way to do this is with SYSV semaphores
>>>>>>> (semop, semget, semctl).  Unfortunately, these are not as
>>>>>>> easy to use, IMHO.
>>>>>>
>>>>>> Yes, they are pretty ugly, and we were hoping to avoid them.
>>>>>> Are there any plans to support either PTHREAD_PROCESS_SHARED
>>>>>> mutexes, or pshared posix semaphores in FreeBSD?
>>>>>
>>>>> It's planned, just not (yet) being actively worked on.
>>>>> It's a API change mostly, and then adding in all the
>>>>> compat hooks so we don't break ABI.
>>>>
>>>> There are also an alternate set of patches on threads@ to allow just
>>>> shared
>>>> semaphores I think w/o the changes to the pthread types.  I can't
>>>> recall
>>>> exactly what they did, but I think rrs@ was playing with using umtx
>>>> directly
>>>> to implement some sort of process-shared primitive.
>>>
>>> That's really not the way to go.  The structs really need
>>> to become public.
>>>
>>
>> It would be great if they were, but that discussion was 6 months
>> ago, and nothing seems to have happened.  Plus we need to support
>> at least 7.X and probably 6, so any changes here might not even
>> help us.
>>
>> What is wrong  with just using umtx directly?  It seems to do
>> exactly what we need.
>
> Because you can't do anything more than use umtx directly,
> like check for mutex types and return appropriate error
> codes.  Just look at other implementations - Solaris,
> Linux, all have their pthread_*_t as public structs.

I'm not saying that having pthread*t public, and getting all
the features of real PTHREAD_PROCESS_SHARED would not be far
better in general.  But in this case all we need is a lock around
a shared resource.  Eg, nothing fance.  So our choices seem to be
either:

1) use sysv semaphores (ick)
2) use a hand rolled spinlock (ick)
3) use some sort of hack built into our driver (ick, ick)
4) use umtx

Is there some bug or limitation in umtx that makes it inappropriate?
(beyond the obvious, like the potential to leave a resource locked
forever if the lock holder exits).

Thanks,

Drew
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Fri, 23 Oct 2009, Andrew Gallatin wrote:

> Daniel Eischen wrote:
>> On Fri, 23 Oct 2009, Andrew Gallatin wrote:
>>
>>> It would be great if they were, but that discussion was 6 months
>>> ago, and nothing seems to have happened.  Plus we need to support
>>> at least 7.X and probably 6, so any changes here might not even
>>> help us.
>>>
>>> What is wrong  with just using umtx directly?  It seems to do
>>> exactly what we need.
>>
>> Because you can't do anything more than use umtx directly,
>> like check for mutex types and return appropriate error
>> codes.  Just look at other implementations - Solaris,
>> Linux, all have their pthread_*_t as public structs.
>
> I'm not saying that having pthread*t public, and getting all
> the features of real PTHREAD_PROCESS_SHARED would not be far
> better in general.  But in this case all we need is a lock around
> a shared resource.  Eg, nothing fance.  So our choices seem to be
> either:
>
> 1) use sysv semaphores (ick)
> 2) use a hand rolled spinlock (ick)
> 3) use some sort of hack built into our driver (ick, ick)
> 4) use umtx
>
> Is there some bug or limitation in umtx that makes it inappropriate?
> (beyond the obvious, like the potential to leave a resource locked
> forever if the lock holder exits).

We already use umtx.  This really is a hack and I wouldn't
advocate it.  I'm not sure how you could make it work and
not break existing ability to return appropriate error
codes without slowing down the path in the non-shared
case.  You'd have to check to see if the address space
was shared or not, which would require a system call.

All our public pthread_foo() symbols are weak.  You
can easily override them in your application code in
the #ifdef freebsd case.  What is wrong with providing
your own library that overrides them to do what you
require - this shouldn't change your application code?

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Andrew Gallatin]

Daniel Eischen wrote:

>
> We already use umtx.  This really is a hack and I wouldn't
> advocate it.  I'm not sure how you could make it work and
> not break existing ability to return appropriate error
> codes without slowing down the path in the non-shared
> case.  You'd have to check to see if the address space
> was shared or not, which would require a system call.

I'm probably missing something.  What does it matter if the
address space is shared, as long as the umtx struct is
in shared memory?

 From my quick read, the umtx operations use a lock word
in userspace. For uncontested locks, they use atomic
ops to flip an id into the lock word.  The kernel takes
over for contested locks, and does sleeping, wakup, etc.
Is this correct?  Is there something here that matters
if the address space (and not just the lock word) is
shared?

> All our public pthread_foo() symbols are weak.  You
> can easily override them in your application code in
> the #ifdef freebsd case.  What is wrong with providing
> your own library that overrides them to do what you
> require - this shouldn't change your application code?
>

For our code, I was thinking of something like:

#ifdef FreeBSD
#define lock(x) umtx_lock(x, getpid())
#define unlock(x) umtx_unlock(x, getpid())
#else
#define lock(x) pthread_mutex_lock(x)
#define unlock(x) pthread_mutex_lock(x)
#endif


I should probably just shut up and try it..

Drew

_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."

[Daniel Eischen]

On Fri, 23 Oct 2009, Andrew Gallatin wrote:

> Daniel Eischen wrote:
>
>>
>> We already use umtx.  This really is a hack and I wouldn't
>> advocate it.  I'm not sure how you could make it work and
>> not break existing ability to return appropriate error
>> codes without slowing down the path in the non-shared
>> case.  You'd have to check to see if the address space
>> was shared or not, which would require a system call.
>
> I'm probably missing something.  What does it matter if the
> address space is shared, as long as the umtx struct is
> in shared memory?
>
> From my quick read, the umtx operations use a lock word
> in userspace. For uncontested locks, they use atomic
> ops to flip an id into the lock word.  The kernel takes
> over for contested locks, and does sleeping, wakup, etc.
> Is this correct?  Is there something here that matters
> if the address space (and not just the lock word) is
> shared?
>
>> All our public pthread_foo() symbols are weak.  You
>> can easily override them in your application code in
>> the #ifdef freebsd case.  What is wrong with providing
>> your own library that overrides them to do what you
>> require - this shouldn't change your application code?
>>
>
> For our code, I was thinking of something like:
>
> #ifdef FreeBSD
> #define lock(x) umtx_lock(x, getpid())
> #define unlock(x) umtx_unlock(x, getpid())
> #else
> #define lock(x) pthread_mutex_lock(x)
> #define unlock(x) pthread_mutex_lock(x)
> #endif
>
>
> I should probably just shut up and try it..

My apologies - I thought you were talking about changing
our pthread_mutex_* functions in src/lib/...

--
DE
_______________________________________________
freebsd-hackers@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@..."