Re: [firebird-support] gds_thread_enter contention

[Bud Millwood]

On Tuesday 06 May 2008, Dmitry Yemanov wrote:

<originally posted on the support forums>

> Bud Millwood wrote:
> > - Does the fbclient library acquire a process-wide lock when a thread
> > passes through it (even when connecting to fb-classic)? If so, doesn't
> > that counteract the gains we get by running multiple fb-classic
> > instances?
>
> Yes, fbclient is synchronized using some kind of global mutex. It is
> acquired when any thread enters the API call and released as soon as the
> network packet is ready to be sent to the remote server. So, the
> blocking time is very short (compared to the network transport overhead
> and the server side processing of the call) and I don't think it really
> limits your performance.

Hi Dmitry:

I grabbed a developer here this morning and went back over our logic and our
numbers, and we are both now reasonably sure that this lock *is* the source
of the high contention. And from what we can tell, this lock is causing
Firebird to fail to scale to its full potential.

We haven't yet looked at the source, but my guess is that it's locking a
container in order to find the thread's TCP socket - is that right? Is is it
doing more than that?

We may start digging around in the fbclient source to try to figure out a way
to make this lock-free. One idea, which could be difficult depending on the
FB API, would be to embed the socket descriptor in the connection handle
that's held by the application thread. If that were possible, and the
fbclient library is doing what I think it's doing, then that would result in
lock-free access to the database.

- Bud

Bud Millwood
Weird Solutions, Inc.
http://www.weird-solutions.com
tel: [phone]
fax: [phone]

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Bud Millwood]

On this same note, we've been talking about the two architectures here, and
the direction of FB 3:

It's my understanding that with FB-Classic, when you make a new connection to
the server (either from a new process or a new thread), you get a new TCP
socket, which effectively gives you scalable IPC bandwidth. Instead of trying
to ram all connections through one socket, you spread them out over multiple
sockets, and this might be a big advantage on a multi-core system.

I read somewhere that the FB 3 architecture is leaning towards SuperServer. I
don't know if SS multiplexes all connections over a single socket today, but
if so, we'd like to put in our two cents worth here and say that we think
that multiple IPC pipes to the back-end server are probably a good thing.

- Bud

Bud Millwood
Weird Solutions, Inc.
http://www.weird-solutions.com
tel: [phone]
fax: [phone]

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

> On Tuesday 06 May 2008, Dmitry Yemanov wrote:
>
> <originally posted on the support forums>
>
>> Bud Millwood wrote:
>> > - Does the fbclient library acquire a process-wide lock when a thread
>> > passes through it (even when connecting to fb-classic)? If so, doesn't
>> > that counteract the gains we get by running multiple fb-classic
>> > instances?
>>
>> Yes, fbclient is synchronized using some kind of global mutex. It is
>> acquired when any thread enters the API call and released as soon as the
>> network packet is ready to be sent to the remote server. So, the
>> blocking time is very short (compared to the network transport overhead
>> and the server side processing of the call) and I don't think it really
>> limits your performance.
>
> Hi Dmitry:
>
> I grabbed a developer here this morning and went back over our logic and our
> numbers, and we are both now reasonably sure that this lock *is* the source
> of the high contention. And from what we can tell, this lock is causing
> Firebird to fail to scale to its full potential.

    I don't think so.

    Anyway - you may try to use fbclient from current HEAD (v2.5).
It have no global mutex. Just download snapshot build and test it.

Regards,
Vlad

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

> On this same note, we've been talking about the two architectures here, and
> the direction of FB 3:
>
> It's my understanding that with FB-Classic, when you make a new connection to
> the server (either from a new process or a new thread), you get a new TCP
> socket, which effectively gives you scalable IPC bandwidth. Instead of trying
> to ram all connections through one socket, you spread them out over multiple
> sockets, and this might be a big advantage on a multi-core system.

    FB never multiplexed all connections via one socket.
 
> I read somewhere that the FB 3 architecture is leaning towards SuperServer. I
> don't know if SS multiplexes all connections over a single socket today, but

    No. Not today, not yesterday, never.

Regards,
Vlad

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Bud Millwood]

On Tuesday 06 May 2008, Vlad Khorsun wrote:

> > On Tuesday 06 May 2008, Dmitry Yemanov wrote:
> >
> > <originally posted on the support forums>
> >
> >> Bud Millwood wrote:
> >> > - Does the fbclient library acquire a process-wide lock when a thread
> >> > passes through it (even when connecting to fb-classic)? If so, doesn't
> >> > that counteract the gains we get by running multiple fb-classic
> >> > instances?
> >>
> >> Yes, fbclient is synchronized using some kind of global mutex. It is
> >> acquired when any thread enters the API call and released as soon as the
> >> network packet is ready to be sent to the remote server. So, the
> >> blocking time is very short (compared to the network transport overhead
> >> and the server side processing of the call) and I don't think it really
> >> limits your performance.
> >
> > Hi Dmitry:
> >
> > I grabbed a developer here this morning and went back over our logic and
> > our numbers, and we are both now reasonably sure that this lock *is* the
> > source of the high contention. And from what we can tell, this lock is
> > causing Firebird to fail to scale to its full potential.
>
>     I don't think so.
>
>     Anyway - you may try to use fbclient from current HEAD (v2.5).
> It have no global mutex. Just download snapshot build and test it.

Thanks for the info, Vlad. We'll build and test it and I'll report my findings
if this version makes a difference.

- Bud

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

>>     Anyway - you may try to use fbclient from current HEAD (v2.5).
>> It have no global mutex. Just download snapshot build and test it.
>
> Thanks for the info, Vlad. We'll build and test it and I'll report my findings
> if this version makes a difference.

    We also very interested in testing of Firebird scalability :)

Regards,
Vlad

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Bud Millwood]

On Tuesday 06 May 2008, Vlad Khorsun wrote:
> >>     Anyway - you may try to use fbclient from current HEAD (v2.5).
> >> It have no global mutex. Just download snapshot build and test it.
> >
> > Thanks for the info, Vlad. We'll build and test it and I'll report my
> > findings if this version makes a difference.
>
>     We also very interested in testing of Firebird scalability :)

We've just built and tested 'libfbclient.so' from the head branch and tested
it against the release version 2.0.3. In other words, we only switched out
the client library, and left the database server itself intact at version
2.0.3.

The performance improvement is huge:

fbclient-head: 6% processing time spent in 'futex'
fbclient-2.0.3: 75% processing time spent in 'futex'

Our throughput almost *doubled* to 1000 TPS. And I think that's the most we're
likely to see on this 8-core system.

To be certain, we tested by switching back and forth between fbclient
versions, with no changes to anything else in the system.

So the fbclient mutex in 2.0.3 is a serious scalability bottleneck. Regardless
of how many CPUs you have, if you run around 3 threads wide-open they'll
spend most of their time waiting on that mutex.

- Bud

Bud Millwood
Weird Solutions, Inc.
http://www.weird-solutions.com
tel: [phone]
fax: [phone]

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Vlad Khorsun-2]

> On Tuesday 06 May 2008, Vlad Khorsun wrote:
>> >>     Anyway - you may try to use fbclient from current HEAD (v2.5).
>> >> It have no global mutex. Just download snapshot build and test it.
>> >
>> > Thanks for the info, Vlad. We'll build and test it and I'll report my
>> > findings if this version makes a difference.
>>
>>     We also very interested in testing of Firebird scalability :)
>
> We've just built and tested 'libfbclient.so' from the head branch and tested
> it against the release version 2.0.3. In other words, we only switched out
> the client library, and left the database server itself intact at version
> 2.0.3.
>
> The performance improvement is huge:
>
> fbclient-head: 6% processing time spent in 'futex'
> fbclient-2.0.3: 75% processing time spent in 'futex'
>
> Our throughput almost *doubled* to 1000 TPS. And I think that's the most we're
> likely to see on this 8-core system.

    Very interesting results, thanks !

Regards,
Vlad

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel