Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B

[Nick Esborn]

The following reply was made to PR threads/136345; it has been noted by GNATS.

From: Nick Esborn <nick@...>
To: bug-followup@...,
 rink@...
Cc:  
Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B
Date: Wed, 15 Jul 2009 14:32:38 -0700

 This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
 --Apple-Mail-19-950902279
 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
 Content-Transfer-Encoding: 7bit
 
 Even after the above patch, I still run into occasional MySQL thread  
 deadlocks, which I originally described in what is now threads/135673.
 
 I also posted on freebsd-current a few days ago:
 
    http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html
 
 I'd be happy to collect whatever data would be helpful in tracking  
 down this deadlock.  This only seems to happen under our production  
 workload, so that might make it harder to capture meaningful debug  
 data, but I'm certainly willing to try.  I can also arrange for  
 developer access to the system in question, if that would help  
 significantly.
 
 -nick
 
 --
 nick@... - all messages cryptographically signed
 
 
 --Apple-Mail-19-950902279
 content-type: application/pgp-signature; x-mac-type=70674453;
  name=PGP.sig
 content-description: This is a digitally signed message part
 content-disposition: inline; filename=PGP.sig
 content-transfer-encoding: 7bit
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.9 (Darwin)
 
 iEYEARECAAYFAkpeSvYACgkQw1bX5UNr2ABajACeNpj/MW4X+4zfvlWNCXnqo6D9
 EZkAoLpGHxs4RoHMd7yqyba4IPzKxsJh
 =5I4q
 -----END PGP SIGNATURE-----
 
 --Apple-Mail-19-950902279--
_______________________________________________
freebsd-threads@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threads
To unsubscribe, send any mail to "freebsd-threads-unsubscribe@..."

[Attilio Rao-2]

2009/7/15 Nick Esborn <nick@...>:

> The following reply was made to PR threads/136345; it has been noted by GNATS.
>
> From: Nick Esborn <nick@...>
> To: bug-followup@...,
>  rink@...
> Cc:
> Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B
> Date: Wed, 15 Jul 2009 14:32:38 -0700
>
>  This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
>  --Apple-Mail-19-950902279
>  Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>  Content-Transfer-Encoding: 7bit
>
>  Even after the above patch, I still run into occasional MySQL thread
>  deadlocks, which I originally described in what is now threads/135673.
>
>  I also posted on freebsd-current a few days ago:
>
>    http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html
>
>  I'd be happy to collect whatever data would be helpful in tracking
>  down this deadlock.  This only seems to happen under our production
>  workload, so that might make it harder to capture meaningful debug
>  data, but I'm certainly willing to try.  I can also arrange for
>  developer access to the system in question, if that would help
>  significantly.

So did you backport this to 7 and still experience deadlocks?
I just committed the fix to HEAD not to STABLE branch.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
freebsd-threads@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threads
To unsubscribe, send any mail to "freebsd-threads-unsubscribe@..."

[Attilio Rao-2]

2009/7/16 Attilio Rao <attilio@...>:

> 2009/7/15 Nick Esborn <nick@...>:
>> The following reply was made to PR threads/136345; it has been noted by GNATS.
>>
>> From: Nick Esborn <nick@...>
>> To: bug-followup@...,
>>  rink@...
>> Cc:
>> Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B
>> Date: Wed, 15 Jul 2009 14:32:38 -0700
>>
>>  This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
>>  --Apple-Mail-19-950902279
>>  Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>  Content-Transfer-Encoding: 7bit
>>
>>  Even after the above patch, I still run into occasional MySQL thread
>>  deadlocks, which I originally described in what is now threads/135673.
>>
>>  I also posted on freebsd-current a few days ago:
>>
>>    http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html
>>
>>  I'd be happy to collect whatever data would be helpful in tracking
>>  down this deadlock.  This only seems to happen under our production
>>  workload, so that might make it harder to capture meaningful debug
>>  data, but I'm certainly willing to try.  I can also arrange for
>>  developer access to the system in question, if that would help
>>  significantly.
>
> So did you backport this to 7 and still experience deadlocks?
> I just committed the fix to HEAD not to STABLE branch.

Ok, I got, you just upgraded.
Can you try the following things?:
- Upgrade to the -CURRENT of today
- Recompile the kernel with the following options:
KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT,
INVARIANTS, WITNESS
- When the deadlock takes place break into DDB and please retrieve the
following info:
db> show allpcpu
db> ps
db> alltrace
db> show alllock

- Save them with a serial console output or using the textdump(4) format.

(if necessary read the ddb(4) and textdump(4) before to set up the
whole system).
This would shade a light if the problem lives within the kernel or not.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
freebsd-threads@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threads
To unsubscribe, send any mail to "freebsd-threads-unsubscribe@..."

[Nick Esborn]

On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote:

> 2009/7/16 Attilio Rao <attilio@...>:
>> 2009/7/15 Nick Esborn <nick@...>:
>>> The following reply was made to PR threads/136345; it has been  
>>> noted by GNATS.
>>>
>>> From: Nick Esborn <nick@...>
>>> To: bug-followup@...,
>>> rink@...
>>> Cc:
>>> Subject: Re: threads/136345: Recursive read rwlocks in thread A  
>>> cause deadlock with write lock in thread B
>>> Date: Wed, 15 Jul 2009 14:32:38 -0700
>>>
>>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
>>> --Apple-Mail-19-950902279
>>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>> Content-Transfer-Encoding: 7bit
>>>
>>> Even after the above patch, I still run into occasional MySQL thread
>>> deadlocks, which I originally described in what is now threads/
>>> 135673.
>>>
>>> I also posted on freebsd-current a few days ago:
>>>
>>>   http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html
>>>
>>> I'd be happy to collect whatever data would be helpful in tracking
>>> down this deadlock.  This only seems to happen under our production
>>> workload, so that might make it harder to capture meaningful debug
>>> data, but I'm certainly willing to try.  I can also arrange for
>>> developer access to the system in question, if that would help
>>> significantly.
>>
>> So did you backport this to 7 and still experience deadlocks?
>> I just committed the fix to HEAD not to STABLE branch.
>
> Ok, I got, you just upgraded.
> Can you try the following things?:
> - Upgrade to the -CURRENT of today
> - Recompile the kernel with the following options:
> KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT,
> INVARIANTS, WITNESS
> - When the deadlock takes place break into DDB and please retrieve the
> following info:
> db> show allpcpu
> db> ps
> db> alltrace
> db> show alllock
>
> - Save them with a serial console output or using the textdump(4)  
> format.
>
> (if necessary read the ddb(4) and textdump(4) before to set up the
> whole system).
> This would shade a light if the problem lives within the kernel or  
> not.
>
> Thanks,
> Attilio
>
>
> --
> Peace can only be achieved by understanding - A. Einstein

I can definitely do the upgrade.

KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on.  I will try  
FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when  
I first upgraded to 8.0, this server was unable to handle its workload  
with the INVARIANTS and WITNESS options turned on.

Also, it can take a while for it to become clear that the deadlock has  
occurred -- usually our monitoring picks it up when replication falls  
behind.  So it may be 15-20 minutes after the deadlock that I am able  
to run the above db commands.  Of course the thread will still be  
deadlocked.  Hopefully that doesn't reduce the value of the data  
obtained.

Thanks,

-nick

--
nick@... - all messages cryptographically signed

[Attilio Rao-2]

2009/7/16 Nick Esborn <nick@...>:
>
>
> KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on.  I will try
> FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when I
> first upgraded to 8.0, this server was unable to handle its workload with
> the INVARIANTS and WITNESS options turned on.

What do you mean with 'unable'? What was happening precisely?

> Also, it can take a while for it to become clear that the deadlock has
> occurred -- usually our monitoring picks it up when replication falls
> behind.  So it may be 15-20 minutes after the deadlock that I am able to run
> the above db commands.  Of course the thread will still be deadlocked.
>  Hopefully that doesn't reduce the value of the data obtained.

It should be still fine.

Thanks,
Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
freebsd-threads@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threads
To unsubscribe, send any mail to "freebsd-threads-unsubscribe@..."

[Nick Esborn]

On Jul 16, 2009, at 10:53 AM, Attilio Rao wrote:

> 2009/7/16 Nick Esborn <nick@...>:
>>
>>
>> KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on.  I will  
>> try
>> FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but  
>> when I
>> first upgraded to 8.0, this server was unable to handle its  
>> workload with
>> the INVARIANTS and WITNESS options turned on.
>
> What do you mean with 'unable'? What was happening precisely?

System time would rise during periods of peak demand, and the system  
would quickly fall behind on its workload of queries.

However, I have some hardware I can dedicate to this, and only run the  
one MySQL data set which exhibits this problem.  That should be enough  
of a workload reduction to allow the server to keep up even with all  
the above options turned on.

-nick

>
>> Also, it can take a while for it to become clear that the deadlock  
>> has
>> occurred -- usually our monitoring picks it up when replication falls
>> behind.  So it may be 15-20 minutes after the deadlock that I am  
>> able to run
>> the above db commands.  Of course the thread will still be  
>> deadlocked.
>> Hopefully that doesn't reduce the value of the data obtained.
>
> It should be still fine.
>
> Thanks,
> Attilio
>
>
> --
> Peace can only be achieved by understanding - A. Einstein

--
nick@... - all messages cryptographically signed