|
»
Concurrent handled of received bundled message
|
View:
New views
5 Messages
—
Rating Filter:
Alert me
|
 |
|
|
|
Re: Concurrent handled of received bundled message
by Bela Ban
::
Rate this Message:
Brian Stansberry wrote: > Bela, > > I saw JGRP-1090 today an it reminded me of something I was thinking > about back in Sept. > > On the receiver side, when a bundled message comes in, a single pool > thread unpacks it and then loops carrying each individual message up > the stack. > > I wonder if you can get some strange effects from that under load. > E.g. sender A sends bundle 1 and then bundle 2. On receiver B, thread > 1 is processing bundle 1 when bundle 2 comes in and gets handled by > thread 2. As a result, bundle 2 messages start arriving in NAKACK > before a bunch of bundle 1 messages, leading to extra retransmission > tasks. This won't happen if bundle 1 is processed before bundle 2: since all of the messages of a bundle are added to the NakReceiverWindow, once the CAS has been acquired, they're all processed before the CAS is released again. BTW: note that I made that process even more efficient by replacing the win.remove() message loop with a win.removeMany() which acquires the NRW lock less often. So all messages of a bundle are always processed as a unit. However, the issue you describe is valid: bundle 2 can be received before bundle 1, e.g. when the datagram representing bundle 1 is dropped by IP and needs to be retransmitted. If bundle 1 has seqnos [100-120] and bundle 2 [121-140], then we'll get retransmissions for [100-120]... > Problem is worse if thread 1 happens to block a bit above NAKACK. > > A possibility is to unpack the bundled message and then resubmit each > individual message to the pool. But I suspect that would cost more > than it gains. That's what we do ! > TBH, I doubt this is a big issue, but wanted to throw it out there in > case it sparks any thoughts on your end. I added your email as a comment/reminder so I can think about this when tackling JGRP-1090. -- Bela Ban Lead JGroups / Clustering Team JBoss ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Javagroups-development mailing list |
|
|
Re: Concurrent handled of received bundled message
by Brian Stansberry-4
::
Rate this Message:
On Nov 4, 2009, at 3:39 PM, Bela Ban wrote:
if(is_message_list) { // used if message bundling is enabled List<Message> msgs=readMessageList(dis); for(Message msg: msgs) { if(msg.isFlagSet(Message.OOB)) { log.warn("bundled message should not be marked as OOB"); } handleMyMessage(msg, multicast); } .... private void handleMyMessage(Message msg, boolean multicast) { if(stats) { num_msgs_received++; num_bytes_received+=msg.getLength(); } passMessageUp(msg, true, multicast, true); } } So the thread that reads the bundled message then carries each one up the stack one at a time. Letting multiple threads carry the messages up could easily be worse though. Then you'd have possible races between threads, e.g. bundle 1 msg 2 gets to NAKACK before bundle 1 msg 1.
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Javagroups-development mailing list |
|
|
Re: Concurrent handled of received bundled message
by Bela Ban
::
Rate this Message:
Brian Stansberry wrote: > >> >>> Problem is worse if thread 1 happens to block a bit above NAKACK. >>> >>> A possibility is to unpack the bundled message and then resubmit each >>> individual message to the pool. But I suspect that would cost more >>> than it gains. >> >> That's what we do ! >> > > Maybe I'm misunderstanding something. I see a loop in > TP.IncomingPacket.run() where each message gets passed up in sequence: > > if(is_message_list) { // used if message bundling is > enabled > List<Message> msgs=readMessageList(dis); > for(Message msg: msgs) { > if(msg.isFlagSet(Message.OOB)) { > log.warn("bundled message should not be > marked as OOB"); > } > handleMyMessage(msg, multicast); > } > .... > > private void handleMyMessage(Message msg, boolean multicast) { > if(stats) { > num_msgs_received++; > num_bytes_received+=msg.getLength(); > } > passMessageUp(msg, true, multicast, true); > } > } > > So the thread that reads the bundled message then carries each one up > the stack one at a time. > > Letting multiple threads carry the messages up could easily be worse > though. Then you'd have possible races between threads, e.g. bundle 1 > msg 2 gets to NAKACK before bundle 1 msg 1. Ah, I see what you mean now. I'll add a comment to 1090 to think about this: #1 A thread sequentially passes all messages of a bundle up the stack - If a bundle 1 [100-120] is processed, thread A passes 100 up first, then 101 and so on. Thread A currently removes and processes 100 - Meanwhile thread B processes bundle 2 [121-130], adds all messages into the NRW. This causes retransmission requests for [101-120] - When thread A returns from processing 100, it adds 101 to the NRW, cancelling the retransmission of 101, then 102 etc - The issue is that processing 101-120 takes time, during which retransmission for 101-120 will be going on A better solution would be if thread A added *all* messages [100-120] into the NRW *before* it started removing them. This would work with submitting each message of a bundle to the thread pool, e.g. if(is_message_list) { // used if message bundling is enabled List<Message> msgs=readMessageList(dis); for(Message msg: msgs) { if(msg.isFlagSet(Message.OOB)) { log.warn("bundled message should not be marked as OOB"); } pool.execute(new Runnable() {public void run() {handleMyMessage(msg, multicast);}} } However, this taxes the thread pool, creates more threads and thus more context switching, which is also overhead. As a matter of fact, I tried this out and got a 5-10% performance drop. Bela Ban Lead JGroups / Clustering Team JBoss ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Javagroups-development mailing list |
|
|
Re: Concurrent handled of received bundled message
by Brian Stansberry-4
::
Rate this Message:
On Nov 4, 2009, at 4:44 PM, Bela Ban wrote: > > Ah, I see what you mean now. I'll add a comment to 1090 to think > about this: > > #1 A thread sequentially passes all messages of a bundle up the stack > - If a bundle 1 [100-120] is processed, thread A passes 100 up > first, then 101 and so on. Thread A currently removes and processes > 100 > - Meanwhile thread B processes bundle 2 [121-130], adds all messages > into the NRW. This causes retransmission requests for [101-120] > - When thread A returns from processing 100, it adds 101 to the NRW, > cancelling the retransmission of 101, then 102 etc > - The issue is that processing 101-120 takes time, during which > retransmission for 101-120 will be going on > > A better solution would be if thread A added *all* messages > [100-120] into the NRW *before* it started removing them. This would > work with submitting each message of a bundle to the thread pool, e.g. > > if(is_message_list) { // used if message bundling is enabled > List<Message> msgs=readMessageList(dis); > for(Message msg: msgs) { > if(msg.isFlagSet(Message.OOB)) { > log.warn("bundled message should not be marked as OOB"); > } > pool.execute(new Runnable() {public void run > () {handleMyMessage(msg, multicast);}} > } > > > However, this taxes the thread pool, creates more threads and thus > more context switching, which is also overhead. As a matter of fact, > I tried this out and got a 5-10% performance drop. > > Wow; you're fast! Yeah, not too surprising it's slower. > > > Bela Ban > Lead JGroups / Clustering Team > JBoss > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Javagroups-development mailing list |
| Free embeddable forum powered by Nabble | Forum Help |
