Nabble - JGroups

How does AUTH now work?

2009-11-29T08:05:16Z

Hi,

How do I introduce authentication in a cluster?
I was assuming that this should be the way to do it:

<AUTH auth_class="org.jgroups.auth.SimpleToken" auth_value="chris"/>

However, in 2.8, the attribute auth_value seems to have disappeared (see http://www.jgroups.org/schema/JGroups-2.8.xsd). The only allowed ones are:
auth_class, name, and stats.

Thanks,

Klaus.
--
Sarah Kreuz, die DSDS-Siegerin der Herzen, mit ihrem eindrucksvollen
Debütalbum "One Moment in Time". http://portal.gmx.net/de/go/musik

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: Help me get Tunneling Right, Please!

2009-11-29T04:53:40Z

You've misunderstood how the GossipRouter works: if you run a
GossipRouter, then *all* nodes have to have the *same* (TUNNEL-based)
config. So in your case this would be tunnel1.xml.

If some nodes run on TUNNEL and some on TCP, then the TCP based nodes
won't be able to access the TUNNEL based nodes, and vice versa.

In JGroups, all nodes are peers, there are no clients or servers

Klaus Teller wrote:

> Hi,
>
> I have one client and one server in my test setup where the client is supposed to run behind a firewall and connect to the gossip router to communicate with the remainder of the cluster.
>
> I still don't manage to get the configuration right.
>
> The test code for the client (the node behind firewall) is here: http://pastebin.com/m6267619d. Its configuration is here: http://pastebin.com/f6e1c133a. The logs produced by the client are here: http://pastebin.com/f7d659a66.
>
>
> The test code for the server (node outside of the firewall+gossip router) is here: http://pastebin.com/m1225cdbc. And its configuration is here:
> http://pastebin.com/f51b6dff3. The logs it produces are here:
> http://pastebin.com/m62c88ddb.
>
> Note: I have no firewall on my computer and no idea why the connection exception is occurring. I'm running 2.8.0.CR6.
>
> What is wrong my configuration or with my code?
>
> I appreciate any input.
> Klaus.
>

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-29T04:40:58Z

This feature has been put on the backburner, I might not even implement
it. If I do, then in 3.0, which requires changes anyway.

I don't know who's using this event currently, are you guys prepared to
change your code if I made this change ?

Galder Zamarreno wrote:

> Hmmm, even if it's not done yet and it can only return one
> PhysicalAddress, that method should return PhysicalAddress[] with only
> one element or similar.
>
> Otherwise, when you do the work, the code in Infinispan will break.
>
> On 11/26/2009 09:34 PM, Bela Ban wrote:
>
>> This is not done yet
>>
>> Galder Zamarreno wrote:
>>
>>> Hey Bela,
>>>
>>> If as per
>>> http://belaban.blogspot.com/2009/02/whats-cool-about-logical-addresses.html,
>>> a logical address can map to one or more physical addresses, why
>>> callling the following returns a single Physical address?
>>>
>>> channel.downcall(new Event(Event.GET_PHYSICAL_ADDRESS,
>>> channel.getAddress()));
>>>
>>> Shouldn't it return a collection of PhysicalAddress?
>>>
>>> Cheers,
>>>
>>>
>
>

From forum: JGroups - Dev

Re: Suspect warning in jgroups 2.8.0 CR6

2009-11-29T04:37:22Z

Did you set bind_addr in udp.xml or -Djgroups.bind_addr=1.2.3.4 ?

Possibly FD_SOCK picked a differrent NIC to bind to and cannot establish
a socket connection to its counterpart...

Shay Banon wrote:

> Hi,
>
> I just upgraded from 2.8.0 CR1 to 2.8.0 CR6 and I started getting strange
> suspect messages. I created a simple program that basically does the
> following:
>
> channel1 = new JChannel("udp.xml");
> channel1.connect("test");
>
> channel2 = new JChannel("udp.xml");
> channel2.connect("test");
>
> And I get two suspect messages:
>
> [11:15:43,806][WARN ][org.jgroups.protocols.FD_SOCK ] I
> (computer.local-49080) was suspected by computer.local-3745; ignoring the
> SUSPECT message
> [11:15:43,806][WARN ][org.jgroups.protocols.FD_SOCK ] I
> (computer.local-3745) was suspected by computer.local-49080; ignoring the
> SUSPECT message
>
> Any idea why?
>
> Cheers,
> Shay
>
>

From forum: JGroups - General

Suspect warning in jgroups 2.8.0 CR6

2009-11-29T01:19:11Z

Hi,

I just upgraded from 2.8.0 CR1 to 2.8.0 CR6 and I started getting strange suspect messages. I created a simple program that basically does the following:

channel1 = new JChannel("udp.xml");

channel1.connect("test");

channel2 = new JChannel("udp.xml");

channel2.connect("test");

And I get two suspect messages:

[11:15:43,806][WARN ][org.jgroups.protocols.FD_SOCK ] I (computer.local-49080) was suspected by computer.local-3745; ignoring the SUSPECT message

[11:15:43,806][WARN ][org.jgroups.protocols.FD_SOCK ] I (computer.local-3745) was suspected by computer.local-49080; ignoring the SUSPECT message

Any idea why?

Cheers,

Shay

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

SimpleChat tutorial sample from jgroups.org

2009-11-28T21:22:50Z

Hi, I'm learning JGroups so I'm sorry for the very basic questions, but I really need some help. I'm running this sample without *any* modifications.: http://www.jgroups.org/tutorial/code/SimpleChat.java

It seems that the callback for the receive() method is not working. Everything else works properly. I have no idea why the callback fails.

If run my own example code, with a class that implements ReceiverAdapter, I can receive a sent message if I explicity invoke the receive() method. But the callback also fails in that case.

I'm sorry since I am very new to this but I could really use an explanation or suggestions.

From forum: JGroups - General

Help me get Tunneling Right, Please!

2009-11-28T18:51:21Z

Hi,

I have one client and one server in my test setup where the client is supposed to run behind a firewall and connect to the gossip router to communicate with the remainder of the cluster.

I still don't manage to get the configuration right.

The test code for the client (the node behind firewall) is here: http://pastebin.com/m6267619d. Its configuration is here: http://pastebin.com/f6e1c133a. The logs produced by the client are here: http://pastebin.com/f7d659a66.

The test code for the server (node outside of the firewall+gossip router) is here: http://pastebin.com/m1225cdbc. And its configuration is here:
http://pastebin.com/f51b6dff3. The logs it produces are here:
http://pastebin.com/m62c88ddb.

Note: I have no firewall on my computer and no idea why the connection exception is occurring. I'm running 2.8.0.CR6.

What is wrong my configuration or with my code?

I appreciate any input.
Klaus.
--
Endlich! Das Debüt-Album von Pop-Diva Sarah Kreuz ist da!
Jetzt bei GMX Musik Downloads. http://portal.gmx.net/de/go/musik01

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Set logging level to SEVERE

2009-11-28T10:04:19Z

Greetings

I'm using JGroups in a project where we use java.util.logging instead of
log4j. I've been unable to find out how to modify the logging level to
SEVERE and avoid the multiple warning messages.

Thanks in advanve.

--
========================================================================
0 0 0 Pablo Chacin | Dept. d'Arquitectura de Computadors
0 0 0 e-mail: pchacin@... | UPC-Campus Nord. Modul D6-212
0 0 0 phone: +34 +93 405 4059 | Jordi Girona, 1-3
U P C fax: +34 +93 401 7055 | 08034 Barcelona - SPAIN
www: http://personals.ac.upc.es/pchacin
========================================================================

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-27T10:14:20Z

Hmmm, even if it's not done yet and it can only return one
PhysicalAddress, that method should return PhysicalAddress[] with only
one element or similar.

Otherwise, when you do the work, the code in Infinispan will break.

On 11/26/2009 09:34 PM, Bela Ban wrote:

> This is not done yet
>
> Galder Zamarreno wrote:
>> Hey Bela,
>>
>> If as per
>> http://belaban.blogspot.com/2009/02/whats-cool-about-logical-addresses.html,
>> a logical address can map to one or more physical addresses, why
>> callling the following returns a single Physical address?
>>
>> channel.downcall(new Event(Event.GET_PHYSICAL_ADDRESS,
>> channel.getAddress()));
>>
>> Shouldn't it return a collection of PhysicalAddress?
>>
>> Cheers,
>>
>

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

Re: Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-27T10:14:13Z

From forum: JGroups - Dev

Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-27T00:27:42Z

Hey Bela,

If as per
http://belaban.blogspot.com/2009/02/whats-cool-about-logical-addresses.html,
a logical address can map to one or more physical addresses, why
callling the following returns a single Physical address?

channel.downcall(new Event(Event.GET_PHYSICAL_ADDRESS,
channel.getAddress()));

Shouldn't it return a collection of PhysicalAddress?

Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

Re: Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-26T12:34:09Z

This is not done yet

Galder Zamarreno wrote:

> Hey Bela,
>
> If as per
> http://belaban.blogspot.com/2009/02/whats-cool-about-logical-addresses.html,
> a logical address can map to one or more physical addresses, why
> callling the following returns a single Physical address?
>
> channel.downcall(new Event(Event.GET_PHYSICAL_ADDRESS,
> channel.getAddress()));
>
> Shouldn't it return a collection of PhysicalAddress?
>
> Cheers,
>

From forum: JGroups - Dev

Shouldn't Event.GET_PHYSICAL_ADDRESS return a collection PhysicalAddress instances?

2009-11-26T09:57:18Z

From forum: JGroups - Dev

Re: Phantom group member causing log-spam

2009-11-24T00:50:29Z

Can you try with 2.6.13.GA ? Also, use the udp.xml shipped with
2.6.13.GA; you have a few non-standard property values

Eric Dalquist wrote:

> We've seen this happen a few times where a member leaves the group due to
> normal app shutdown but the group doesn't get the leave request and never
> realizes the old member is gone. It appears to result in one member
> constantly trying to connect to this phantom machine and spamming the logs
> with the following every second:
>
> DEBUG [FD_SOCK pinger,EH_CACHE,1.1.1.1:37931] protocols.FD_SOCK.[] Nov/04
> 16:28:30.245 - determinePingDest()=2.2.2.2:61297
> ERROR [FD_SOCK pinger,EH_CACHE,1.1.1.1:37931] protocols.FD_SOCK.[] Nov/04
> 16:28:31.265 - socket address for 2.2.2.2:61297 could not be fetched,
> retrying
>
> Our config string is:
> UDP(mcast_addr=M.M.M.M;mcast_port=4447;ip_ttl=8;receive_on_all_interfaces=true):PING:MERGE2:FD_SOCK:VERIFY_SUSPECT:pbcast.NAKACK:UNICAST:pbcast.STABLE:FRAG:pbcast.GMS
>
>
> Here is a synopsis that I've been able to glean from the logs. Linked at the
> bottom of the page are the full logs from the time period for each of the
> four members of the group.
>
>
> 16:19:21.600 - server1 (1.1.1.1) leaves due to app shutdown
> 16:20:07.170 - server1 (1.1.1.1) app startup
> 16:20:12.012 - server4 (4.4.4.4:59176) fails to receive ACK from server2
> (2.2.2.2:61297) after merge from server1 (1.1.1.1:37931)
>
> 16:25:46.434 - server2 (2.2.2.2:61297) starts leave process due to app
> shutdown - notifies server4 (4.4.4.4:59176)
> ***** NO LOG ON TETI FOR PEPI LEAVING!!!!! *****
> 16:26:01.501 - server2 (2.2.2.2:61297) leave complete
>
> 16:28:24.166 - server2 (2.2.2.2:61899) app start
> 16:28:30.061 - server4 (4.4.4.4:59176) MERGE request from server2 - multiple
> coordinators
> 16:28:30.118 - server4 (4.4.4.4:59176) two leaders (server4 & server2)
> determines server4 is leader
> 16:28:30.173 - server1 (1.1.1.1:37931) MergeView
> 16:28:30.196 - server1 (1.1.1.1:37931) discarded message from non-member
> 2.2.2.2:61899
> 16:28:30.240 - joined: server2 (2.2.2.2:61899)
> 16:28:30.245 - socket to server4 (4.4.4.4:59176) was reset
> 16:28:30.245 - Starting SPAM: determinePingDest()=2.2.2.2:61297 / socket
> address for 2.2.2.2:61297 could not be fetched, retrying
>
>
>
> http://old.nabble.com/file/p26483925/server1.portal.log.2009-11-04-16
> server1.portal.log.2009-11-04-16
> http://old.nabble.com/file/p26483925/server2.portal.log.2009-11-04-16
> server2.portal.log.2009-11-04-16
> http://old.nabble.com/file/p26483925/server3.portal.log.2009-11-04-16
> server3.portal.log.2009-11-04-16
> http://old.nabble.com/file/p26483925/server4.portal.log.2009-11-04-16
> server4.portal.log.2009-11-04-16
>
>

From forum: JGroups - General

Phantom group member causing log-spam

2009-11-23T11:00:07Z

We've seen this happen a few times where a member leaves the group due to normal app shutdown but the group doesn't get the leave request and never realizes the old member is gone. It appears to result in one member constantly trying to connect to this phantom machine and spamming the logs with the following every second:

DEBUG [FD_SOCK pinger,EH_CACHE,1.1.1.1:37931] protocols.FD_SOCK.[] Nov/04 16:28:30.245 - determinePingDest()=2.2.2.2:61297
ERROR [FD_SOCK pinger,EH_CACHE,1.1.1.1:37931] protocols.FD_SOCK.[] Nov/04 16:28:31.265 - socket address for 2.2.2.2:61297 could not be fetched, retrying

Our config string is:
UDP(mcast_addr=M.M.M.M;mcast_port=4447;ip_ttl=8;receive_on_all_interfaces=true):PING:MERGE2:FD_SOCK:VERIFY_SUSPECT:pbcast.NAKACK:UNICAST:pbcast.STABLE:FRAG:pbcast.GMS

Here is a synopsis that I've been able to glean from the logs. Linked at the bottom of the page are the full logs from the time period for each of the four members of the group.

16:19:21.600 - server1 (1.1.1.1) leaves due to app shutdown
16:20:07.170 - server1 (1.1.1.1) app startup
16:20:12.012 - server4 (4.4.4.4:59176) fails to receive ACK from server2 (2.2.2.2:61297) after merge from server1 (1.1.1.1:37931)

16:25:46.434 - server2 (2.2.2.2:61297) starts leave process due to app shutdown - notifies server4 (4.4.4.4:59176)
***** NO LOG ON TETI FOR PEPI LEAVING!!!!! *****
16:26:01.501 - server2 (2.2.2.2:61297) leave complete

16:28:24.166 - server2 (2.2.2.2:61899) app start
16:28:30.061 - server4 (4.4.4.4:59176) MERGE request from server2 - multiple coordinators
16:28:30.118 - server4 (4.4.4.4:59176) two leaders (server4 & server2) determines server4 is leader
16:28:30.173 - server1 (1.1.1.1:37931) MergeView
16:28:30.196 - server1 (1.1.1.1:37931) discarded message from non-member 2.2.2.2:61899
16:28:30.240 - joined: server2 (2.2.2.2:61899)
16:28:30.245 - socket to server4 (4.4.4.4:59176) was reset
16:28:30.245 - Starting SPAM: determinePingDest()=2.2.2.2:61297 / socket address for 2.2.2.2:61297 could not be fetched, retrying

server1.portal.log.2009-11-04-16
server2.portal.log.2009-11-04-16
server3.portal.log.2009-11-04-16
server4.portal.log.2009-11-04-16

From forum: JGroups - General

Re: Advice on UDP configuration for large cluster

2009-11-22T02:33:34Z

I used the stock udp.xml.

Changes I'd make (n random order):

* UDP: adjust the thread pool sizes
* UDP: increase the number of timer threads
* Use FD_ALL instead of FD
* NAKACK: set use_mcast_xmit to true
* GMS: enable view bundling (handles concurrent joins or leaves)
* FC: increase max_credits

I haven't tested these suggestions, but I will starting early in 2009.

Note that I hope to get optimal values for properties with ergonomics
[1]. For example, rather than setting retransmit timeouts in NAKACK and
UNICAST, these values will be adjusted by JGroups itself, and can be
omitted from the config. This is similar to what the JVM does.

For example, currently FC.max_credits is fixed, however it should be a
function of the number of cluster nodes, the loss rate / receiver and
the number of bytes sent and received. This would allow us to drop the
send rate when the rate of lost / retransmitted messages gets high and -
conversely - increase it when the loss rate decreases. This is also
similar to TCP's sliding window protocol with linear increase and
exponential backoff nased on the ASCKs lost/received.

Again, large scale testing is in a very early stage, bu I hope to
collect important information through folks like you (or others) running
JGroups in large clusters...

[1] https://jira.jboss.org/jira/browse/JGRP-1052

seamus donohue wrote:

> Thanks Bela,
> I will keep an eye on the wiki, certainly will add our experiences to
> this page too when we are finished.
> Would you care to share the protocol stack configuration that you used for
> your 20 process X 4 Server ? would be interesting to see what protocol
> parameters you tweeked.
>
> -Seamus
>
> Bela Ban wrote:
>
>> I recently ran an 80 node cluster, with 20 processes on 4 physical
>> servers. This is not the same of course as having more physical servers,
>> as the bandwidth and memory/CPU are shared, but there were no issues in
>> doing this.
>>
>> I've started a wiki [1] on best practices for running large JGroups
>> clusters. Currently, this points to JGRP-100, but it is intended as a
>> tips&tricks page, for everybody to add their experience with large clusters.
>>
>> I hope to start testing on a 1000 node EC2 cluster (on TCP though) early
>> next year, and will certainly share my experience
>>
>> [1] http://www.jboss.org/community/wiki/Largeclusters
>>
>>
>> Seamus Donohue wrote:
>>
>>
>>> Hi,
>>> I am planning to configure a cluster of approximately 60~100 JGroup
>>> Members (estimate 4 to 5 JVM members per server) probably on JGroups
>>> 2.8 when it GA's later this year. From searching the forums there is
>>> anecdotal evidence that this cluster size is feasible (e.g. see
>>> thread:
>>> http://sourceforge.net/projects/javagroups/forums/forum/18796/topic/2087819).
>>> The cluster will be used to distribute small events < 10K size approx.
>>> The volume is not expected to be high as these are management related
>>> events.I am planning to use a UDP configuration based on the standard
>>> udp.xml that is delivered with the JGroups distribution.
>>>
>>> Can anyone share their advice and or experience on protocol stack
>>> configuration for a cluster of this size?
>>>
>>>
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> javagroups-users mailing list
> javagroups-users@...
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
>

From forum: JGroups - General

Re: Advice on UDP configuration for large cluster

2009-11-21T09:46:56Z

Thanks Bela,
I will keep an eye on the wiki, certainly will add our experiences to
this page too when we are finished.
Would you care to share the protocol stack configuration that you used for
your 20 process X 4 Server ? would be interesting to see what protocol
parameters you tweeked.

-Seamus

Bela Ban wrote:

> I recently ran an 80 node cluster, with 20 processes on 4 physical
> servers. This is not the same of course as having more physical servers,
> as the bandwidth and memory/CPU are shared, but there were no issues in
> doing this.
>
> I've started a wiki [1] on best practices for running large JGroups
> clusters. Currently, this points to JGRP-100, but it is intended as a
> tips&tricks page, for everybody to add their experience with large clusters.
>
> I hope to start testing on a 1000 node EC2 cluster (on TCP though) early
> next year, and will certainly share my experience
>
> [1] http://www.jboss.org/community/wiki/Largeclusters
>
>
> Seamus Donohue wrote:
>
>> Hi,
>> I am planning to configure a cluster of approximately 60~100 JGroup
>> Members (estimate 4 to 5 JVM members per server) probably on JGroups
>> 2.8 when it GA's later this year. From searching the forums there is
>> anecdotal evidence that this cluster size is feasible (e.g. see
>> thread:
>> http://sourceforge.net/projects/javagroups/forums/forum/18796/topic/2087819).
>> The cluster will be used to distribute small events < 10K size approx.
>> The volume is not expected to be high as these are management related
>> events.I am planning to use a UDP configuration based on the standard
>> udp.xml that is delivered with the JGroups distribution.
>>
>> Can anyone share their advice and or experience on protocol stack
>> configuration for a cluster of this size?
>>
>
>

From forum: JGroups - General

Re: Advice on UDP configuration for large cluster

2009-11-21T01:32:10Z

I recently ran an 80 node cluster, with 20 processes on 4 physical
servers. This is not the same of course as having more physical servers,
as the bandwidth and memory/CPU are shared, but there were no issues in
doing this.

I've started a wiki [1] on best practices for running large JGroups
clusters. Currently, this points to JGRP-100, but it is intended as a
tips&tricks page, for everybody to add their experience with large clusters.

I hope to start testing on a 1000 node EC2 cluster (on TCP though) early
next year, and will certainly share my experience

[1] http://www.jboss.org/community/wiki/Largeclusters

Seamus Donohue wrote:

> Hi,
> I am planning to configure a cluster of approximately 60~100 JGroup
> Members (estimate 4 to 5 JVM members per server) probably on JGroups
> 2.8 when it GA's later this year. From searching the forums there is
> anecdotal evidence that this cluster size is feasible (e.g. see
> thread:
> http://sourceforge.net/projects/javagroups/forums/forum/18796/topic/2087819).
> The cluster will be used to distribute small events < 10K size approx.
> The volume is not expected to be high as these are management related
> events.I am planning to use a UDP configuration based on the standard
> udp.xml that is delivered with the JGroups distribution.
>
> Can anyone share their advice and or experience on protocol stack
> configuration for a cluster of this size?

From forum: JGroups - General

Advice on UDP configuration for large cluster

2009-11-20T19:16:21Z

Hi,
I am planning to configure a cluster of approximately 60~100 JGroup Members (estimate 4 to 5 JVM members per server) probably on JGroups 2.8 when it GA's later this year. From searching the forums there is anecdotal evidence that this cluster size is feasible (e.g. see thread: http://sourceforge.net/projects/javagroups/forums/forum/18796/topic/2087819). The cluster will be used to distribute small events < 10K size approx. The volume is not expected to be high as these are management related events.I am planning to use a UDP configuration based on the standard udp.xml that is delivered with the JGroups distribution.

Can anyone share their advice and or experience on protocol stack configuration for a cluster of this size?

much appreciated,
-Seamus

Windows 7: Find the right PC for you. Learn more.
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: [jgoups-dev] Partial state from individual nodes

2009-11-20T14:38:48Z

Adi wrote:
>> The FC protocol takes care of flow control, ie. no sender can send at a
> faster rate than the receiver can process the messages.
>
> Ok, if FC takes care of congestion control, what happens if a recv. is
> slow
> and sender keeps doing sends. Do the messages get buffered on the sender
> side or does send return an error? If it is buffered, is there a way to
> control the size of this buffer and get notified if things get full or
> dropped or is it just bound by memory available?

Flow control (FC) [1] has credits, which are replenished by the
receiver(s). If a sender doesn't have enough credits, it'll block.
FC.max_credits is your sliding window size if you wish, except that the
window is static, unlike in TCP.

> We monitor warn and error logs in our app servers for potential
> problems. I
> also wanted to know if jgroups has its own warnings and error messages,
> which can be directed to the app logs? An example is a block() is blocking
> for too long etc.

All protocols expose attributes and operations via JMX. Yes, JGroups
does log warnings and errors.

[1] http://www.jboss.org/community/docs/DOC-10884

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

Re: [jgoups-dev] Partial state from individual nodes

2009-11-20T12:45:58Z

>The FC protocol takes care of flow control, ie. no sender can send at a faster rate than the receiver can process the messages.

Ok, if FC takes care of congestion control, what happens if a recv. is slow and sender keeps doing sends. Do the messages get buffered on the sender side or does send return an error? If it is buffered, is there a way to control the size of this buffer and get notified if things get full or dropped or is it just bound by memory available?

We monitor warn and error logs in our app servers for potential problems. I also wanted to know if jgroups has its own warnings and error messages, which can be directed to the app logs? An example is a block() is blocking for too long etc.

Thanks,
Adi

On Thu, Nov 19, 2009 at 11:32 PM, Bela Ban <belaban@...> wrote:

Adi wrote:

Thanks Bela.

Another question: the NAKACK makes sure messages are recvd and acked. On
what layer is this ack? In the jgroups layer?

Yes. NAKACK actually doesn't ack messages, but only asks the sender for retransmission when it detects a gap. For example, when receiving 1, 2, 3, 5 from P, it'll ask P for retransmission of #4.

For example if i recv a message in receive() function (using
ReceiverAdapter()), will the ack happen only after recveive() callback
returns to the caller?

No. The message is considered *delivered* as soon as it (a) has been received and (b) is in order. Then it is sent up the stack for consumption (receive()).

I was worried about the possiblity of the jgroups
message queue getting full and messages being dropped because receive() is
not able to process messages fast enough.

We don't use queues anymore in JGroups. The FC protocol takes care of flow control, ie. no sender can send at a faster rate than the receiver can process the messages.

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

--
Thanks,
Adi

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

https://jira.jboss.org/jira/browse/JGRP-1103

2009-11-20T08:21:22Z

FYI,

A critical regression in 2.8 where messages can be delivered twice:

Turns out this is not just related to OOB messages but to *regular*
messages as well, because the culprit is NakReceiverWindow: when I
optimized it by moving certain removals and insertions to the hashmap
used by NRW out of the lock scope, I introduced a regression.

Although the bug can only by reproduced by running a @Test with
invocationCount=5, with 200 threads each concurrently inserting 5000
seqnos, and even then it only happens in 1 or 2 out of 15 tests, this is
a critical bug !

I'll fix this tomorrow (the stress test actually passes 100% all the
times when I undo the optimization !)...

It would cause some messages to be delivered multiple times.

This issue does not exist in 2.6.x, as the 'optimizations' were done
only in the 2.8 branch.

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

Re: Node started before Gossip = no new view after Gossip started

2009-11-20T03:07:12Z

Linas Virbalas wrote:
> Thanks for the links, I've read:
>
> JGRP-716> In a TCP based config *without* UNICAST, messages can get
> delivered out of order if a thread pool is enabled at the transport level
>
> Incidentally, my config has thread_pool disabled

This is not good. I know we do have the enable_xxx_pool property, but
I'll remove it in 3.0, becaue all of our tests assume a thread pool.
Disabling a thread pool is just not the default, and we don't test it.

> <TCP loopback="false" discard_incompatible_packets="true"
> use_incoming_packet_handler="true" enable_bundling="false"
> enable_diagnostics="true" thread_naming_pattern="cl"
> use_concurrent_stack="false" thread_pool.enabled="false"
> thread_pool.min_threads="8" thread_pool.max_threads="40"
> thread_pool.keep_alive_time="5000" thread_pool.queue_enabled="false"
> thread_pool.queue_max_size="100" thread_pool.rejection_policy="Run"
> oob_thread_pool.enabled="false" oob_thread_pool.min_threads="8"
> oob_thread_pool.max_threads="20"
> oob_thread_pool.keep_alive_time="5000"
> oob_thread_pool.queue_enabled="false"
> oob_thread_pool.queue_max_size="100"
> oob_thread_pool.rejection_policy="Run"/>
>
> Does this mean that in this case UNICAST is not needed after all?

No, you'll always need UNICAST. Messages could get retransmitted, e.g.
via an OOB bundle, and then you have a race between a unicast message
handled by the regular pool and one handled by the OOB pool, so UNICAST
is always needed.

Note that I modified SequencerOrderTest to have multiple threads sending
messages randomly on 3 channels, and then check (1) whether all messages
have been received and (2) whether all messages are in total order.

I've also made a few modifications to SEQUENCER (committed to head).

What was your original issue again, was it ordering related ?

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: [jgoups-dev] Partial state from individual nodes

2009-11-19T23:32:47Z

Adi wrote:
> Thanks Bela.
>
> Another question: the NAKACK makes sure messages are recvd and acked. On
> what layer is this ack? In the jgroups layer?

Yes. NAKACK actually doesn't ack messages, but only asks the sender for
retransmission when it detects a gap. For example, when receiving 1, 2,
3, 5 from P, it'll ask P for retransmission of #4.

> For example if i recv a message in receive() function (using
> ReceiverAdapter()), will the ack happen only after recveive() callback
> returns to the caller?

No. The message is considered *delivered* as soon as it (a) has been
received and (b) is in order. Then it is sent up the stack for
consumption (receive()).

> I was worried about the possiblity of the jgroups
> message queue getting full and messages being dropped because receive() is
> not able to process messages fast enough.

We don't use queues anymore in JGroups. The FC protocol takes care of
flow control, ie. no sender can send at a faster rate than the receiver
can process the messages.

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev

Re: [jgoups-dev] Partial state from individual nodes

2009-11-19T11:17:28Z

Thanks Bela.

Another question: the NAKACK makes sure messages are recvd and acked. On what layer is this ack? In the jgroups layer?

For example if i recv a message in receive() function (using ReceiverAdapter()), will the ack happen only after recveive() callback returns to the caller? I was worried about the possiblity of the jgroups message queue getting full and messages being dropped because receive() is not able to process messages fast enough.

Thanks,
Adi

On Wed, Nov 18, 2009 at 11:19 PM, Bela Ban <belaban@...> wrote:

Adi wrote:

Hi all,

I am checking out Jgroups and it looks great for supporting my clustering
needs. I had a basic question on sharing state when nodes join a cluster.

I read the getState() documentation and it claims, it gets state from one of
the nodes (usually the oldest).

Yes. Usually, you pass null as first arg of getState(), that way the state is fetched from the coordinator (oldest cluster node).

I also read about paritial states using a String.

Can we do something like getState(nodeID) and get the state of a particular
node in the cluster?

Partial state transfer is used to fetch only a subset of the state from a given member, e.g. if you have web sessions and don't want all of them. To do this, you also have to implement the corresponding callbacks in ExtendedMessageListener

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

From forum: JGroups - Dev

Re: Node started before Gossip = no new view after Gossip started

2009-11-19T06:26:36Z

Re: [javagroups-users] Node started before Gossip = no new view after Gossip started Thanks for the links, I’ve read:

JGRP-716> In a TCP based config *without* UNICAST, messages can get delivered out of order if a thread pool is enabled at the transport level

Incidentally, my config has thread_pool disabled:

<TCP loopback="false" discard_incompatible_packets="true" use_incoming_packet_handler="true" enable_bundling="false" enable_diagnostics="true" thread_naming_pattern="cl" use_concurrent_stack="false" thread_pool.enabled="false" thread_pool.min_threads="8" thread_pool.max_threads="40" thread_pool.keep_alive_time="5000" thread_pool.queue_enabled="false" thread_pool.queue_max_size="100" thread_pool.rejection_policy="Run" oob_thread_pool.enabled="false" oob_thread_pool.min_threads="8" oob_thread_pool.max_threads="20" oob_thread_pool.keep_alive_time="5000" oob_thread_pool.queue_enabled="false" oob_thread_pool.queue_max_size="100" oob_thread_pool.rejection_policy="Run"/>

Does this mean that in this case UNICAST is not needed after all?

Is it safe to enable thread_pool if I add UNICAST? I remember that I disabled thread_pool some time ago exactly because of ordering issues...

Thank you!
Linas

On 11/19/09 3:14 PM, "Bela Ban" <belaban@...> wrote:

Linas Virbalas wrote:
> Hello Bela,
>
> Could you please explain what do I loose if I don't have UNICAST in
> the mentioned stack (i.e. why should I add it):

Without UNICAST, ordering is destroyed:
http://www.jboss.org/community/wiki/TcpAndUnicast.

> <config>
> <TCP ... />
> <TCPGOSSIP initial_hosts="centosa.localdomain[12001]"
> gossip_refresh_rate="10000"
> num_initial_members="2" />
> <MERGE2 />
> <FD_SOCK />
> <FD timeout="3000" max_tries="10" shun="false" />
> <pbcast.NAKACK use_mcast_xmit="false"
> gc_lag="0"
> retransmit_timeout="50,75,100,300,600,2400,4800"
> discard_delivered_msgs="true"
> />
> <UNICAST timeout="300,600,1200" />
> <pbcast.STABLE stability_delay="500" desired_avg_gossip="1000"
> max_bytes="0"/>
> 
> <pbcast.GMS print_local_addr="true"
> join_timeout="20000"
> shun="false"/>
> 
> </config>
>
> I understand that JGroups stands here:
>
> Unreliable Reliable
> Unicast UDP TCP
> Multicast IP Multicast JGroups
>
> Also, that for reliable multicast we need to define NAKACK protocol
> and, as documentation states, that UNICAST defines a reliable unicast
> layer. But here I'm confused:
>
> 1. As this protocol stack is on top of TCP, which is already reliable,
> why do we need to define UNICAST in addition?
> 2. And when does JGroups use unicast instead of multicast?
>
> Please excuse me if these are too simple questions, but your
> clarification is appreciated.

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

From forum: JGroups - General

Re: Configuration without multicast

2009-11-19T06:04:56Z

It was the jgroups.bind_addr property. It works now. Thaks. I should
had seen that by myself, it is in the tcp.xml configuration file's comments.

Bela Ban wrote:

> Did you replace TCPPING.initial_hosts with your real host names ?
>
> Also, set a bind_addr in TCP, or set system prop jgroups.bind_addr
>
> pchacin wrote:
>> I Did. Same problem. If I switch to default UPD configuration in a
>> network with multicast, it works, so I don't think it is an application
>> problem, but is there some requirement like starting first the initial
>> host(s) and then, after a delay, start the rest? Maybe all the nodes are
>> starting too fast?
>>
>>
>> Bela Ban wrote:
>>
>>> The stack in the manual was just an example; use tcp.xml shipped with
>>> JGroups
>>>
>>> pchacin wrote:
>>>
>>>> Hi all
>>>>
>>>> I'm still having problems to configure Jgropus in a local network
>>>> without ip multicast. The scenario is simple: I'm making some running
>>>> multiple instances of the same application in my own machine. I can't
>>>> figure out how to make TCPPING work. The problem is that the
>>>> instances
>>>> can't see each other (their view don't have any other member of the
>>>> group).
>>>>
>>>> I'm using the configuration example given at
>>>> http://www.jgroups.org/manual/html/user-advanced.html, which I copy
>>>> below.
>>>>
>>>> I'll appreciate any help. Thanks in advance.
>>>>
>>>>
>>>> <config xmlns="urn:org:jgroups"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="urn:org:jgroups
>>>> http://www.jgroups.org/schema/JGroups-2.8.xsd">
>>>> <TCP bind_port="7800" />
>>>> <TCPPING timeout="3000"
>>>> initial_hosts="localhost[7800],localhost[7801]"
>>>> port_range="10"
>>>> num_initial_members="3"/>
>>>> <VERIFY_SUSPECT timeout="1500" />
>>>> <pbcast.NAKACK
>>>> use_mcast_xmit="false" gc_lag="0"
>>>> retransmit_timeout="300,600,1200,2400,4800"
>>>> discard_delivered_msgs="true"/>
>>>> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
>>>> max_bytes="400000"/>
>>>> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>>>> shun="true"
>>>> view_bundling="true"/>
>>>> </config>
>>>>
>>>>
>>
>>
>>
>

--
========================================================================
0 0 0 Pablo Chacin | Dept. d'Arquitectura de Computadors
0 0 0 e-mail: pchacin@... | UPC-Campus Nord. Modul D6-212
0 0 0 phone: +34 +93 405 4059 | Jordi Girona, 1-3
U P C fax: +34 +93 401 7055 | 08034 Barcelona - SPAIN
www: http://personals.ac.upc.es/pchacin
========================================================================

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: Node started before Gossip = no new view after Gossip started

2009-11-19T05:14:24Z

Linas Virbalas wrote:
> Hello Bela,
>
> Could you please explain what do I loose if I don't have UNICAST in
> the mentioned stack (i.e. why should I add it):

Without UNICAST, ordering is destroyed:
http://www.jboss.org/community/wiki/TcpAndUnicast.

> <config>
> <TCP ... />
> <TCPGOSSIP initial_hosts="centosa.localdomain[12001]"
> gossip_refresh_rate="10000"
> num_initial_members="2" />
> <MERGE2 />
> <FD_SOCK />
> <FD timeout="3000" max_tries="10" shun="false" />
> <pbcast.NAKACK use_mcast_xmit="false"
> gc_lag="0"
> retransmit_timeout="50,75,100,300,600,2400,4800"
> discard_delivered_msgs="true"
> />
> <UNICAST timeout="300,600,1200" />
> <pbcast.STABLE stability_delay="500" desired_avg_gossip="1000"
> max_bytes="0"/>
> 
> <pbcast.GMS print_local_addr="true"
> join_timeout="20000"
> shun="false"/>
> 
> </config>
>
> I understand that JGroups stands here:
>
> Unreliable Reliable
> Unicast UDP TCP
> Multicast IP Multicast JGroups
>
> Also, that for reliable multicast we need to define NAKACK protocol
> and, as documentation states, that UNICAST defines a reliable unicast
> layer. But here I'm confused:
>
> 1. As this protocol stack is on top of TCP, which is already reliable,
> why do we need to define UNICAST in addition?
> 2. And when does JGroups use unicast instead of multicast?
>
> Please excuse me if these are too simple questions, but your
> clarification is appreciated.

From forum: JGroups - General

Re: [JGRP-981] concurrent startup problem in 2.8

2009-11-19T05:08:18Z

Ok, your problem is the following:

* Your OOB pool has 1 min thread, a queue of 1000 (enabled) and a
rejection policy of "abort"
o This means that only 1 OOB thread will ever run until the
queue is full, then additional threads will be created
* Now when there is a merge, every member gets a MERGE request. This
is sent as OOB
* When a MERGE request is received, we use up the 1 OOB thread from
the pool
* The receiver (in the process of handling this request) multicasts
another OOB request out to all cluster members (to fetch the digests)
* This request gets added to the queue (because the queue is not
full and the 1 OOB thread is already used)
* The latter request will never be handled, so the member times out
and returns an empty digest
* The result is that the MergeView will be installed, but *not* the
merge *digest* !

The fix is simple: either disable the OOB queue (preferred), or set
min_threads to more than 1 (not preferred), or set the rejection_policy
to "run".

Had you actually used udp.xml or tcp.xml from the 2.8 distro, you would
not have run into this issue !

BTW: I changed some code, so we return at least our own digest, so this
issue will not occur even with your config. This is in the latest code
in CVS head.

Victor wrote:

> Hello Bela,
>
> I am playing with JGroups 2.8 RC6 in our test cluster (and going to
> move to production soon).
> The cluster consists of 2 nodes running on the same physical host -
> node1 and node2.
> Config file is mentioned in the end of this email; it is TCP-based,
> almost like tcp.xml shipped with JGroups.
>
> One time I started both nodes concurrently, JChannel was created
> almost at the same time. The result is that the nodes see each other
> (according to View), they were sending some messages each other, but
> my own messages were completely dropped or ignored. I mean messages
> sent via JChannel.send()
>
> This is how it looks in log files on both nodes (you can compare
> timestamps because it is on the same computer!):
>
> -----------------------------------------------------
> node1 log
>
> 18/11/2009 11:56:21| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.start(): Connecting channel...
> 18/11/2009 11:56:21| INFO | org.jgroups.JChannel.init(): JGroups
> version: 2.8.0.CR6
> 18/11/2009 11:56:25| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.viewAccepted(): Cluster
> view: [node1|0] [node1]
> 18/11/2009 11:56:25| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.channelConnected(): Channel
> connected: org.jgroups.JChannel@28294f62
> 18/11/2009 11:56:30| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=[node1|0] [node1]
> 18/11/2009 11:56:30| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=[node1|0] [node1]
> 18/11/2009 11:56:32| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=[node1|0] [node1]
> 18/11/2009 11:56:33| DEBUG |
> myserver.cluster.JGroupsClusterManagerImpl.sendMessageToAllGates():
> Sending message: Kill 4E266D67D697_D7E7C7687F66
> 18/11/2009 11:56:33| DEBUG |
> myserver.cluster.JGroupsClusterManagerImpl.doSendMessageToAllGates():
> Message sent: Kill 4E266D67D697_D7E7C7687F66
> 18/11/2009 11:56:46| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=[node1|0] [node1]
> 18/11/2009 11:56:48| WARN |
> org.jgroups.protocols.pbcast.GMS.castViewChangeWithDest(): node1:
> failed to collect all ACKs (expected=1) for view MergeView::[node1|1]
> [node1, node2], subgroups=[[node2|0] [node2], [node1|0] [node1]] after
> 2000ms, missing ACKs from [node1]
> 18/11/2009 11:56:48| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.viewAccepted(): Cluster
> view: MergeView::[node1|1] [node1, node2], subgroups=[[node2|0]
> [node2], [node1|0] [node1]]
>
> 18/11/2009 11:56:48| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=MergeView::[node1|1] [node1, node2], subgroups=[[node2|0]
> [node2], [node1|0] [node1]]
>
>
> -----------------------------------------------------
> node2 log
>
> 18/11/2009 11:56:19| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.start(): Connecting channel...
> 18/11/2009 11:56:19| INFO | org.jgroups.JChannel.init(): JGroups
> version: 2.8.0.CR6
> 18/11/2009 11:56:20| ERROR | org.jgroups.protocols.TP.down(): failed
> sending message to 87.106.188.148:40002 (136 bytes):
> java.net.ConnectException: Connection refused
> 18/11/2009 11:56:23| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.viewAccepted(): Cluster
> view: [node2|0] [node2]
>
> 18/11/2009 11:56:23| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.channelConnected(): Channel
> connected: org.jgroups.JChannel@6460029d
> 18/11/2009 11:56:31| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node2: dropped
> message from node1 (not in retransmission table), members are [node2],
> view=[node2|0] [node2]
> 18/11/2009 11:56:32| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node2: dropped
> message from node1 (not in retransmission table), members are [node2],
> view=[node2|0] [node2]
> 18/11/2009 11:56:32| DEBUG |
> myserver.cluster.JGroupsClusterManagerImpl.sendMessageToAllGates():
> Sending message: Kill 327B3A2F717C_6B636B186C62
> 18/11/2009 11:56:32| DEBUG |
> myserver.cluster.JGroupsClusterManagerImpl.doSendMessageToAllGates():
> Message sent: Kill 327B3A2F717C_6B636B186C62
> 18/11/2009 11:56:33| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node2: dropped
> message from node1 (not in retransmission table), members are [node2],
> view=[node2|0] [node2]
> 18/11/2009 11:56:46| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node2: dropped
> message from node1 (not in retransmission table), members are [node2],
> view=[node2|0] [node2]
> 18/11/2009 11:56:48| WARN |
> org.jgroups.protocols.pbcast.GMS.castViewChangeWithDest(): node2:
> failed to collect all ACKs (expected=1) for view MergeView::[node1|1]
> [node1, node2], subgroups=[[node2|0] [node2], [node1|0] [node1]] after
> 2000ms, missing ACKs from [node2]
> 18/11/2009 11:56:48| INFO |
> myserver.cluster.JGroupsClusterManagerImpl.viewAccepted(): Cluster
> view: MergeView::[node1|1] [node1, node2], subgroups=[[node2|0]
> [node2], [node1|0] [node1]]
>
> 18/11/2009 11:56:48| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node2: dropped
> message from node1 (not in retransmission table), members are [node2],
> view=MergeView::[node1|1] [node1, node2], subgroups=[[node2|0]
> [node2], [node1|0] [node1]]
> -----------------------------------------------------
>
>
> We can see "failed to collect all ACKs" here and also "dropped
> message". I did not really understand what happenned... I tried to
> wait up to 30 min but without success.
>
>
> After that I turned on more detailed logging and tried to send my
> messages again:
>
> -----------------------------------------------------
> node1 log
>
> 18/11/2009 13:03:05| TRACE | org.jgroups.protocols.TP.passMessageUp():
> message is [dst: <null>, src: node2 (3 headers), size=0 bytes,
> flags=OOB], headers are STABLE: [STABLE_GOSSIP]: digest is node2: [0 :
> 52 (52)], NAKACK: [MSG, seqno=53], TCP: [channel_name=GateCluster]
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): [node1: received
> node2#53
> 18/11/2009 13:03:05| WARN |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): node1: dropped
> message from node2 (not in retransmission table), members are [node1],
> view=MergeView::[node1|1] [node1, node2], subgroups=[[node2|0]
> [node2], [node1|0] [node1]]
>
> -----------------------------------------------------
> node2 log
>
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.pbcast.STABLE$StableTask.run(): node2: setting
> latest_local_digest from NAKACK: [node2#52]
>
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.pbcast.STABLE.sendStableMessage(): node2:
> sending stable msg [node2#52]
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.pbcast.NAKACK.send(): sending node2#53
> 18/11/2009 13:03:05| TRACE | org.jgroups.protocols.TP.down(): sending
> msg to null, src=node2, headers are STABLE: [STABLE_GOSSIP]: digest is
> node2: [0 : 52 (52)], NAKACK: [MSG, seqno=53], TCP:
> [channel_name=GateCluster]
> 18/11/2009 13:03:05| TRACE | org.jgroups.protocols.TP.down(): looping
> back message [dst: <null>, src: node2 (3 headers), size=0 bytes,
> flags=OOB]
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.BasicTCP.sendUnicast(): dest=192.168.1.102:40002
> (131 bytes)
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.BasicTCP.sendUnicast(): dest=192.168.1.101:40002
> (131 bytes)
> 18/11/2009 13:03:05| TRACE | org.jgroups.protocols.TP.passMessageUp():
> message is [dst: <null>, src: node2 (3 headers), size=0 bytes,
> flags=OOB], headers are STABLE: [STABLE_GOSSIP]: digest is node2: [0 :
> 52 (52)], NAKACK: [MSG, seqno=53], TCP: [channel_name=GateCluster]
> 18/11/2009 13:03:05| TRACE |
> org.jgroups.protocols.pbcast.NAKACK.handleMessage(): [node2: received
> node2#53
> 18/11/2009 13:03:05| TRACE | org.jgroups.protocols.TP.passMessageUp():
> message is [dst: <null>, src: node2 (3 headers), size=0 bytes,
> flags=OOB], headers are STABLE: [STABLE_GOSSIP]: digest is node2: [0 :
> 52 (52)], NAKACK: [MSG, seqno=53], TCP: [channel_name=GateCluster]
> -----------------------------------------------------
>
> We see "dropped message" again and a strange View:
> MergeView::[node1|1] [node1, node2], subgroups=[[node2|0] [node2],
> [node1|0] [node1]]
>
>
> After googling, I have found 2 bugs in Jira:
> https://jira.jboss.org/jira/browse/JGRP-981 (Wrong view with
> concurrent startup)
> https://jira.jboss.org/jira/browse/JGRP-14 (TCP protocol stack won't
> merge view with initial hosts specified as IP addresses)
>
> Is this the same problem as in 981?
> In 981 you said that it could be fixed with VIEW_SYNC, but it seems to
> be deprecated in 2.8? Is this problem fixed or you just recommend to
> not start cluster nodes at the same time?
>
>
> As for xml config, it is here:
> -----------------------------------------------------
> <config xmlns="urn:org:jgroups"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
> <TCP bind_addr="${bindAddress}"
> bind_port="40002"
> loopback="true"
> recv_buf_size="20000000"
> send_buf_size="640000"
> discard_incompatible_packets="true"
> max_bundle_size="64000"
> max_bundle_timeout="30"
> enable_bundling="true"
> use_send_queues="true"
> sock_conn_timeout="300"
> skip_suspected_members="true"
>
> thread_pool.enabled="true"
> thread_pool.min_threads="1"
> thread_pool.max_threads="25"
> thread_pool.keep_alive_time="5000"
> thread_pool.queue_enabled="true"
> thread_pool.queue_max_size="10000"
> thread_pool.rejection_policy="abort"
>
> oob_thread_pool.enabled="true"
> oob_thread_pool.min_threads="1"
> oob_thread_pool.max_threads="8"
> oob_thread_pool.keep_alive_time="5000"
> oob_thread_pool.queue_enabled="true"
> oob_thread_pool.queue_max_size="1000"
> oob_thread_pool.rejection_policy="abort"/>
>
> <TCPPING timeout="3000"
> initial_hosts="${nodes}"
> port_range="1"
> num_initial_members="3"/>
> <MERGE2 min_interval="10000"
> max_interval="30000"/>
> <FD_SOCK/>
> <FD timeout="10000" max_tries="5" />
> <VERIFY_SUSPECT timeout="1500" />
> <BARRIER />
> <pbcast.NAKACK
> use_mcast_xmit="false" gc_lag="0"
> retransmit_timeout="300,600,1200,2400,4800"
> discard_delivered_msgs="true"/>
> <UNICAST timeout="300,600,1200" />
> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="400000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>
> view_bundling="true"/>
> <FC max_credits="2000000"
> min_threshold="0.10"/>
> <FRAG2 frag_size="60000" />
> </config>
> -----------------------------------------------------
>
>
> Thanks,
>
> Victor
>

From forum: JGroups - General

Re: Node started before Gossip = no new view after Gossip started

2009-11-19T04:02:53Z

Re: [javagroups-users] Node started before Gossip = no new view after Gossip started Hello Bela,

Could you please explain what do I loose if I don’t have UNICAST in the mentioned stack (i.e. why should I add it):

<config>
    <TCP ... />
    <TCPGOSSIP initial_hosts="centosa.localdomain[12001]"
         gossip_refresh_rate="10000"
         num_initial_members="2" />
    <MERGE2 />
    <FD_SOCK />
    <FD timeout="3000" max_tries="10" shun="false" />
    <pbcast.NAKACK use_mcast_xmit="false"
                   gc_lag="0"
                   retransmit_timeout="50,75,100,300,600,2400,4800"
                   discard_delivered_msgs="true"
                   />
    <UNICAST timeout="300,600,1200" />
    <pbcast.STABLE stability_delay="500" desired_avg_gossip="1000"
                   max_bytes="0"/>
    
    <pbcast.GMS print_local_addr="true"
                join_timeout="20000"
                shun="false"/>
    
</config>

I understand that JGroups stands here:

          Unreliable     Reliable
Unicast    UDP            TCP
Multicast IP Multicast   JGroups

Also, that for reliable multicast we need to define NAKACK protocol and, as documentation states, that UNICAST defines a reliable unicast layer. But here I’m confused:

As this protocol stack is on top of TCP, which is already reliable, why do we need to define UNICAST in addition?
And when does JGroups use unicast instead of multicast?

Please excuse me if these are too simple questions, but your clarification is appreciated.

Thanks,
Linas

On 11/9/09 6:50 PM, "Bela Ban" <belaban@...> wrote:

#3 Your XML config file looks weird (no UNICAST, concurrent stack
disabled etc); I suggest start with tcpgossip.xml from your distro
(2.6.10.GA).

From forum: JGroups - General

Re: Configuration without multicast

2009-11-19T03:05:02Z

Did you replace TCPPING.initial_hosts with your real host names ?

Also, set a bind_addr in TCP, or set system prop jgroups.bind_addr

pchacin wrote:

> I Did. Same problem. If I switch to default UPD configuration in a
> network with multicast, it works, so I don't think it is an application
> problem, but is there some requirement like starting first the initial
> host(s) and then, after a delay, start the rest? Maybe all the nodes are
> starting too fast?
>
>
> Bela Ban wrote:
>
>> The stack in the manual was just an example; use tcp.xml shipped with
>> JGroups
>>
>> pchacin wrote:
>>
>>> Hi all
>>>
>>> I'm still having problems to configure Jgropus in a local network
>>> without ip multicast. The scenario is simple: I'm making some running
>>> multiple instances of the same application in my own machine. I can't
>>> figure out how to make TCPPING work. The problem is that the instances
>>> can't see each other (their view don't have any other member of the
>>> group).
>>>
>>> I'm using the configuration example given at
>>> http://www.jgroups.org/manual/html/user-advanced.html, which I copy
>>> below.
>>>
>>> I'll appreciate any help. Thanks in advance.
>>>
>>>
>>> <config xmlns="urn:org:jgroups"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="urn:org:jgroups
>>> http://www.jgroups.org/schema/JGroups-2.8.xsd">
>>> <TCP bind_port="7800" />
>>> <TCPPING timeout="3000"
>>> initial_hosts="localhost[7800],localhost[7801]"
>>> port_range="10"
>>> num_initial_members="3"/>
>>> <VERIFY_SUSPECT timeout="1500" />
>>> <pbcast.NAKACK
>>> use_mcast_xmit="false" gc_lag="0"
>>> retransmit_timeout="300,600,1200,2400,4800"
>>> discard_delivered_msgs="true"/>
>>> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
>>> max_bytes="400000"/>
>>> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>>> shun="true"
>>> view_bundling="true"/>
>>> </config>
>>>
>>>
>>>
>
>
>

From forum: JGroups - General

Re: Configuration without multicast

2009-11-19T03:00:33Z

I Did. Same problem. If I switch to default UPD configuration in a
network with multicast, it works, so I don't think it is an application
problem, but is there some requirement like starting first the initial
host(s) and then, after a delay, start the rest? Maybe all the nodes are
starting too fast?

Bela Ban wrote:

> The stack in the manual was just an example; use tcp.xml shipped with
> JGroups
>
> pchacin wrote:
>> Hi all
>>
>> I'm still having problems to configure Jgropus in a local network
>> without ip multicast. The scenario is simple: I'm making some running
>> multiple instances of the same application in my own machine. I can't
>> figure out how to make TCPPING work. The problem is that the instances
>> can't see each other (their view don't have any other member of the
>> group).
>>
>> I'm using the configuration example given at
>> http://www.jgroups.org/manual/html/user-advanced.html, which I copy
>> below.
>>
>> I'll appreciate any help. Thanks in advance.
>>
>>
>> <config xmlns="urn:org:jgroups"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="urn:org:jgroups
>> http://www.jgroups.org/schema/JGroups-2.8.xsd">
>> <TCP bind_port="7800" />
>> <TCPPING timeout="3000"
>> initial_hosts="localhost[7800],localhost[7801]"
>> port_range="10"
>> num_initial_members="3"/>
>> <VERIFY_SUSPECT timeout="1500" />
>> <pbcast.NAKACK
>> use_mcast_xmit="false" gc_lag="0"
>> retransmit_timeout="300,600,1200,2400,4800"
>> discard_delivered_msgs="true"/>
>> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
>> max_bytes="400000"/>
>> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>> shun="true"
>> view_bundling="true"/>
>> </config>
>>
>>
>

From forum: JGroups - General

Re: Configuration without multicast

2009-11-19T02:50:43Z

The stack in the manual was just an example; use tcp.xml shipped with
JGroups

pchacin wrote:

> Hi all
>
> I'm still having problems to configure Jgropus in a local network
> without ip multicast. The scenario is simple: I'm making some running
> multiple instances of the same application in my own machine. I can't
> figure out how to make TCPPING work. The problem is that the instances
> can't see each other (their view don't have any other member of the group).
>
> I'm using the configuration example given at
> http://www.jgroups.org/manual/html/user-advanced.html, which I copy below.
>
> I'll appreciate any help. Thanks in advance.
>
>
> <config xmlns="urn:org:jgroups"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:org:jgroups
> http://www.jgroups.org/schema/JGroups-2.8.xsd">
> <TCP bind_port="7800" />
> <TCPPING timeout="3000"
> initial_hosts="localhost[7800],localhost[7801]"
> port_range="10"
> num_initial_members="3"/>
> <VERIFY_SUSPECT timeout="1500" />
> <pbcast.NAKACK
> use_mcast_xmit="false" gc_lag="0"
> retransmit_timeout="300,600,1200,2400,4800"
> discard_delivered_msgs="true"/>
> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="400000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="3000"
> shun="true"
> view_bundling="true"/>
> </config>
>
>

From forum: JGroups - General

Configuration without multicast

2009-11-19T01:29:58Z

Hi all

I'm still having problems to configure Jgropus in a local network
without ip multicast. The scenario is simple: I'm making some running
multiple instances of the same application in my own machine. I can't
figure out how to make TCPPING work. The problem is that the instances
can't see each other (their view don't have any other member of the group).

I'm using the configuration example given at
http://www.jgroups.org/manual/html/user-advanced.html, which I copy below.

I'll appreciate any help. Thanks in advance.

<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-2.8.xsd">
<TCP bind_port="7800" />
<TCPPING timeout="3000"
initial_hosts="localhost[7800],localhost[7801]"
port_range="10"
num_initial_members="3"/>
<VERIFY_SUSPECT timeout="1500" />
<pbcast.NAKACK
use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"
shun="true"
view_bundling="true"/>
</config>

--
========================================================================
0 0 0 Pablo Chacin | Dept. d'Arquitectura de Computadors
0 0 0 e-mail: pchacin@... | UPC-Campus Nord. Modul D6-212
0 0 0 phone: +34 +93 405 4059 | Jordi Girona, 1-3
U P C fax: +34 +93 401 7055 | 08034 Barcelona - SPAIN
www: http://personals.ac.upc.es/pchacin
========================================================================

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
javagroups-users mailing list
javagroups-users@...
https://lists.sourceforge.net/lists/listinfo/javagroups-users

From forum: JGroups - General

Re: [jgoups-dev] Partial state from individual nodes

2009-11-18T23:19:12Z

Adi wrote:
> Hi all,
>
> I am checking out Jgroups and it looks great for supporting my clustering
> needs. I had a basic question on sharing state when nodes join a cluster.
>
> I read the getState() documentation and it claims, it gets state from
> one of
> the nodes (usually the oldest).

Yes. Usually, you pass null as first arg of getState(), that way the
state is fetched from the coordinator (oldest cluster node).

> I also read about paritial states using a String.
>
> Can we do something like getState(nodeID) and get the state of a
> particular
> node in the cluster?

Partial state transfer is used to fetch only a subset of the state from
a given member, e.g. if you have web sessions and don't want all of
them. To do this, you also have to implement the corresponding callbacks
in ExtendedMessageListener

--
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Javagroups-development mailing list

From forum: JGroups - Dev