Server discards message from client on same mcast addr and ports

[boltimuss]

Hello everyone, I have been struggling with this proble for over a day now and am at my wit's end. I normally try to avoid forums and figure things out on my own, but this time I'm stumped. I've searched and searched until Google told me, "no more searches.. please!" Here is my setup, I'm running a JBOSS server on a windowsXP machine. Inside of JBOSS, I have a deployed service (mBean) that sends out a signal using JGroups to say that it is alive, a heartbeat. Now here is the problem. When I run a client on the same machine, the client sees the "heartbeat" from the server on the mCast channel. If I run the same client on a different computer running RedHat Linux it doesn't work. Instead I get the discarded message error. This is what I see from the server screen:

"15:15:01,072 WARN  [NAKACK] traj4xp-49622: discarded message from non-member cts7-45710, my view is [traj4xp-49622|0] [traj4xp-49622]" where traj4xp-49622 is the computer running the JBOSS server. and cts7-45710 is the linux computer.

The wierd thing is that they see each other but the server somehow believes that it is not part of the group. I tried the org.jgroups.demos.Draw and it works on the windowsXP machine and the Linux machine. I can draw in the Draw program with one window open on the Linux machine and the other on the Windows machine. I can even get the clients to see each other and braodcast to each other. It's just that I cannot get the JBOSS server mBean (using JGroups) running on windows to talk sucessfully to the client on the linux box.

Any help would be so very uch apreciated ...


oh yeah, almost forgot, here is the UDP.xl file I'm using


<!--
  Default stack using IP multicasting. It is similar to the "udp"
  stack in stacks.xml, but doesn't use streaming state transfer and flushing
  author: Bela Ban
  version: $Id: udp.xml,v 1.33 2009/08/28 09:04:07 belaban Exp $
-->

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
    <UDP
         bind_addr="139.169.8.194"
         bind_port="3333"
         mcast_addr="${jgroups.udp.mcast_addr:224.0.0.11}"
         mcast_port="${jgroups.udp.mcast_port:61234}"
         receive_on_all_interfaces="true"
         tos="8"
         ucast_recv_buf_size="20000000"
         ucast_send_buf_size="640000"
         mcast_recv_buf_size="25000000"
         mcast_send_buf_size="640000"
         loopback="true"
         discard_incompatible_packets="false"
         max_bundle_size="64000"
         max_bundle_timeout="30"
         ip_ttl="${jgroups.udp.ip_ttl:32}"
         enable_bundling="true"
         enable_diagnostics="true"
         thread_naming_pattern="cl"

         thread_pool.enabled="true"
         thread_pool.min_threads="2"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
         thread_pool.queue_enabled="true"
         thread_pool.queue_max_size="10000"
         thread_pool.rejection_policy="discard"

         oob_thread_pool.enabled="true"
         oob_thread_pool.min_threads="1"
         oob_thread_pool.max_threads="8"
         oob_thread_pool.keep_alive_time="5000"
         oob_thread_pool.queue_enabled="false"
         oob_thread_pool.queue_max_size="100"
         oob_thread_pool.rejection_policy="Run"/>

    <PING timeout="2000"
            num_initial_members="32"/>
    <MERGE2 max_interval="30000"
            min_interval="10000"/>
    <FD_ALL msg_counts_as_heartbeat="false"/>
    <FD/>
    <VERIFY_SUSPECT timeout="1500"  />
    <BARRIER />
    <pbcast.NAKACK use_stats_for_retransmission="true "
                   exponential_backoff="0"
                   use_mcast_xmit="true" gc_lag="0"
                   retransmit_timeout="50,300,600,1200"
                   discard_delivered_msgs="false"/>
    <UNICAST timeout="300,600,1200"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="1000000"/>
    <pbcast.GMS print_local_addr="true" join_timeout="3000"

                view_bundling="true"/>
    <FC max_credits="500000"
                    min_threshold="0.20"/>
    <FRAG2 frag_size="60000"  />
    <!--pbcast.STREAMING_STATE_TRANSFER /-->
    <pbcast.STATE_TRANSFER  />
    <pbcast.FLUSH  />
</config>

[Bela Ban]

A few things to look at:

    * Do you have firewalls enabled on the Linux box (iptables -L) ? If
      so, turn em off
    * mcast_addr of 224.x.x.x are not recommended, as a matter of fact,
      Cisco routers might discard them. Use 239.5.5.5 instead, for example
          o Do the Windows and Linux box hang off of the same switch ?
          o If they are in separate subnets, make sure ip_ttl is greater
            than 2
    * Are you using exactly the same udp.xml in both the JBoss server
      and the Linux client ?
          o Make sure that no *different* udp.xml is picked up somewhere
            in the classpath on the JBoss server
          o As a matter of fact, either use an absolute path for the XML
            config (e.g. /home/bela/config.xml), or name your config
            file unique_conf.xml or something so that this is avoided
    * I assume bind_addr is different on the Linux and Windows boxes,
      right ?
    * Make sure mcast_addr and mcast_port are the same on the Windows
      and Linux box
    * Check whether there is a routing table entry for multicasts on
      Linux and Windows (netstat -nr)
          o Either the default interface is used, or if there is a
            224.0.0.0 network, it will be used
    * Run the test program shipped with JGroups at [1] to detect whether
      multicasting is working
    * Use wireshark to look at the packets and see whether your NIC on
      the Linux box receive them


[1] http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork



boltimuss wrote:

> Hello everyone, I have been struggling with this proble for over a day now
> and am at my wit's end. I normally try to avoid forums and figure
> things out
> on my own, but this time I'm stumped. I've searched and searched until
> Google told me, "no more searches.. please!" Here is my setup, I'm
> running a
> JBOSS server on a windowsXP machine. Inside of JBOSS, I have a deployed
> service (mBean) that sends out a signal using JGroups to say that it is
> alive, a heartbeat. Now here is the problem. When I run a client on
> the same
> machine, the client sees the "heartbeat" from the server on the mCast
> channel. If I run the same client on a different computer running RedHat
> Linux it doesn't work. Instead I get the discarded message error. This is
> what I see from the server screen:
>
> "15:15:01,072 WARN [NAKACK] traj4xp-49622: discarded message from
> non-member cts7-45710, my view is [traj4xp-49622|0] [traj4xp-49622]" where
> traj4xp-49622 is the computer running the JBOSS server. and cts7-45710 is
> the linux computer.
>
> The wierd thing is that they see each other but the server somehow
> believes
> that it is not part of the group. I tried the org.jgroups.demos.Draw
> and it
> works on the windowsXP machine and the Linux machine. I can draw in
> the Draw
> program with one window open on the Linux machine and the other on the
> Windows machine. I can even get the clients to see each other and
> braodcast
> to each other. It's just that I cannot get the JBOSS server mBean (using
> JGroups) running on windows to talk sucessfully to the client on the linux
> box.
>
> Any help would be so very uch apreciated ...
>
>
> oh yeah, almost forgot, here is the UDP.xl file I'm using
>
>
> <!--
> Default stack using IP multicasting. It is similar to the "udp"
> stack in stacks.xml, but doesn't use streaming state transfer and flushing
> author: Bela Ban
> version: $Id: udp.xml,v 1.33 2009/08/28 09:04:07 belaban Exp $
> -->
>
> <config xmlns="urn:org:jgroups"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
> <UDP
> bind_addr="139.169.8.194"
> bind_port="3333"
> mcast_addr="${jgroups.udp.mcast_addr:224.0.0.11}"
> mcast_port="${jgroups.udp.mcast_port:61234}"
> receive_on_all_interfaces="true"
> tos="8"
> ucast_recv_buf_size="20000000"
> ucast_send_buf_size="640000"
> mcast_recv_buf_size="25000000"
> mcast_send_buf_size="640000"
> loopback="true"
> discard_incompatible_packets="false"
> max_bundle_size="64000"
> max_bundle_timeout="30"
> ip_ttl="${jgroups.udp.ip_ttl:32}"
> enable_bundling="true"
> enable_diagnostics="true"
> thread_naming_pattern="cl"
>
> thread_pool.enabled="true"
> thread_pool.min_threads="2"
> thread_pool.max_threads="8"
> thread_pool.keep_alive_time="5000"
> thread_pool.queue_enabled="true"
> thread_pool.queue_max_size="10000"
> thread_pool.rejection_policy="discard"
>
> oob_thread_pool.enabled="true"
> oob_thread_pool.min_threads="1"
> oob_thread_pool.max_threads="8"
> oob_thread_pool.keep_alive_time="5000"
> oob_thread_pool.queue_enabled="false"
> oob_thread_pool.queue_max_size="100"
> oob_thread_pool.rejection_policy="Run"/>
>
> <PING timeout="2000"
> num_initial_members="32"/>
> <MERGE2 max_interval="30000"
> min_interval="10000"/>
> <FD_ALL msg_counts_as_heartbeat="false"/>
> <FD/>
> <VERIFY_SUSPECT timeout="1500" />
> <BARRIER />
> <pbcast.NAKACK use_stats_for_retransmission="true "
> exponential_backoff="0"
> use_mcast_xmit="true" gc_lag="0"
> retransmit_timeout="50,300,600,1200"
> discard_delivered_msgs="false"/>
> <UNICAST timeout="300,600,1200"/>
> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="1000000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>
> view_bundling="true"/>
> <FC max_credits="500000"
> min_threshold="0.20"/>
> <FRAG2 frag_size="60000" />
> <!--pbcast.STREAMING_STATE_TRANSFER /-->
> <pbcast.STATE_TRANSFER />
> <pbcast.FLUSH />
> </config>
>

--
Bela Ban
Lead JGroups / Clustering Team
JBoss


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Javagroups-development mailing list

[boltimuss]

Bela!! so nice to hear from you. I've been reading nothing but your posts for the last several hours, I feel like I know you... hehe :) Really though, thx. I'll try all of those things and get back...

Bolt

Bela Ban wrote:

A few things to look at:

    * Do you have firewalls enabled on the Linux box (iptables -L) ? If
      so, turn em off
    * mcast_addr of 224.x.x.x are not recommended, as a matter of fact,
      Cisco routers might discard them. Use 239.5.5.5 instead, for example
          o Do the Windows and Linux box hang off of the same switch ?
          o If they are in separate subnets, make sure ip_ttl is greater
            than 2
    * Are you using exactly the same udp.xml in both the JBoss server
      and the Linux client ?
          o Make sure that no *different* udp.xml is picked up somewhere
            in the classpath on the JBoss server
          o As a matter of fact, either use an absolute path for the XML
            config (e.g. /home/bela/config.xml), or name your config
            file unique_conf.xml or something so that this is avoided
    * I assume bind_addr is different on the Linux and Windows boxes,
      right ?
    * Make sure mcast_addr and mcast_port are the same on the Windows
      and Linux box
    * Check whether there is a routing table entry for multicasts on
      Linux and Windows (netstat -nr)
          o Either the default interface is used, or if there is a
            224.0.0.0 network, it will be used
    * Run the test program shipped with JGroups at [1] to detect whether
      multicasting is working
    * Use wireshark to look at the packets and see whether your NIC on
      the Linux box receive them


[1] http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork



boltimuss wrote:
> Hello everyone, I have been struggling with this proble for over a day now
> and am at my wit's end. I normally try to avoid forums and figure
> things out
> on my own, but this time I'm stumped. I've searched and searched until
> Google told me, "no more searches.. please!" Here is my setup, I'm
> running a
> JBOSS server on a windowsXP machine. Inside of JBOSS, I have a deployed
> service (mBean) that sends out a signal using JGroups to say that it is
> alive, a heartbeat. Now here is the problem. When I run a client on
> the same
> machine, the client sees the "heartbeat" from the server on the mCast
> channel. If I run the same client on a different computer running RedHat
> Linux it doesn't work. Instead I get the discarded message error. This is
> what I see from the server screen:
>
> "15:15:01,072 WARN [NAKACK] traj4xp-49622: discarded message from
> non-member cts7-45710, my view is [traj4xp-49622|0] [traj4xp-49622]" where
> traj4xp-49622 is the computer running the JBOSS server. and cts7-45710 is
> the linux computer.
>
> The wierd thing is that they see each other but the server somehow
> believes
> that it is not part of the group. I tried the org.jgroups.demos.Draw
> and it
> works on the windowsXP machine and the Linux machine. I can draw in
> the Draw
> program with one window open on the Linux machine and the other on the
> Windows machine. I can even get the clients to see each other and
> braodcast
> to each other. It's just that I cannot get the JBOSS server mBean (using
> JGroups) running on windows to talk sucessfully to the client on the linux
> box.
>
> Any help would be so very uch apreciated ...
>
>
> oh yeah, almost forgot, here is the UDP.xl file I'm using
>
>
> <!--
> Default stack using IP multicasting. It is similar to the "udp"
> stack in stacks.xml, but doesn't use streaming state transfer and flushing
> author: Bela Ban
> version: $Id: udp.xml,v 1.33 2009/08/28 09:04:07 belaban Exp $
> -->
>
> <config xmlns="urn:org:jgroups"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
> <UDP
> bind_addr="139.169.8.194"
> bind_port="3333"
> mcast_addr="${jgroups.udp.mcast_addr:224.0.0.11}"
> mcast_port="${jgroups.udp.mcast_port:61234}"
> receive_on_all_interfaces="true"
> tos="8"
> ucast_recv_buf_size="20000000"
> ucast_send_buf_size="640000"
> mcast_recv_buf_size="25000000"
> mcast_send_buf_size="640000"
> loopback="true"
> discard_incompatible_packets="false"
> max_bundle_size="64000"
> max_bundle_timeout="30"
> ip_ttl="${jgroups.udp.ip_ttl:32}"
> enable_bundling="true"
> enable_diagnostics="true"
> thread_naming_pattern="cl"
>
> thread_pool.enabled="true"
> thread_pool.min_threads="2"
> thread_pool.max_threads="8"
> thread_pool.keep_alive_time="5000"
> thread_pool.queue_enabled="true"
> thread_pool.queue_max_size="10000"
> thread_pool.rejection_policy="discard"
>
> oob_thread_pool.enabled="true"
> oob_thread_pool.min_threads="1"
> oob_thread_pool.max_threads="8"
> oob_thread_pool.keep_alive_time="5000"
> oob_thread_pool.queue_enabled="false"
> oob_thread_pool.queue_max_size="100"
> oob_thread_pool.rejection_policy="Run"/>
>
> <PING timeout="2000"
> num_initial_members="32"/>
> <MERGE2 max_interval="30000"
> min_interval="10000"/>
> <FD_ALL msg_counts_as_heartbeat="false"/>
> <FD/>
> <VERIFY_SUSPECT timeout="1500" />
> <BARRIER />
> <pbcast.NAKACK use_stats_for_retransmission="true "
> exponential_backoff="0"
> use_mcast_xmit="true" gc_lag="0"
> retransmit_timeout="50,300,600,1200"
> discard_delivered_msgs="false"/>
> <UNICAST timeout="300,600,1200"/>
> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="1000000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="3000"
>
> view_bundling="true"/>
> <FC max_credits="500000"
> min_threshold="0.20"/>
> <FRAG2 frag_size="60000" />
> <!--pbcast.STREAMING_STATE_TRANSFER /-->
> <pbcast.STATE_TRANSFER />
> <pbcast.FLUSH />
> </config>
>

--
Bela Ban
Lead JGroups / Clustering Team
JBoss


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Javagroups-development mailing list

[Bela Ban]

boltimuss wrote:
> Bela!! so nice to hear from you. I've been reading nothing but your
> posts for the last several hours,

Have I written so many posts ? :-) LOL

> I feel like I know you... hehe :) Really though,
> thx. I'll try all of those things and get back...
OK

--
Bela Ban
Lead JGroups / Clustering Team
JBoss


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Javagroups-development mailing list

[boltimuss]

One other extra piece of information I wanted to add. When I run jGroups from within my mBean on JBOSS, GMS says that my physical address is 127.0.0.1:3333. But when I run my client on the same machine as the JBOSS server it's address comes up as 139.169.8.194 Also, when it is just the JBOSS mBean broadcasting on multicast and the client is running on another computer, the jgroup on my mBean discards the messages from the client, however if I start another client on the same computer as JBOSS then the views merge and JBOSS accepts the client on another computer. Does all of this have to do with the 127.0.0.1 IP?

Thx

Bolt

[boltimuss]

Okay, I figured it out. The problem was that although I set bind_addr in the xml file, I didn't set the VM options to -Dignore.bind.address=true. By setting this to true, this allowed the mBean residing in the running JBOSS server to bind to the correct ip and send multicast.

~Bolt