Reggie

View: New views
6 Messages — Rating Filter:   Alert me  

Reggie

by Per Edlund :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi!

We'ev been using Jini for quite som time now but havn't upgraded after the 2.1 release. We're currently experiencing some issues with reggie choking the machine and we can't really tell why. The number of jiniservices we have is around 60 (these are client's as well, and on top of this we have webservers which are clients (around 100-200)) , we use the LookupCache etc, but efery now and then we have to restart the VM running reggie. Doing a discovery during this time get's no lookup, and services can't be found.

If we look at how services register/unregister, that seam to take an aweful long time when reggie is starting to choke, and out theory is that reggie uses some kind of queue which grows more than it can be processed.

Does any one have an idea on how to dig into this?

Env: Linux Debian, Sun Java 1.5&1.6

Kind regards
/Per
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...

Re: Reggie

by Jeff Harmes-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Per,

  What I normally do is hookup a JVM Profiler to see if there is a memory leak.  Most are pretty good at showing you the heap usage and whether its growing and not being recovered.  The one I'm most familiar with is JProfiler - but that costs money.  But you could have a look at VisualVM ( https://visualvm.dev.java.net/index.html ).  In most cases the profiler comes with an agent jar that you have to put in either your classpath or bootclasspath.  Then you typically have to set a command line property to enable the profiler agent.  I took a quick peak at VisalVM and it looks like it uses the jstatd that is packaged with JDK 1.6 (http://java.sun.com/javase/6/docs/technotes/tools/share/jstatd.html), so with VisualVM you may not have to follow as many steps.  Probably just run jstatd on the machine that is Reggie is running on, then connect VisualVM to it.  The other option is to just specify -verbose:gc in the script that starts up Reggie and watch what verbose:gc has to report.  But you won't get as detailed information that you'd get from a Profiler.

Hope this helps?



Regards,


Jeff


-----Original Message-----
From: Per Edlund [mailto:per.edlund@...]
Sent: Wed 5/27/2009 8:17 AM
To: JINI-USERS@...
Subject: Reggie
 
Hi!

We'ev been using Jini for quite som time now but havn't upgraded after the
2.1 release. We're currently experiencing some issues with reggie choking
the machine and we can't really tell why. The number of jiniservices we have
is around 60 (these are client's as well, and on top of this we have
webservers which are clients (around 100-200)) , we use the LookupCache etc,
but efery now and then we have to restart the VM running reggie. Doing a
discovery during this time get's no lookup, and services can't be found.

If we look at how services register/unregister, that seam to take an aweful
long time when reggie is starting to choke, and out theory is that reggie
uses some kind of queue which grows more than it can be processed.

Does any one have an idea on how to dig into this?

Env: Linux Debian, Sun Java 1.5&1.6

Kind regards
/Per

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Re: Reggie

by Juan Ramirez :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi, Peter.  Reggie does queue up discovery requests and event
notifications and services these with threads from the same pool.  It is
possible that there is a large number of event notifications to be
delivered to the LookupCache instances thus delaying the processing of
discovery requests.  You could set Reggie's logging level to FINEST to
see what Reggie is doing at the time it appears to "choke".  There is
additional info on this scenario in the following CRs.

https://issues.apache.org/jira/browse/RIVER-52
https://issues.apache.org/jira/browse/RIVER-197

Hope this helps,

Juan

On 05/27/09 07:17, Per Edlund wrote:

> Hi!
>
> We'ev been using Jini for quite som time now but havn't upgraded after
> the 2.1 release. We're currently experiencing some issues with reggie
> choking the machine and we can't really tell why. The number of
> jiniservices we have is around 60 (these are client's as well, and on
> top of this we have webservers which are clients (around 100-200)) , we
> use the LookupCache etc, but efery now and then we have to restart the
> VM running reggie. Doing a discovery during this time get's no lookup,
> and services can't be found.
>
> If we look at how services register/unregister, that seam to take an
> aweful long time when reggie is starting to choke, and out theory is
> that reggie uses some kind of queue which grows more than it can be
> processed.
>
> Does any one have an idea on how to dig into this?
>
> Env: Linux Debian, Sun Java 1.5&1.6
>
> Kind regards
> /Per
> --------------------------------------------------------------------------
> Getting Started: http://www.jini.org/wiki/Category:Getting_Started 
> Community Web Site: http://jini.org jini-users Archive:
> http://archives.java.sun.com/archives/jini-users.html Unsubscribing:
> email "signoff JINI-USERS" to listserv@...

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to listserv@...

Re: Reggie

by Per Edlund :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi!

Thanks for the replys!

As it seams, this issue is related to client machines. If a client machine get's memoty issues, and reggie is trying to notify the client (which uses a servicechache listening on all service events), reggie chokes when the client can't reply to reggie.

Fixiing the memory issues on the client get's rid of the behaviour on reggie

Just thouth I should tell

Cheers
/Per

On Wed, May 27, 2009 at 3:47 PM, Juan Ramirez <Juan.Ramirez@...> wrote:
Hi, Peter.  Reggie does queue up discovery requests and event notifications and services these with threads from the same pool.  It is possible that there is a large number of event notifications to be delivered to the LookupCache instances thus delaying the processing of discovery requests.  You could set Reggie's logging level to FINEST to see what Reggie is doing at the time it appears to "choke".  There is additional info on this scenario in the following CRs.

https://issues.apache.org/jira/browse/RIVER-52
https://issues.apache.org/jira/browse/RIVER-197

Hope this helps,

Juan


On 05/27/09 07:17, Per Edlund wrote:
Hi!

We'ev been using Jini for quite som time now but havn't upgraded after the 2.1 release. We're currently experiencing some issues with reggie choking the machine and we can't really tell why. The number of jiniservices we have is around 60 (these are client's as well, and on top of this we have webservers which are clients (around 100-200)) , we use the LookupCache etc, but efery now and then we have to restart the VM running reggie. Doing a discovery during this time get's no lookup, and services can't be found.

If we look at how services register/unregister, that seam to take an aweful long time when reggie is starting to choke, and out theory is that reggie uses some kind of queue which grows more than it can be processed.

Does any one have an idea on how to dig into this?

Env: Linux Debian, Sun Java 1.5&1.6

Kind regards
/Per
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...




--
********************************************
Per Edlund
www.bozoka.com
per@...
+46(0)8 545 068 05
+46(0)703 71 70 02
********************************************
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...

Re: Reggie

by Gregg Wonderly :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message


On Jun 26, 2009, at 3:29 AM, Per Edlund wrote:
As it seams, this issue is related to client machines. If a client machine get's memoty issues, and reggie is trying to notify the client (which uses a servicechache listening on all service events), reggie chokes when the client can't reply to reggie.

Could this also be an issue for clients with firewalls up which can't accept connections for the notifications, and the notification connections timeout?  I've seen some strange things from time to time on more restricted networks and not really been able to pinpoint the issues.  Many times, it seems like discovery degenerates into a really slow and painful process.  An outbound call, initiating a TCP connection which must timeout before anything can be done to eliminate the call context, can be quite long lived, on the order of minutes.  I wonder if this would cause other discovery/notification to be delayed and thus build up more held context in the reggie JVM and elsewhere.

Gregg Wonderly

Fixiing the memory issues on the client get's rid of the behaviour on reggie

Just thouth I should tell

Cheers
/Per

On Wed, May 27, 2009 at 3:47 PM, Juan Ramirez <Juan.Ramirez@...> wrote:
Hi, Peter.  Reggie does queue up discovery requests and event notifications and services these with threads from the same pool.  It is possible that there is a large number of event notifications to be delivered to the LookupCache instances thus delaying the processing of discovery requests.  You could set Reggie's logging level to FINEST to see what Reggie is doing at the time it appears to "choke".  There is additional info on this scenario in the following CRs.

https://issues.apache.org/jira/browse/RIVER-52
https://issues.apache.org/jira/browse/RIVER-197

Hope this helps,

Juan


On 05/27/09 07:17, Per Edlund wrote:
Hi!

We'ev been using Jini for quite som time now but havn't upgraded after the 2.1 release. We're currently experiencing some issues with reggie choking the machine and we can't really tell why. The number of jiniservices we have is around 60 (these are client's as well, and on top of this we have webservers which are clients (around 100-200)) , we use the LookupCache etc, but efery now and then we have to restart the VM running reggie. Doing a discovery during this time get's no lookup, and services can't be found.

If we look at how services register/unregister, that seam to take an aweful long time when reggie is starting to choke, and out theory is that reggie uses some kind of queue which grows more than it can be processed.

Does any one have an idea on how to dig into this?

Env: Linux Debian, Sun Java 1.5&1.6

Kind regards
/Per
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...




--
********************************************
Per Edlund
www.bozoka.com
per@...
+46(0)8 545 068 05
+46(0)703 71 70 02
********************************************
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...

-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...

Re: Reggie

by Per Edlund :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Could be, don't really know. I don't have the abillity to test this out, but we now have more surveillance of the jvm status of the webapps and making manual restarts when memory becomes too close to max. After this we haven't had any issues at all.

It seamed like reggie ran out of thread's trying to notify (don't know how many it used), and this caused a big queue of jobs and in that q there where other jobs that would timeout as well witch led to new services wouldn't get the serviceId and stuff like that.

/Per

On Fri, Jun 26, 2009 at 2:39 PM, Gregg Wonderly <gregg@...> wrote:

On Jun 26, 2009, at 3:29 AM, Per Edlund wrote:
As it seams, this issue is related to client machines. If a client machine get's memoty issues, and reggie is trying to notify the client (which uses a servicechache listening on all service events), reggie chokes when the client can't reply to reggie.

Could this also be an issue for clients with firewalls up which can't accept connections for the notifications, and the notification connections timeout?  I've seen some strange things from time to time on more restricted networks and not really been able to pinpoint the issues.  Many times, it seems like discovery degenerates into a really slow and painful process.  An outbound call, initiating a TCP connection which must timeout before anything can be done to eliminate the call context, can be quite long lived, on the order of minutes.  I wonder if this would cause other discovery/notification to be delayed and thus build up more held context in the reggie JVM and elsewhere.

Gregg Wonderly

Fixiing the memory issues on the client get's rid of the behaviour on reggie

Just thouth I should tell

Cheers
/Per

On Wed, May 27, 2009 at 3:47 PM, Juan Ramirez <Juan.Ramirez@...> wrote:
Hi, Peter.  Reggie does queue up discovery requests and event notifications and services these with threads from the same pool.  It is possible that there is a large number of event notifications to be delivered to the LookupCache instances thus delaying the processing of discovery requests.  You could set Reggie's logging level to FINEST to see what Reggie is doing at the time it appears to "choke".  There is additional info on this scenario in the following CRs.

https://issues.apache.org/jira/browse/RIVER-52
https://issues.apache.org/jira/browse/RIVER-197

Hope this helps,

Juan


On 05/27/09 07:17, Per Edlund wrote:
Hi!

We'ev been using Jini for quite som time now but havn't upgraded after the 2.1 release. We're currently experiencing some issues with reggie choking the machine and we can't really tell why. The number of jiniservices we have is around 60 (these are client's as well, and on top of this we have webservers which are clients (around 100-200)) , we use the LookupCache etc, but efery now and then we have to restart the VM running reggie. Doing a discovery during this time get's no lookup, and services can't be found.

If we look at how services register/unregister, that seam to take an aweful long time when reggie is starting to choke, and out theory is that reggie uses some kind of queue which grows more than it can be processed.

Does any one have an idea on how to dig into this?

Env: Linux Debian, Sun Java 1.5&1.6

Kind regards
/Per
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...




--
********************************************
Per Edlund
www.bozoka.com
per@...
+46(0)8 545 068 05
+46(0)703 71 70 02
********************************************
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...




--
********************************************

Per Edlund Bozoka.com, per@...
Web:     www.bozoka.com
Office: +46(0)8 545 068 05
Mobile:+46(0)703 71 70 02
Address:
 Tegnérgatan 28, 2tr
 113 59 Stockholm
 Sweden
********************************************
-------------------------------------------------------------------------- Getting Started: http://www.jini.org/wiki/Category:Getting_Started Community Web Site: http://jini.org jini-users Archive: http://archives.java.sun.com/archives/jini-users.html Unsubscribing: email "signoff JINI-USERS" to listserv@...