Server stops responding due to Glassfish

[glassfish-2]

Ryan,

I've reconfigured one of my production servers as you suggested. I'll post to this thread when I have more information.

Kevin
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272247

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Ryan de Laplante]

FYI, the original JVM option I was given was missing a   ".connector."  
part in it.  Grizzly is still running on our system.   Here's the
correct JVM option:

-Dcom.sun.enterprise.web.connector.useCoyoteConnector=true


Ryan

glassfish@... wrote:
> Ryan,
>
> I've reconfigured one of my production servers as you suggested. I'll post to this thread when I have more information.
>
> Kevin
>
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

I corrected the JVM option as you noted.

Assuming that Grizzly is the problem, is there a downside to using Coyote? Also, I don't see log entries specifically that say Coyote is being used instead of Grizzly. Should I expect such?
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272320

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Ryan de Laplante]

glassfish@... wrote:
> I corrected the JVM option as you noted.
>
> Assuming that Grizzly is the problem, is there a downside to using Coyote? Also, I don't see log entries specifically that say Coyote is being used instead of Grizzly. Should I expect such?
>  
Run this command at the command prompt, then search the file for the
word grizzly:

asadmin generate-jvm-report --type=thread > threaddump.txt


I can't touch my production server during the day unless it goes down,
so I'm not running coyote yet. I don't know how to tell if it is
running, that's just a guess.  Here is what Jeanfrancois said about
Coyote earlier in this thread:

> You gonna face the same problem I suspect, or some requests will be
> dropped by Coyote if all the thread gets dead locked like It looks
> like. The difference between Grizzly & Coyote is Coyote drops requests
> (404), Grizzly queue them until it reach the max (4096 connection in
> the queue), and then close connection as well. The problem is if all
> Grizzly threads deadlock, then issuing a request will exactly produce
> what you are seeing, which is a browser spin. Now based on you
> exception, the queue is executed and Grizzly try to write response on
> closed connection. That means a thread has been released and grizzly
> is using it. So your Servlet seems to starts executing really slowly.
> We need to find why  :-)

At the moment he is thinking that GlassFish is working properly, by my
program is deadlocking servlet threads somehow causing the symptoms to
manifest.  He's looking at some of my thread dumps, which do indicate a
lot of my own threads are locked... but they are supposed to lock in
some cases.  I wrote a JCA connector that manages it's own threads. It
is exposed as a web service on one http listener, and my EJB is calling
into it.  It's possible that this whole time the issues have been my
fault, but I need help finding out what I did wrong.  Hopefully Sun can
help, or find that it really is GlassFish.   All I know is that from the
user's point of view, my app is working fine and is responsive until it
just dies.


Thanks,
Ryan


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Jeanfrancois Arcand-2]

Salut,

glassfish@... wrote:
> I corrected the JVM option as you noted.
>
> Assuming that Grizzly is the problem, is there a downside to using Coyote? Also, I don't see log entries specifically that say Coyote is being used instead of Grizzly. Should I expect such?
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

If you execute a jstack <PID>, you will see there is some classes that
start with org.apache.tomcat

Thanks

-- Jeanfrancois

>
> http://forums.java.net/jive/thread.jspa?messageID=272320
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Jeanfrancois Arcand-2]

Hi,

glassfish@... wrote:
> Where before the server stopped responding within a few hours it has run for maybe 18 hours now without problem. This is after adding the Coyote JVM option below:
> -Dcom.sun.enterprise.web.connector.useCoyoteConnector=true
>
> I would not yet call this conclusive. I think I should leave it running for several days.
>
> I also used ulimit to increase the number of allowable open files from 1024 to 8192, like so:
> ulimit -n 8192.
>

Well that make a big difference ;-) If there is a file descriptor leaks,
it will just takes more time to crash.


> I did this because I was seeing "too many open files" exceptions on my server.

That's the issue. GF will stop responding if it face that issue. Can you
do a netstat -an | grep web_port and post it? The current run (the one
with Coyote).

  The server is an Amazon EC2 instance running Ubuntu that hits S3
frequently. I believe this error to be unrelated to the unresponsive
server issue because I was seeing the server stop responding on both
machines that were experiencing the "too many open files" exceptions and
those that were not.

Hum....GF might not be able to throw any exception when the "too many
open files"....so I would think this is the same issue.

A+

-- Jeanfrancois



> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

Y


>
> http://forums.java.net/jive/thread.jspa?messageID=272391
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

Where before the server stopped responding within a few hours it has run for maybe 18 hours now without problem. This is after adding the Coyote JVM option below:
-Dcom.sun.enterprise.web.connector.useCoyoteConnector=true

I would not yet call this conclusive. I think I should leave it running for several days.

I also used ulimit to increase the number of allowable open files from 1024 to 8192, like so:
ulimit -n 8192.

I did this because I was seeing "too many open files" exceptions on my server. The server is an Amazon EC2 instance running Ubuntu that hits S3 frequently. I believe this error to be unrelated to the unresponsive server issue because I was seeing the server stop responding on both machines that were experiencing the "too many open files" exceptions and those that were not.
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272391

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

Perhaps I was unclear. Please note that the "too many open files" exceptions corresponds closely with how frequently files are read from S3. On a machine that does NOT hit S3 frequently I was still seeing Glassfish stop responding. I am not at all ready to draw a connection between these two issues.
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272397

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Jeanfrancois Arcand-2]

Salut,

glassfish@... wrote:
> In my original post at the top of this thread I noted that the server _instantly_ (no time for the domain to stop/start) comes back to life if I fiddle with a setting in the HTTP Service section of the admin console. So, what happens when such settings are changed? What is it about saving a settings change in Glassfish that could explain this? What gets replaced or flushed or reset?
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

I've missed that...the http-listener is restarted, unlocking all the
thread. If you can reproduce the problem easily, next time it happens
try to do a jstack PID (or use the admin gui to generate a jvm report).

You seems to have the same issue than Ryan, which is all thread are in
use and request are queued...since the max queue size is 4096 and your
default ulimit was 1024, that explain the exception you were getting ;-)
If you try again with GlassFish I suspect that exception will no longer
happens, but request processing will be quite slow as all threads will
have to process 4096 queued requests.

We need to find why your request are queued. Like I said, it's usually
not a Grizzly/WebContainer issue. You can reproduce that issue easily by
writing a Servlet that does:

Thread.sleep(60000);

Do 10 requests, and you will get the same unresponsiveness as you has
been observed. All requests will be serviced, but the last 4 will take a
while.

Thanks

-- Jeanfrancois


>
> http://forums.java.net/jive/thread.jspa?messageID=272417
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

In my original post at the top of this thread I noted that the server _instantly_ (no time for the domain to stop/start) comes back to life if I fiddle with a setting in the HTTP Service section of the admin console. So, what happens when such settings are changed? What is it about saving a settings change in Glassfish that could explain this? What gets replaced or flushed or reset?
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272417

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

After switching form Grizzly to Coyote it looks like my original problem of the server no longer responding to HTTP has not recurred after several days of operation. Now, there was some speculation as to whether or not my issue was also related to the fact that the maximum number of open files (set via ulimit -n) was set at only 1024. I tried to increase it to 8192, but discovered that the setting does not persist. If I close the Putty session into the server and re-open it I have discovered that the setting reverts back to 1024. These settings are viewable by running "ulimit -a". So, two things seem clear to me. 1) I don't understand how ulimit is supposed to work. Perhaps someone can tell me. 2) I don't see how this "max open files" setting can have anything to do with my problem because it reverts back to it's original setting.

It does appear however, that Grizzly vs. Coyote has made a difference. So, I ask, what is the downside, if any, of using Coyote? Is it slower? Why is the default behavior of Glassfish to use Grizzly in the first place?
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272784

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Jeanfrancois Arcand-2]

Salut,

glassfish@... wrote:
> After switching form Grizzly to Coyote it looks like my original problem of the server no longer responding to HTTP has not recurred after several days of operation. Now, there was some speculation as to whether or not my issue was also related to the fact that the maximum number of open files (set via ulimit -n) was set at only 1024. I tried to increase it to 8192, but discovered that the setting does not persist. If I close the Putty session into the server and re-open it I have discovered that the setting reverts back to 1024. These settings are viewable by running "ulimit -a". So, two things seem clear to me. 1) I don't understand how ulimit is supposed to work. Perhaps someone can tell me.

  2) I don't see how this "max open files" setting can have anything to
do with my problem because it reverts back to it's original setting.

Are you using the default configuration (domain.xml)? Without any
change, you are running fine with Coyote, right? That's impressive as
you are only running with 5 threads.


>
> It does appear however, that Grizzly vs. Coyote has made a difference. So, I ask, what is the downside, if any, of using Coyote? Is it slower?

Yes, it is (Coyote isn't scaling very well).


  Why is the default behavior of Glassfish to use Grizzly in the first
place?

I'm not aware of any issues with Grizzly. I would be very interested to
see if you can run GF with the log level set to finest to see if I can
find something? Any application you can share to reproduce the issue?

Thanks

-- Jeanfrancois


> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272784
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

>Are you using the default configuration (domain.xml)? Without any
>change, you are running fine with Coyote, right? That's impressive as
>you are only running with 5 threads.

I have thread-count set to 32 under the Request Processing section in GF. Note that this is "thread-count", not "acceptor-threads", which are two different things that I hear being constantly confused with each other.

>Yes, it is (Coyote isn't scaling very well).
I>'m not aware of any issues with Grizzly. I would be very interested to
>see if you can run GF with the log level set to finest to see if I can
>find something? Any application you can share to reproduce the issue?

But it was on this very thread that is was suggested to me to use Coyote to try to eliminate Glassfish as being a cause of my original problem (see the very first post on this thread). The conclusion was that changing to Coyote did in fact make a difference. So, doesn't that point the finger at Grizzly as a possible cause?

The fact that Coyote doesn't scale well brings me back to my initial conclusion that perhaps I have to move off of Glassfish. I can try switching back to Grizzly, but that puts me back to my original configuration where I was having the problem to begin with.
[Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

http://forums.java.net/jive/thread.jspa?messageID=272815

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Jeanfrancois Arcand-2]

glassfish@... wrote:
>> Are you using the default configuration (domain.xml)? Without any
>> change, you are running fine with Coyote, right? That's impressive as
>> you are only running with 5 threads.
>
> I have thread-count set to 32 under the Request Processing section in GF. Note that this is "thread-count", not "acceptor-threads", which are two different things that I hear being constantly confused with each other.

Ya, this is really confusion. Just let's that value set to the default
("1").


>
>> Yes, it is (Coyote isn't scaling very well).
> I>'m not aware of any issues with Grizzly. I would be very interested to
>> see if you can run GF with the log level set to finest to see if I can
>> find something? Any application you can share to reproduce the issue?
>
> But it was on this very thread that is was suggested to me to use Coyote to try to eliminate Glassfish as being a cause of my original problem (see the very first post on this thread). The conclusion was that changing to Coyote did in fact make a difference. So, doesn't that point the finger at Grizzly as a possible cause?

Yes, except I need more information to really say Yes Grizzly is the
issue. Turning off Grizzly turn off some features in GlassFish,
functionalities Coyote isn't supporting.



>
> The fact that Coyote doesn't scale well brings me back to my initial conclusion that perhaps I have to move off of Glassfish. I can try switching back to Grizzly, but that puts me back to my original configuration where I was having the problem to begin with.

Right. That would be really helpfull if you can turn it on, and when it
hangs, just to a jstack PIU and send me the file. From that I will be
able to find what's happening. Can you also send me your domain.xml?

Thanks

-- Jeanfrancois





> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272815
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Ryan de Laplante]

In case anyone still following this thread is interested, there are some
new developments:

Sun support now has a whole team of people involved helping me figure
this out.  I've managed to provide them with two full thread dumps while
it was in a locked state.  The Grizzly threads were blocked waiting for
the resource manager to return a database connection from the pool.  
Further examination revealed that the MS SQL Server JDBC driver
connected to the database, sent a login request, and was waiting forever
for a response.  At least one of our lockups happened in the middle of
MS SQL Server 2005 backup which made the DB unusable for about 40
minutes.  

We found that the JDBC driver was not configured to have a login
timeout, and the default behavior is to wait forever.  Even when the db
came back to life, it doesn't know and stays blocked.  This keeps the
resource manager blocked, and therefore the grizzly thread blocked.  
More users try to access our system, and eventually all five grizzly
threads are blocked.  Users get a blank screen waiting for a response
forever until I restart GlassFish.  Disabling the http listener then
re-enabling it brings it back to life because new threads are created,
but they would all block when trying to access the db again.


The MS SQL Server 2005 JDBC driver manual says I should add this
property to my datasource (the restart app server):

loginTimeout

       

int [>=0..65535]

       

0

       

The number of seconds the driver should wait before timing out a failed
connection. A zero value indicates no time-out value. A non-zero value
is the number of seconds the driver should wait before timing out a
failed connection.



I'm going to do that tonight. I'll let everyone know if this solves our
lockup problems for good.


I know Kevin MacDonald experiences the blank screen symptom and does not
use a database.  Something else is causing the threads to block.  If you
can capture a thread dump while it is in a locked up state, that will
help Sun determine the cause.  


Thanks,
Ryan

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

Hi,

I've been following this issue as I've come across this problem as well.  My details are:

- Sun Java System Application Server 9.0 update 1(so older than the one Kevin is using)

- Redhat Linux 5

I am running web services on my HTTPS port and after a while (it changes depending upon the load of course), usually a day or so, the calls to the HTTPS port just stop working.  No exception, nothing.  Calls to the HTTP port and the Admin port work fine.

So I first tried enabling/disabling some property on the http listener for the HTTPS port and sure enough it started working again right away.

Next I am going to try switching from Grizzly to Coyote as talked about here.  I will post result after running with that for a while.
[Message sent by forum member 'bkatnich' (bkatnich)]

http://forums.java.net/jive/thread.jspa?messageID=275069

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Ryan de Laplante]

Restarting the HTTP listener creates new threads which is why it starts
working again.  You need to figure out why the threads are locking in
the first place.   When it locks up, please run this command:

asadmin generate-jvm-report --type=thread > thread_dump.txt

Send it to this mailing list.  Someone from Sun will be able to help you
determine why the threads are blocked.  As in my case, you will probably
find that Grizzly is not the cause.  Grizzly was waiting on resource
manager to give me a database connection, and resource manager was
waiting on JDBC driver to log in.  JDBC driver was waiting forever
because it had no timeout set.    In this scenario, coyote will just
give you an HTTP error instead of waiting for one of the blocked threads
to become available.



Ryan


glassfish@... wrote:

> Hi,
>
> I've been following this issue as I've come across this problem as well.  My details are:
>
> - Sun Java System Application Server 9.0 update 1(so older than the one Kevin is using)
>
> - Redhat Linux 5
>
> I am running web services on my HTTPS port and after a while (it changes depending upon the load of course), usually a day or so, the calls to the HTTPS port just stop working.  No exception, nothing.  Calls to the HTTP port and the Admin port work fine.
>
> So I first tried enabling/disabling some property on the http listener for the HTTPS port and sure enough it started working again right away.
>
> Next I am going to try switching from Grizzly to Coyote as talked about here.  I will post result after running with that for a while.
> [Message sent by forum member 'bkatnich' (bkatnich)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275069
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

Okay, I ran it with Grizzly just as before and waited for it crap out again.  Attached is the output of the command:

asadmin generate-jvm-report --type=thread > thread_dump.txt
[Message sent by forum member 'bkatnich' (bkatnich)]

http://forums.java.net/jive/thread.jspa?messageID=275116

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[glassfish-2]

From the output it appears all the http threads (15 of them) for my ssl port (43921) are waiting for a lock:

Thread "httpWorkerThread-43921-0" thread-id 71 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline@83f892
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:485)
         at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:279)
         at: com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:73)
[Message sent by forum member 'bkatnich' (bkatnich)]

http://forums.java.net/jive/thread.jspa?messageID=275117

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Ryan de Laplante]

Excellent!    Your thread dump looks quite different than mine: I don't
see any threads that are "BLOCKED" and any mention of ResourceManager.

Now we have to wait for Jeanfrancois Arcand to look through this.  He is
the Sun employee who created Grizzly and is the one who's been helping
me.  Jeanfrancois lives in Canada and today is a holiday, so hopefully
he will have time to look at it tomorrow.


Thanks,
Ryan


glassfish@... wrote:

> Okay, I ran it with Grizzly just as before and waited for it crap out again.  Attached is the output of the command:
>
> asadmin generate-jvm-report --type=thread > thread_dump.txt
> [Message sent by forum member 'bkatnich' (bkatnich)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275116
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...