Comet - Selector spinning - 100% CPU
|
View:
New views
7 Messages
—
Rating Filter:
Alert me
|
 |
Comet - Selector spinning - 100% CPU
by Rodrigo Ramirez
::
Rate this Message:
Hi,
We are having this high CPU issue with Comet: After few minutes or hours some times, we get a 100% CPU usage for one of the server’s CPUs. The situation is pretty similar to the one described at: http://www.nabble.com/Comet-handler-starts-terminating-TCP-connections-with-RST--td20337445.html and http://www.nabble.com/do-select-spin-on-linux-td22974389.html We used jconsole / jtop to find the thread using high CPU and got this: Name: SelectorThread-443 State: RUNNABLE Total blocked: 327 Total waited: 0 Stack trace: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.Util$1@3c304a - locked java.util.Collections$UnmodifiableSet@18682d0 - locked sun.nio.ch.EPollSelectorImpl@1a7f9da sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348) com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295) - locked [Ljava.lang.Object;@104959c com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258) com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234) Some points to note: We see the issue in different environments, including: - Glassfish v2 ur2 and Glassfish v2.1 - Ubuntu Linux 2.6.15-26-server and 2.6.27-7-server - JVM 1.6.0 and 1.6.10 This is what we have tried since we upgraded to Glassfish V2: 1. Applied the patch grizzly-comet-selector-patch2.jar 2. Removed the patch, set CometContext.setExpirationDelay(-1) and used a ScheduledExecutorService to expire the Comet Handlers, but even that is not working. Since all my comet implementation is encapsulated in a single class, I am attaching it. May be I am doing something silly. The commented out code is for my failed workaround using set CometContext.setExpirationDelay(-1) and a ScheduledExecutorService to expire the Comet Handlers. Does any one can suggest a workaround different from using java 1.7? Should the latest Grizzly help? Thanks :-) PortalCometQueue.java |
|
|
Re: Comet - Selector spinning - 100% CPU
by Jeanfrancois Arcand-2
::
Rate this Message:
Salut,
Rodrigo Ramirez wrote: > Hi, > > We are having this high CPU issue with Comet: After few minutes or hours > some times, we get a 100% CPU usage for one of the server’s CPUs. The > situation is pretty similar to the one described at: > http://www.nabble.com/Comet-handler-starts-terminating-TCP-connections-with-RST--td20337445.html > and http://www.nabble.com/do-select-spin-on-linux-td22974389.html > > We used jconsole / jtop to find the thread using high CPU and got this: > > Name: SelectorThread-443 > State: RUNNABLE > Total blocked: 327 Total waited: 0 > > Stack trace: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked sun.nio.ch.Util$1@3c304a > - locked java.util.Collections$UnmodifiableSet@18682d0 > - locked sun.nio.ch.EPollSelectorImpl@1a7f9da > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348) > com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295) > - locked [Ljava.lang.Object;@104959c > com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258) > com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234) > > > Some points to note: > > We see the issue in different environments, including: > - Glassfish v2 ur2 and Glassfish v2.1 > - Ubuntu Linux 2.6.15-26-server and 2.6.27-7-server > - JVM 1.6.0 and 1.6.10 > > > This is what we have tried since we upgraded to Glassfish V2: > > 1. Applied the patch grizzly-comet-selector-patch2.jar > > 2. Removed the patch, set CometContext.setExpirationDelay(-1) and used a > ScheduledExecutorService to expire the Comet Handlers, but even that is not > working. > > > Since all my comet implementation is encapsulated in a single class, I am > attaching it. May be I am doing something silly. The commented out code is > for my failed workaround using set CometContext.setExpirationDelay(-1) and a > ScheduledExecutorService to expire the Comet Handlers. > > Does any one can suggest a workaround different from using java 1.7? Should > the latest Grizzly help? I suspect the latest Grizzly will not help. I need to find a way to reproduce the issue and then I will try to apply the patch. Let me try to find something...jdk 6ur16 will have the fix. Meanwhile, can you try with JDK 1.7 b50 and makes sure the problem never happens? Thanks! -- Jeanfrancois > > Thanks :-) > > http://www.nabble.com/file/p23142529/PortalCometQueue.java > PortalCometQueue.java --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Comet - Selector spinning - 100% CPU
by Rodrigo Ramirez
::
Rate this Message:
Hi Jeanfrancois,
I tested with JDK 1.7 b55 and I didn't get the selector thread spining. So I think we hit the same JDK bug everybody complains about.
|
|
|
Re: Comet - Selector spinning - 100% CPU
by Jeanfrancois Arcand-2
::
Rate this Message:
Salut,
Rodrigo Ramirez wrote: > Hi Jeanfrancois, > > I tested with JDK 1.7 b55 and I didn't get the selector thread spining. So I > think we hit the same JDK bug everybody complains about. Thanks!! I will push for the fix for an upcoming JDK 6 release. I'm trying to find a workaround right now. Will keep you posted on the issue. --Jeanfrancois > > > > Rodrigo Ramirez wrote: >> Hi, >> >> We are having this high CPU issue with Comet: After few minutes or hours >> some times, we get a 100% CPU usage for one of the server’s CPUs. The >> situation is pretty similar to the one described at: >> http://www.nabble.com/Comet-handler-starts-terminating-TCP-connections-with-RST--td20337445.html >> and http://www.nabble.com/do-select-spin-on-linux-td22974389.html >> >> We used jconsole / jtop to find the thread using high CPU and got this: >> >> Name: SelectorThread-443 >> State: RUNNABLE >> Total blocked: 327 Total waited: 0 >> >> Stack trace: >> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.Util$1@3c304a >> - locked java.util.Collections$UnmodifiableSet@18682d0 >> - locked sun.nio.ch.EPollSelectorImpl@1a7f9da >> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348) >> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295) >> - locked [Ljava.lang.Object;@104959c >> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258) >> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234) >> >> >> Some points to note: >> >> We see the issue in different environments, including: >> - Glassfish v2 ur2 and Glassfish v2.1 >> - Ubuntu Linux 2.6.15-26-server and 2.6.27-7-server >> - JVM 1.6.0 and 1.6.10 >> >> >> This is what we have tried since we upgraded to Glassfish V2: >> >> 1. Applied the patch grizzly-comet-selector-patch2.jar >> >> 2. Removed the patch, set CometContext.setExpirationDelay(-1) and used a >> ScheduledExecutorService to expire the Comet Handlers, but even that is >> not working. >> >> >> Since all my comet implementation is encapsulated in a single class, I am >> attaching it. May be I am doing something silly. The commented out code is >> for my failed workaround using set CometContext.setExpirationDelay(-1) and >> a ScheduledExecutorService to expire the Comet Handlers. >> >> Does any one can suggest a workaround different from using java 1.7? >> Should the latest Grizzly help? >> >> Thanks :-) >> >> http://www.nabble.com/file/p23142529/PortalCometQueue.java >> PortalCometQueue.java >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Comet - Selector spinning - 100% CPU
by Rodrigo Ramirez
::
Rate this Message:
Hey Jeanfrancois, have you been able to create a patch for this? Is anything I can help with?
Thanks, Rodrigo
|
|
|
Re: Comet - Selector spinning - 100% CPU
by Jeanfrancois Arcand-2
::
Rate this Message:
Salut,
Rodrigo Ramirez wrote: > Hey Jeanfrancois, have you been able to create a patch for this? Is anything > I can help with? we patched the main Selector in 1.9.x, but I still have to backport this to 2.1. Completely swamped those days...will try before the end of the week. Thanks! -- Jeanfrancois > > Thanks, > > Rodrigo > > > Jeanfrancois Arcand-2 wrote: >> Salut, >> >> Rodrigo Ramirez wrote: >>> Hi Jeanfrancois, >>> >>> I tested with JDK 1.7 b55 and I didn't get the selector thread spining. >>> So I >>> think we hit the same JDK bug everybody complains about. >> Thanks!! >> >> I will push for the fix for an upcoming JDK 6 release. I'm trying to >> find a workaround right now. Will keep you posted on the issue. >> >> --Jeanfrancois >> >>> >>> >>> Rodrigo Ramirez wrote: >>>> Hi, >>>> >>>> We are having this high CPU issue with Comet: After few minutes or hours >>>> some times, we get a 100% CPU usage for one of the server’s CPUs. The >>>> situation is pretty similar to the one described at: >>>> http://www.nabble.com/Comet-handler-starts-terminating-TCP-connections-with-RST--td20337445.html >>>> and http://www.nabble.com/do-select-spin-on-linux-td22974389.html >>>> >>>> We used jconsole / jtop to find the thread using high CPU and got this: >>>> >>>> Name: SelectorThread-443 >>>> State: RUNNABLE >>>> Total blocked: 327 Total waited: 0 >>>> >>>> Stack trace: >>>> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) >>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>> - locked sun.nio.ch.Util$1@3c304a >>>> - locked java.util.Collections$UnmodifiableSet@18682d0 >>>> - locked sun.nio.ch.EPollSelectorImpl@1a7f9da >>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295) >>>> - locked [Ljava.lang.Object;@104959c >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234) >>>> >>>> >>>> Some points to note: >>>> >>>> We see the issue in different environments, including: >>>> - Glassfish v2 ur2 and Glassfish v2.1 >>>> - Ubuntu Linux 2.6.15-26-server and 2.6.27-7-server >>>> - JVM 1.6.0 and 1.6.10 >>>> >>>> >>>> This is what we have tried since we upgraded to Glassfish V2: >>>> >>>> 1. Applied the patch grizzly-comet-selector-patch2.jar >>>> >>>> 2. Removed the patch, set CometContext.setExpirationDelay(-1) and used a >>>> ScheduledExecutorService to expire the Comet Handlers, but even that is >>>> not working. >>>> >>>> >>>> Since all my comet implementation is encapsulated in a single class, I >>>> am >>>> attaching it. May be I am doing something silly. The commented out code >>>> is >>>> for my failed workaround using set CometContext.setExpirationDelay(-1) >>>> and >>>> a ScheduledExecutorService to expire the Comet Handlers. >>>> >>>> Does any one can suggest a workaround different from using java 1.7? >>>> Should the latest Grizzly help? >>>> >>>> Thanks :-) >>>> >>>> http://www.nabble.com/file/p23142529/PortalCometQueue.java >>>> PortalCometQueue.java >>>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Comet - Selector spinning - 100% CPU
by Jeanfrancois Arcand-2
::
Rate this Message:
Salut,
this is now fixed in 1.0.30_SNAPSHOT. If you have a chance, try it and let me know. Thanks! -- Jeanfrancois Rodrigo Ramirez wrote: > Hey Jeanfrancois, have you been able to create a patch for this? Is anything > I can help with? > > Thanks, > > Rodrigo > > > Jeanfrancois Arcand-2 wrote: >> Salut, >> >> Rodrigo Ramirez wrote: >>> Hi Jeanfrancois, >>> >>> I tested with JDK 1.7 b55 and I didn't get the selector thread spining. >>> So I >>> think we hit the same JDK bug everybody complains about. >> Thanks!! >> >> I will push for the fix for an upcoming JDK 6 release. I'm trying to >> find a workaround right now. Will keep you posted on the issue. >> >> --Jeanfrancois >> >>> >>> >>> Rodrigo Ramirez wrote: >>>> Hi, >>>> >>>> We are having this high CPU issue with Comet: After few minutes or hours >>>> some times, we get a 100% CPU usage for one of the server’s CPUs. The >>>> situation is pretty similar to the one described at: >>>> http://www.nabble.com/Comet-handler-starts-terminating-TCP-connections-with-RST--td20337445.html >>>> and http://www.nabble.com/do-select-spin-on-linux-td22974389.html >>>> >>>> We used jconsole / jtop to find the thread using high CPU and got this: >>>> >>>> Name: SelectorThread-443 >>>> State: RUNNABLE >>>> Total blocked: 327 Total waited: 0 >>>> >>>> Stack trace: >>>> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) >>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>> - locked sun.nio.ch.Util$1@3c304a >>>> - locked java.util.Collections$UnmodifiableSet@18682d0 >>>> - locked sun.nio.ch.EPollSelectorImpl@1a7f9da >>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1348) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1295) >>>> - locked [Ljava.lang.Object;@104959c >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1258) >>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1234) >>>> >>>> >>>> Some points to note: >>>> >>>> We see the issue in different environments, including: >>>> - Glassfish v2 ur2 and Glassfish v2.1 >>>> - Ubuntu Linux 2.6.15-26-server and 2.6.27-7-server >>>> - JVM 1.6.0 and 1.6.10 >>>> >>>> >>>> This is what we have tried since we upgraded to Glassfish V2: >>>> >>>> 1. Applied the patch grizzly-comet-selector-patch2.jar >>>> >>>> 2. Removed the patch, set CometContext.setExpirationDelay(-1) and used a >>>> ScheduledExecutorService to expire the Comet Handlers, but even that is >>>> not working. >>>> >>>> >>>> Since all my comet implementation is encapsulated in a single class, I >>>> am >>>> attaching it. May be I am doing something silly. The commented out code >>>> is >>>> for my failed workaround using set CometContext.setExpirationDelay(-1) >>>> and >>>> a ScheduledExecutorService to expire the Comet Handlers. >>>> >>>> Does any one can suggest a workaround different from using java 1.7? >>>> Should the latest Grizzly help? >>>> >>>> Thanks :-) >>>> >>>> http://www.nabble.com/file/p23142529/PortalCometQueue.java >>>> PortalCometQueue.java >>>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
| Free embeddable forum powered by Nabble | Forum Help |
