Intermittent CF MX 7.0.2 hangs

[Shana & Iann Gorrill]

Hi everyone,

I'm quite a bit of a newb at CF system administration, and I've been  
tasked with fixing a problem as our regular CF guy is gone for a few  
weeks.  So, please take it easy on me if I don't provide what needs  
to be provided :)

The problem we are experiencing is happening on a Windows 2003 SP2  
machine with 4 GB RAM.  CF is 7.0.2.142559 Standard edition with an  
update level of C:/CFusionMX7/lib/updates/hf702-71489.jar taken from  
the System Information section in the Administrator.

Basically, our problem is this.  About once or twice a day CF will  
crash/die/hang/whatever have you and it requires a restart of the  
service.  The only thing I think I've been able to find that might be  
related is this error in runtime/logs:

java.lang.RuntimeException: Request timed out waiting for an  
available thread to run. You may want to consider increasing the  
number of active threads in the thread pool.

Through some googling, that seems to mean to increase the  
Simultaneous Request setting in the Settings section of the  
Administrator.  It was at 16 before and I've increased it to 32 with  
no resolve in sight.

So, that's about it.  Not much information to go off of, but I'm at  
anyone's mercy to point me in the right direction or bring back more  
information to the table.

Thanks,
Iann


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6657
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jochem van Dieten-2]

Shana & Iann Gorrill wrote:
> The problem we are experiencing is happening on a Windows 2003 SP2  
> machine with 4 GB RAM.  CF is 7.0.2.142559 Standard edition with an  
> update level of C:/CFusionMX7/lib/updates/hf702-71489.jar taken from  
> the System Information section in the Administrator.
>
> Basically, our problem is this.  About once or twice a day CF will  
> crash/die/hang/whatever have you and it requires a restart of the  
> service.

What is the CPU load when this happens? How long have you waited for CF
to recover? What do you have in jvm.config and how much memory was in
use by the JRun process? Have you tried collecting a full thread dump?
http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18339

Jochem


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6658
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Shana & Iann Gorrill]

On 25-Jul-08, at 1:57 PM, Jochem van Dieten wrote:

> Shana & Iann Gorrill wrote:
>> The problem we are experiencing is happening on a Windows 2003 SP2
>> machine with 4 GB RAM.  CF is 7.0.2.142559 Standard edition with an
>> update level of C:/CFusionMX7/lib/updates/hf702-71489.jar taken from
>> the System Information section in the Administrator.
>>
>> Basically, our problem is this.  About once or twice a day CF will
>> crash/die/hang/whatever have you and it requires a restart of the
>> service.
>
> What is the CPU load when this happens? How long have you waited  
> for CF
> to recover? What do you have in jvm.config and how much memory was in
> use by the JRun process? Have you tried collecting a full thread dump?
> http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18339
>
> Jochem
>
>

Thanks for your reply Jochem.  I don't have numbers to tell you, but  
I'll give you the basic idea and get the real deal when it happens  
again.  CPU load is a non-issue when this has happened.  I've slept  
through a few Nagios pages when this has happened, so I'd say upwards  
of 30 minutes without it resolving itself.

Here is the contents of jvm.config:

#
# VM configuration
#
# Where to find JVM, if {java.home}/jre exists then that JVM is used
# if not then it must be the path to the JRE itself
java.home=C:/CFusionMX7/runtime/jre
#
# If no java.home is specified a VM is located by looking in these  
places in this
# order:
#
#  1) bin directory for java.dll (windows) or lib/<ARCH>/libjava.so  
(unix)
#  2) ../jre
#  3) registry (windows only)
#  4) JAVA_HOME env var plus jre (ie $JAVA_HOME/jre)
#

# Arguments to VM
java.args=-server  -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -
Djava.security.manager "-Djava.security.policy=C:\\CFusionMX7\\lib\
\coldfusion.policy" "-Djava.security.auth.policy=C:\\CFusionMX7\\lib\
\neo_jaas.policy" -Xmx1024m -Dsun.io.useCanonCaches=false -
XX:MaxPermSize=128m -Dcoldfusion.rootDir={application.home}/../ -
Dcoldfusion.libPath={application.home}/../lib -XX:+UseParallelGC -
Dcoldfusion.classPath={application.home}/../../classes,
{application.home}/../lib/updates,{application.home}/../lib/,
{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-
INF/cfform/jars

#
# commas will be converted to platform specific separator and the  
result will be passed
# as -Djava.ext.dirs= to the VM
java.ext.dirs={jre.home}/lib/ext

#
# where to find shared libraries
java.library.path={application.home}/../lib,{application.home}/../
jintegra/bin,{application.home}/../jintegra/bin/international
system.path.first=false

#
# set the current working directory - useful for Windows to control
# the default search path used when loading DLLs since it comes
# before system directory, windows directory and PATH
java.user.dir={application.home}/../../lib

# JVM classpath
java.class.path={application.home}/servers/lib,{application.home}/../
lib/macromedia_drivers.jar,{application.home}/lib/cfmx_mbean.jar,
{application.home}/lib

As for a thread dump, I haven't done that but will do it the next  
time we see this issue and report back.

Thanks
Iann

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6659
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[John Mason-6]

Also, check your CF exception logs for Java out of memory errors. Just do a
search for "memory". See if you're running into memory constraints.

John Mason
mason@...
770.337.8363
 
www.FusionLink.com - ColdFusion and Flex hosting
Now offering VPS Plans running with VMware technology
Now offering ColdFusion 8 Enterprise hosting
FREE Subversion hosting

-----Original Message-----
From: Shana & Iann Gorrill [mailto:shana.iann@...]
Sent: Friday, July 25, 2008 6:44 PM
To: CF-Server
Subject: Re: Intermittent CF MX 7.0.2 hangs

On 25-Jul-08, at 1:57 PM, Jochem van Dieten wrote:

> Shana & Iann Gorrill wrote:
>> The problem we are experiencing is happening on a Windows 2003 SP2
>> machine with 4 GB RAM.  CF is 7.0.2.142559 Standard edition with an
>> update level of C:/CFusionMX7/lib/updates/hf702-71489.jar taken from
>> the System Information section in the Administrator.
>>
>> Basically, our problem is this.  About once or twice a day CF will
>> crash/die/hang/whatever have you and it requires a restart of the
>> service.
>
> What is the CPU load when this happens? How long have you waited for
> CF to recover? What do you have in jvm.config and how much memory was
> in use by the JRun process? Have you tried collecting a full thread
> dump?
> http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18339
>
> Jochem
>
>

Thanks for your reply Jochem.  I don't have numbers to tell you, but I'll
give you the basic idea and get the real deal when it happens again.  CPU
load is a non-issue when this has happened.  I've slept through a few Nagios
pages when this has happened, so I'd say upwards of 30 minutes without it
resolving itself.

Here is the contents of jvm.config:

#
# VM configuration
#
# Where to find JVM, if {java.home}/jre exists then that JVM is used # if
not then it must be the path to the JRE itself
java.home=C:/CFusionMX7/runtime/jre
#
# If no java.home is specified a VM is located by looking in these places in
this # order:
#
#  1) bin directory for java.dll (windows) or lib/<ARCH>/libjava.so
(unix)
#  2) ../jre
#  3) registry (windows only)
#  4) JAVA_HOME env var plus jre (ie $JAVA_HOME/jre) #

# Arguments to VM
java.args=-server  -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -
Djava.security.manager "-Djava.security.policy=C:\\CFusionMX7\\lib\
\coldfusion.policy" "-Djava.security.auth.policy=C:\\CFusionMX7\\lib\
\neo_jaas.policy" -Xmx1024m -Dsun.io.useCanonCaches=false -
XX:MaxPermSize=128m -Dcoldfusion.rootDir={application.home}/../ -
Dcoldfusion.libPath={application.home}/../lib -XX:+UseParallelGC -
Dcoldfusion.classPath={application.home}/../../classes,
{application.home}/../lib/updates,{application.home}/../lib/,
{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-
INF/cfform/jars

#
# commas will be converted to platform specific separator and the result
will be passed # as -Djava.ext.dirs= to the VM
java.ext.dirs={jre.home}/lib/ext

#
# where to find shared libraries
java.library.path={application.home}/../lib,{application.home}/../
jintegra/bin,{application.home}/../jintegra/bin/international
system.path.first=false

#
# set the current working directory - useful for Windows to control
# the default search path used when loading DLLs since it comes
# before system directory, windows directory and PATH
java.user.dir={application.home}/../../lib

# JVM classpath
java.class.path={application.home}/servers/lib,{application.home}/../
lib/macromedia_drivers.jar,{application.home}/lib/cfmx_mbean.jar,
{application.home}/lib

As for a thread dump, I haven't done that but will do it the next  
time we see this issue and report back.

Thanks
Iann



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6660
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[James Holmes-3]

Agreed, as assuming your using most of the -Xmx1024m,
XX:MaxPermSize=128m is likely to be too small.

On Sat, Jul 26, 2008 at 6:48 AM, John Masonwrote:
> Also, check your CF exception logs for Java out of memory errors. Just do a
> search for "memory". See if you're running into memory constraints.

--
mxAjax / CFAjax docs and other useful articles:
http://www.bifrost.com.au/blog/

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6661
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Shana & Iann Gorrill]

On 25-Jul-08, at 8:17 PM, James Holmes wrote:

> Agreed, as assuming your using most of the -Xmx1024m,
> XX:MaxPermSize=128m is likely to be too small.
>
> On Sat, Jul 26, 2008 at 6:48 AM, John Masonwrote:
>> Also, check your CF exception logs for Java out of memory errors.  
>> Just do a
>> search for "memory". See if you're running into memory constraints.
>
>

What is a sane number for this to be set at?

Thanks,
Iann

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6663
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Shana & Iann Gorrill]

On 25-Jul-08, at 4:43 PM, Shana & Iann Gorrill wrote:

> On 25-Jul-08, at 1:57 PM, Jochem van Dieten wrote:
>
>> Shana & Iann Gorrill wrote:
>>> The problem we are experiencing is happening on a Windows 2003 SP2
>>> machine with 4 GB RAM.  CF is 7.0.2.142559 Standard edition with an
>>> update level of C:/CFusionMX7/lib/updates/hf702-71489.jar taken from
>>> the System Information section in the Administrator.
>>>
>>> Basically, our problem is this.  About once or twice a day CF will
>>> crash/die/hang/whatever have you and it requires a restart of the
>>> service.
>>
>> What is the CPU load when this happens? How long have you waited
>> for CF
>> to recover? What do you have in jvm.config and how much memory was in
>> use by the JRun process? Have you tried collecting a full thread  
>> dump?
>> http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_18339
>>
>> Jochem
>>
>>
> As for a thread dump, I haven't done that but will do it the next
> time we see this issue and report back.
>

Where exactly is the location of the thread dump log after it's been  
done?  The Adobe link just says what name it might have, but that was  
it.  I've tailed (over samba) the c:\CFusionMX7\runtime\logs
\coldfusion-out.log file and it seems to be getting some entries when  
we try the CTRL+BREAK key combo, but I imagine there should be  
another file?

Thanks,
Iann




~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6664
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Chris Norloff-2]

The Permanent Generation's default size is 128 MB (XX:MaxPermSize=128m)...
that's up from the default 64 MB originally in CFMX v6.  

128 MB has been fine for everything I've done before, but in the the current
job we learned PermGen was increasing up to 99% usage, then a Full Garbage
Collection, then it would drop to 50% and start increasing again.

We increased the Permanent Generation to 256 MB and it stayed between
62-78%. Much better!

Chris


-----Original Message-----
From: Shana & Iann Gorrill [mailto:shana.iann@...]
Sent: Monday, July 28, 2008 3:59 PM
To: CF-Server
Subject: Re: Intermittent CF MX 7.0.2 hangs

On 25-Jul-08, at 8:17 PM, James Holmes wrote:

> Agreed, as assuming your using most of the -Xmx1024m,
> XX:MaxPermSize=128m is likely to be too small.
>
> On Sat, Jul 26, 2008 at 6:48 AM, John Masonwrote:
>> Also, check your CF exception logs for Java out of memory errors.  
>> Just do a
>> search for "memory". See if you're running into memory constraints.
>
>

What is a sane number for this to be set at?

Thanks,
Iann



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6665
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Shana & Iann Gorrill]

On 29-Jul-08, at 8:04 AM, Chris Norloff wrote:

> The Permanent Generation's default size is 128 MB  
> (XX:MaxPermSize=128m)...
> that's up from the default 64 MB originally in CFMX v6.
>
> 128 MB has been fine for everything I've done before, but in the  
> the current
> job we learned PermGen was increasing up to 99% usage, then a Full  
> Garbage
> Collection, then it would drop to 50% and start increasing again.
>
> We increased the Permanent Generation to 256 MB and it stayed between
> 62-78%. Much better!
>
> Chris
>
>

Hello everyone,

It seems that increasing the MaxPermSize entry has made a difference  
on our one server, but within the same time period we started having  
issues with another ColdFusion server.  Similar types of issues being  
shown, but increasing that value hasn't made as big of a difference.

I asked previously about the location of the thread dump file, but no  
one answered, so I'll ask again.  After performing a thread dump when  
running CF with the cmd window active (CTRL+BREAK) where is the file  
that is being created located?

Also, another question, how do you determine if you are up to date as  
far as patches/updates/hotfixes go?  I didn't see an easy way to  
ensure that you are completely updated on Adobe's site, so I'd  
welcome any assistance on that one.

Thanks,
Iann

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6681
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jochem van Dieten-2]

Shana & Iann Gorrill wrote:
> I asked previously about the location of the thread dump file, but no  
> one answered, so I'll ask again.  After performing a thread dump when  
> running CF with the cmd window active (CTRL+BREAK) where is the file  
> that is being created located?

In the *-out.log.


> Also, another question, how do you determine if you are up to date as  
> far as patches/updates/hotfixes go?  I didn't see an easy way to  
> ensure that you are completely updated on Adobe's site, so I'd  
> welcome any assistance on that one.

Compare /cfusion/lob/updates/ with the list on the Adobe site. All our
CF 7.02 servers are at Cumulative Hotfix 3 level.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6682
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Hemant Khandelwal]

Rupesh has written a very nice article about Thread Dumps. You may want to look at that.

http://coldfused.blogspot.com/2005/11/thread-dumps.html 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6684
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Shana & Iann Gorrill]

On 7-Aug-08, at 6:10 AM, Hemant Khandelwal wrote:

> Rupesh has written a very nice article about Thread Dumps. You may  
> want to look at that.
>
> http://coldfused.blogspot.com/2005/11/thread-dumps.html
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Server/message.cfm/messageid:6685
Subscription: http://www.houseoffusion.com/groups/CF-Server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10