Memory Leak

[soundphed]

Hi all, I just moved a huge coldfusion site from a CF5/MySql3/Windows server to a CF7/MySql4/RHEL4 server. Needless to say, this caused a few problems. I have taken care of some of them, but this one problem is getting out of hand.

It seems like there is a leak in the cfmx7 bin...if I monitor the memory usage it will usually start out fresh with 80+ threads, each thread taking a VIRT/RES 437m/150m. If I monitor this within an hour it starts climbing, and never comes back down until the server is restarted. The tech support at the hosting company told me that the server crashes were cause by numerous 'File not found' errors.

Is this possible? Because it does seem to choke at random times...if the server takes on a large number of request at one time it will tank.

I was looking at the thread on the SQL Injection attack and wondering if that could be causing some trouble as well, i have been noticing a huge amount of those attacks, but have put a rewrite in place so an actual exception does not hit Coldfusion, it only touches Apache's access_log.

Any help would be greatly appreciated, as many long nights stressful situations have been a product of this problem, Thank you!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4444
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14

[Mike Chytracek]

If you have the ability to install Fusion Reactor, go grab the trial and
you'll likely find the problem very quickly.  If threads are getting stuck
open and the requests are piling up the memory and cpu usage will slowly
climb to a point where JRUN will stop serving pages.

Mike Chytracek
Ignite Solutions
Managing Partner

-----Original Message-----
From: Jo Ling [mailto:soundphed@...]
Sent: Thursday, September 04, 2008 12:11 PM
To: CF-Linux
Subject: Memory Leak

Hi all, I just moved a huge coldfusion site from a CF5/MySql3/Windows server
to a CF7/MySql4/RHEL4 server. Needless to say, this caused a few problems. I
have taken care of some of them, but this one problem is getting out of
hand.

It seems like there is a leak in the cfmx7 bin...if I monitor the memory
usage it will usually start out fresh with 80+ threads, each thread taking a
VIRT/RES 437m/150m. If I monitor this within an hour it starts climbing, and
never comes back down until the server is restarted. The tech support at the
hosting company told me that the server crashes were cause by numerous 'File
not found' errors.

Is this possible? Because it does seem to choke at random times...if the
server takes on a large number of request at one time it will tank.

I was looking at the thread on the SQL Injection attack and wondering if
that could be causing some trouble as well, i have been noticing a huge
amount of those attacks, but have put a rewrite in place so an actual
exception does not hit Coldfusion, it only touches Apache's access_log.

Any help would be greatly appreciated, as many long nights stressful
situations have been a product of this problem, Thank you!



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4445
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14

[Wil Genovese]

First verify if there really are 'File Not Found" errors.  A quick look at
the apache and cf application logs will tell you that.  Next look in your
application and exception logs.  Fix the errors.  Next it's down to tuning
the JVM and validating your code does what you think it does.

Examine your code first (again).  Many times people think that the code is
just fine and it's Coldfusion's fault.  It does no good to keep telling the
hosting company something is wrong with their servers only to find it's a
coding error.  One thing that can have a negative effect on memory usage are
very large query result sets.  I've seen cases where 10s of thousands of
results are returned when only a few are needed.  This causes extra memory
to be used. I've seen CF logic in the where clause of a query that leads to
an open ended query that returns all the data in the tables.  ie. select *
from tblProducts <cfif url.prodid gt ''>where prodId =
'#url.prodid#'</cfif>.  It may look innocent at first glance, but it is a
killer if the expected prodid is not given and spider traffic is notorious
for hitting pages out of order and without the expected url data.  This is
just a simple example, but the problems can be far more difficult to find or
fix. Also look for external dependencies such at cfhttp calls with no
timeout value. That can cause a thread to hang if the other server is slow
or failing then it takes your server down too.

If this hosting company has little experience with Coldfusion they may not
understand the need to tune the JVM.  In a shared CF hosting environment I'm
sure the JVM tuning has greater impact than when we do the tuning on our
dedicated servers. If the hosting company is will to work with you there is
plenty of help for JVM tuning.  Some critical points are the type of garbage
collection being done and the intervals of which it is performed.  Also as
critical are the min and max heap size settings. They need to be equal to
each other. There are other fine tuning points that may help but only if
they are willing to work with you.  If not find a host that has the
experience to run Coldfusion.


Wil Genovese
Sr. Web Application Developer





On Thu, Sep 4, 2008 at 12:18 PM, Mike Chytracek <mikechy@...> wrote:

> If you have the ability to install Fusion Reactor, go grab the trial and
> you'll likely find the problem very quickly.  If threads are getting stuck
> open and the requests are piling up the memory and cpu usage will slowly
> climb to a point where JRUN will stop serving pages.
>
> Mike Chytracek
> Ignite Solutions
> Managing Partner
>
> -----Original Message-----
> From: Jo Ling [mailto:soundphed@...]
> Sent: Thursday, September 04, 2008 12:11 PM
> To: CF-Linux
> Subject: Memory Leak
>
> Hi all, I just moved a huge coldfusion site from a CF5/MySql3/Windows
> server
> to a CF7/MySql4/RHEL4 server. Needless to say, this caused a few problems.
> I
> have taken care of some of them, but this one problem is getting out of
> hand.
>
> It seems like there is a leak in the cfmx7 bin...if I monitor the memory
> usage it will usually start out fresh with 80+ threads, each thread taking
> a
> VIRT/RES 437m/150m. If I monitor this within an hour it starts climbing,
> and
> never comes back down until the server is restarted. The tech support at
> the
> hosting company told me that the server crashes were cause by numerous
> 'File
> not found' errors.
>
> Is this possible? Because it does seem to choke at random times...if the
> server takes on a large number of request at one time it will tank.
>
> I was looking at the thread on the SQL Injection attack and wondering if
> that could be causing some trouble as well, i have been noticing a huge
> amount of those attacks, but have put a rewrite in place so an actual
> exception does not hit Coldfusion, it only touches Apache's access_log.
>
> Any help would be greatly appreciated, as many long nights stressful
> situations have been a product of this problem, Thank you!
>
>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4446
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14

[soundphed]

Thanks for the reply, I greatly appreciate it.

I should have mentioned a few more details:

I recently took the site over from a developer that had been working on it 10+ years. It is huge, I've only recently stopped my head from spinning.

There are numerous 'File Not Found' exceptions in the logs.
The site is on a dedicated virtual host, so i have root ssh access.

I guess I was mainly just curious if all the many exceptions _could_ cause a leak.
Thank you for your responses, I will try the trial of Fusion Reactor and slowly fix the errors one by one.

>First verify if there really are 'File Not Found" errors.  A quick look at
>the apache and cf application logs will tell you that.  Next look in your
>application and exception logs.  Fix the errors.  Next it's down to tuning
>the JVM and validating your code does what you think it does.
>
>Examine your code first (again).  Many times people think that the code is
>just fine and it's Coldfusion's fault.  It does no good to keep telling the
>hosting company something is wrong with their servers only to find it's a
>coding error.  One thing that can have a negative effect on memory usage are
>very large query result sets.  I've seen cases where 10s of thousands of
>results are returned when only a few are needed.  This causes extra memory
>to be used. I've seen CF logic in the where clause of a query that leads to
>an open ended query that returns all the data in the tables.  ie. select *
>from tblProducts <cfif url.prodid gt ''>where prodId =
>'#url.prodid#'</cfif>.  It may look innocent at first glance, but it is a
>killer if the expected prodid is not given and spider traffic is notorious
>for hitting pages out of order and without the expected url data.  This is
>just a simple example, but the problems can be far more difficult to find or
>fix. Also look for external dependencies such at cfhttp calls with no
>timeout value. That can cause a thread to hang if the other server is slow
>or failing then it takes your server down too.
>
>If this hosting company has little experience with Coldfusion they may not
>understand the need to tune the JVM.  In a shared CF hosting environment I'm
>sure the JVM tuning has greater impact than when we do the tuning on our
>dedicated servers. If the hosting company is will to work with you there is
>plenty of help for JVM tuning.  Some critical points are the type of garbage
>collection being done and the intervals of which it is performed.  Also as
>critical are the min and max heap size settings. They need to be equal to
>each other. There are other fine tuning points that may help but only if
>they are willing to work with you.  If not find a host that has the
>experience to run Coldfusion.
>
>
>Wil Genovese
>Sr. Web Application Developer
>
>
>
>
>
>On Thu, Sep 4, 2008 at 12:18 PM, Mike Chytracek <mikechy@...> wrote:
>
>>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4447
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14

[Wil Genovese]

Set the whole site up on an in house dev server and beat the crap out of it
and debug it completely. Then you have total control.  I think it may be
possible for file not founds on a large scale to cause problems, depending
on the file(s) that are not found and at what level.  If they are apache
errors then they have or should have no effect on CF.  If they are CF file
not found as in file includes not found then you got problems.

Another tool beside Fusion Reactor is Seefusion.  I've used both extensively
and both have their merits and unique features.

Wil Genovese
Sr. Web Application Developer



On Thu, Sep 4, 2008 at 1:21 PM, Jo Ling <soundphed@...> wrote:

> Thanks for the reply, I greatly appreciate it.
>
> I should have mentioned a few more details:
>
> I recently took the site over from a developer that had been working on it
> 10+ years. It is huge, I've only recently stopped my head from spinning.
>
> There are numerous 'File Not Found' exceptions in the logs.
> The site is on a dedicated virtual host, so i have root ssh access.
>
> I guess I was mainly just curious if all the many exceptions _could_ cause
> a leak.
> Thank you for your responses, I will try the trial of Fusion Reactor and
> slowly fix the errors one by one.
>
> >First verify if there really are 'File Not Found" errors.  A quick look at
> >the apache and cf application logs will tell you that.  Next look in your
> >application and exception logs.  Fix the errors.  Next it's down to tuning
> >the JVM and validating your code does what you think it does.
> >
> >Examine your code first (again).  Many times people think that the code is
> >just fine and it's Coldfusion's fault.  It does no good to keep telling
> the
> >hosting company something is wrong with their servers only to find it's a
> >coding error.  One thing that can have a negative effect on memory usage
> are
> >very large query result sets.  I've seen cases where 10s of thousands of
> >results are returned when only a few are needed.  This causes extra memory
> >to be used. I've seen CF logic in the where clause of a query that leads
> to
> >an open ended query that returns all the data in the tables.  ie. select *
> >from tblProducts <cfif url.prodid gt ''>where prodId =
> >'#url.prodid#'</cfif>.  It may look innocent at first glance, but it is a
> >killer if the expected prodid is not given and spider traffic is notorious
> >for hitting pages out of order and without the expected url data.  This is
> >just a simple example, but the problems can be far more difficult to find
> or
> >fix. Also look for external dependencies such at cfhttp calls with no
> >timeout value. That can cause a thread to hang if the other server is slow
> >or failing then it takes your server down too.
> >
> >If this hosting company has little experience with Coldfusion they may not
> >understand the need to tune the JVM.  In a shared CF hosting environment
> I'm
> >sure the JVM tuning has greater impact than when we do the tuning on our
> >dedicated servers. If the hosting company is will to work with you there
> is
> >plenty of help for JVM tuning.  Some critical points are the type of
> garbage
> >collection being done and the intervals of which it is performed.  Also as
> >critical are the min and max heap size settings. They need to be equal to
> >each other. There are other fine tuning points that may help but only if
> >they are willing to work with you.  If not find a host that has the
> >experience to run Coldfusion.
> >
> >
> >Wil Genovese
> >Sr. Web Application Developer
> >
> >
> >
> >
> >
> >On Thu, Sep 4, 2008 at 12:18 PM, Mike Chytracek <mikechy@...>
> wrote:
> >
> >>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4448
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14

[Judith Dinowitz-2]

We did a review of FusionReactor 3 in Vol. 2 Issue 4 of FAQU, and we
posted that article online... the only excerpt we've posted online
from that issue. So if you're interested in what FusionReactor can
offer you, this will help you:

http://www.fusionauthority.com/quarterly/do-more-code-less/fusionreactor-coldfusion-server-healthcare.pdf

Judith

On Thu, Sep 4, 2008 at 3:03 PM, Wil Genovese <juggler@...> wrote:

> Set the whole site up on an in house dev server and beat the crap out of it
> and debug it completely. Then you have total control.  I think it may be
> possible for file not founds on a large scale to cause problems, depending
> on the file(s) that are not found and at what level.  If they are apache
> errors then they have or should have no effect on CF.  If they are CF file
> not found as in file includes not found then you got problems.
>
> Another tool beside Fusion Reactor is Seefusion.  I've used both extensively
> and both have their merits and unique features.
>
> Wil Genovese
> Sr. Web Application Developer

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;203748912;27390454;j

Archive: http://www.houseoffusion.com/groups/CF-Linux/message.cfm/messageid:4449
Subscription: http://www.houseoffusion.com/groups/CF-Linux/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.14