CF8 web service invocation error - possibly when under high load or after 2-3 days uptime

[Simon Hooker]

Hi,

I currently have a web service "MYSERVICE".  This service is on 3 physically seperate web servers, "DEV", "TEST", "LIVE".  

"DEV" is accessed by developers only
"TEST" is accessed by developers and clients only
"LIVE" is public

Coldfusion version for all servers is 8.0.1.  Axis version is the same on all servers.

The consuming websites for "DEV" resides on the same physical server as the web service, and has very low traffic.  The consuming websites for "TEST" reside on the same physical server as the consuming websites for "LIVE", which is a seperate physical server to the "TEST" web service.  The consuming websites for "LIVE" reside on the same physical server as the "LIVE" web service.
The consuming websites for "DEV" resides on the same physical server as the web service, and has very low traffic.  The consuming websites for "TEST" reside on the same physical server as the consuming websites for "LIVE", which is a seperate physical server to the "TEST" web service.  The consuming websites for "LIVE" reside on the same physical server as the "LIVE" web service.

Periodically the following error is experienced :

AXIS ERROR org.apache.axis.encoding.ser.BeanPropertyTarget - Could not convert MYSERVICE.types.common.login to bean field 'auth', type MYSERVICE.types.common.login

This only appears to affect "TEST" and "LIVE".  Previously the "TEST" webservice was  on the same phsyical webserver as "LIVE" webservice + "TEST" + "LIVE" websites.  During this time the "TEST" webservice was accessed as "MYSERVICE_test".

Currently on "LIVE" server there is a directory mapping for /MYSERVICE/ to the root directory of the "LIVE" web service.  There is also a coldfusion webservice "MYSERVICE_search" pointing to the WSDL of the live web service.  There is also a coldfusion webservice "MYSERVICE_search_test" pointing to the wsdl of the test web service.

The error appears to happen most commonly when under high load, and the only way I have so far found to correct it involves resetting the coldfusion server which is not ideal.

The process prior to the error is as follows :

obj_service = CreateObject('webservice','MYSERVICE');
obj_login = obj_service.authLogin('user','password');
obj_search = obj_service.doSearch(obj_login,'somevar');

The AXIS error appears to relate to that .doSearch method call.

Does anyone have any thoughts as to what may be causing this and how I can work around this?  If there is additional information required then please let me know and I will get that together.  All indications are that this is a problem on the side of the consuming website, although I cannot tell any more than that and any research I have done over the last day or so has proved fruitless (or has provided outdated, i.e. 5-7 year old, articles)

- Simon H


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:327987
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

As an additional note to the above, the "LIVE" server restarts every day at approx 0700, and this problem only ever seems to occur at some point past 1900-2100 and then seems to continue sporadically until server restart at 0700.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:327988
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

> As an additional note to the above, the "LIVE" server restarts every
> day at approx 0700, and this problem only ever seems to occur at some
> point past 1900-2100 and then seems to continue sporadically until
> server restart at 0700.


Further information.  I have kept an eye on this issue, and it appears that the error only occurs soon after the .class files in /opt/coldfusion/wwwroot/WEB-INF/cfc-skeletons/.... are modified, which indicates to me something is reloading them possibly?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:327997
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

> > As an additional note to the above, the "LIVE" server restarts every
>
> > day at approx 0700, and this problem only ever seems to occur at
> some
> > point past 1900-2100 and then seems to continue sporadically until
> > server restart at 0700.
>
>
> Further information.  I have kept an eye on this issue, and it appears
> that the error only occurs soon after the .class files in
> /opt/coldfusion/wwwroot/WEB-INF/cfc-skeletons/.... are modified, which
> indicates to me something is reloading them possibly?

If noone on here has any ideas, does anyone have any thoughts on another place I could post this to try and find a response?

Thanks,

Simon H

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328063
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Paul Vernon]

> > > As an additional note to the above, the "LIVE" server restarts
> every
> >
> > > day at approx 0700, and this problem only ever seems to occur at
> > some
> > > point past 1900-2100 and then seems to continue sporadically until
> > > server restart at 0700.
> >
> >
> > Further information.  I have kept an eye on this issue, and it
> appears
> > that the error only occurs soon after the .class files in
> > /opt/coldfusion/wwwroot/WEB-INF/cfc-skeletons/.... are modified,
> which
> > indicates to me something is reloading them possibly?
>
>
> If noone on here has any ideas, does anyone have any thoughts on
> another place I could post this to try and find a response?

Your original post states that the problems occur in the evening/ overnight
and your CF service is re-started at 7am every morning. With those symptoms
I'd really look at doing some investigation work on your JVM settings and
memory usage. If your memory usage is getting tight after 12 hours, you may
need to change the way your JVM is performing its garbage collection.

When you are invoking the web service, the class files may be being
re-generated and becoming corrupt if there are memory issues which would
then cause your subsequent problems over the rest of the evening.

We are running CF8.01 and handling an average of 100 DB queries per second
(under load we peak at around  on our site and our JVM settings are as
follows:

Min JVM heap MB: 1152 Max: 1152

JVM arguments

-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=384m -Xmn192m
-Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+UseParNewGC


What we found was that our EDEN memory allocation was too low and out
garbage collection wasn't working effectively with the standard settings so
we switched to using two GC collectors and increased the EDEN memory
allocation to 192Mb.

Since we tuned the JVM the server will stay up for months at a time without
any service restarts or memory issues :-).

Paul




~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328064
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

>> > > As an additional note to the above, the "LIVE" server restarts
>
>Your original post states that the problems occur in the evening/ overnight
>and your CF service is re-started at 7am every morning. With those symptoms
>I'd really look at doing some investigation work on your JVM settings and
>memory usage. If your memory usage is getting tight after 12 hours, you may
>need to change the way your JVM is performing its garbage collection.
>
>When you are invoking the web service, the class files may be being
>re-generated and becoming corrupt if there are memory issues which would
>then cause your subsequent problems over the rest of the evening.
>
>We are running CF8.01 and handling an average of 100 DB queries per second
>(under load we peak at around  on our site and our JVM settings are as
>follows:
>
>Min JVM heap MB: 1152 Max: 1152
>
>JVM arguments
>
>-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=384m -Xmn192m
>-Dcoldfusion.rootDir={application.home}/../
>-Dcoldfusion.libPath={application.home}/../lib -XX:+UseConcMarkSweepGC
>-XX:+CMSIncrementalMode -XX:+UseParNewGC
>
>
>What we found was that our EDEN memory allocation was too low and out
>garbage collection wasn't working effectively with the standard settings so
>we switched to using two GC collectors and increased the EDEN memory
>allocation to 192Mb.
>
>Since we tuned the JVM the server will stay up for months at a time without
>any service restarts or memory issues :-).
>
>Paul

Hi Paul,

Thanks for the response, I'll check through the logs on the server and the settings as advised as that explanation does sound viable.  The observations over timing are very rough, it did occur yesterday at approx 14:00 also but that would still be after many hours uptime.

- Simon H

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328066
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

> >> > > As an additional note to the above, the "LIVE" server restarts
> >
> >Your original post states that the problems occur in the evening/
> overnight
> >and your CF service is re-started at 7am every morning. With those
> symptoms
> >I'd really look at doing some investigation work on your JVM settings
> and
> >memory usage. If your memory usage is getting tight after 12 hours,
> you may
> >need to change the way your JVM is performing its garbage collection.
>
> >
> >When you are invoking the web service, the class files may be being
> >re-generated and becoming corrupt if there are memory issues which
> would
> >then cause your subsequent problems over the rest of the evening.
> >
> >We are running CF8.01 and handling an average of 100 DB queries per
> second
> >(under load we peak at around  on our site and our JVM settings are
> as
> >follows:
> >
> >Min JVM heap MB: 1152 Max: 1152
> >
> >JVM arguments
> >
> >-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=384m -Xmn192m
> >-Dcoldfusion.rootDir={application.home}/../
> >-Dcoldfusion.libPath={application.home}/../lib
> -XX:+UseConcMarkSweepGC
> >-XX:+CMSIncrementalMode -XX:+UseParNewGC
> >
> >
> >What we found was that our EDEN memory allocation was too low and
> out
> >garbage collection wasn't working effectively with the standard
> settings so
> >we switched to using two GC collectors and increased the EDEN memory
> >allocation to 192Mb.
> >
> >Since we tuned the JVM the server will stay up for months at a time
> without
> >any service restarts or memory issues :-).
> >
> >Paul
>
> Hi Paul,
>
> Thanks for the response, I'll check through the logs on the server and
> the settings as advised as that explanation does sound viable.  The
> observations over timing are very rough, it did occur yesterday at
> approx 14:00 also but that would still be after many hours uptime.
>
> - Simon H

It looks as though this issue still recurred though I have no new information to go with the issue.  I will probably make some adjustments later on to see if that helps


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328082
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Paul Vernon]

> It looks as though this issue still recurred though I have no new
information
> to go with the issue.  I will probably make some adjustments later on to
see
> if that helps

Try checking the runtime logs as well as the CF logs.

http://www.coldfusionmuse.com/index.cfm/2005/6/23/coldfusion_logs

I particularly find the *servername*-event.log and the *servername*-out.log
useful as they can show up errors that don’t show up in the standard CF
logs.

Paul



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328085
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4

[Simon Hooker]

>> It looks as though this issue still recurred though I have no new
>information
>> to go with the issue.  I will probably make some adjustments later on to
>see
>> if that helps
>
>Try checking the runtime logs as well as the CF logs.
>
>http://www.coldfusionmuse.com/index.cfm/2005/6/23/coldfusion_logs
>
>I particularly find the *servername*-event.log and the *servername*-out.log
>useful as they can show up errors that don’t show up in the standard CF
>logs.
>
>Paul

Hi,

It looks as though this issue was resolved.  The above I think was a total red herring.  There was a task running on another server calling a script that in turn occaisionally (if it thought an unrelated webservice had halted) cleared the trusted cache.  This in turn was able to trigger the CFC to reload perhaps at a bad time causing corruption.

What pointed us to this was that a script we have that triggers clearing the trusted cache (for internal use) was being logged as being called by user agent "coldfusion", which should have never happened.

I expect this was more possible when 2 threads tried to call the same web service however the cache got cleared between them or similar.

Since removing that functionality it has been 100% stable for the last 3 days, so all being well that's the end of that.

Thanks for the help Paul

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:328123
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.4