Cluster stops responding, even after server reboot

[Jimmy Gilley]

We are using MX7 and have an issue with a cluster we made.  

It has happen approx. 4 times in two weeks.

What it does is the cluster stops responding and when you reboot the server the cluster still isn't responding.  You have to remove the cluster, and then re-add the instances into the cluster for it to start working again.  

i am fairly new to ColdFusion so I apologize for my lack of knowledge in some areas.  

thanks in advance for your help on this issue.  Hopefully we will be able to get this up and working consistently.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6696
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[listman]

Are the individual instances responsive when the cluster hangs?

Do you have metrics logging turned on?  What do they show?

Usually, the instances stop responding because they fill up with active
threads, and the instances seem to have hanged, but they are actually
still accepting requests, just that they go straight into a queue.

Try not rebooting the server, just shutting down ALL instances, and
turning them all on.  Make sure to SHUT DOWN ALL INSTANCES FIRST.

Also, when you reboot the servers, not all instances might be up, as
JRUN takes a while to start up, especially with multiple instances, so
it may seem that the cluster has died, but is in fact, still starting up.

Jimmy Gilley wrote:

> We are using MX7 and have an issue with a cluster we made.  
>
> It has happen approx. 4 times in two weeks.
>
> What it does is the cluster stops responding and when you reboot the server the cluster still isn't responding.  You have to remove the cluster, and then re-add the instances into the cluster for it to start working again.  
>
> i am fairly new to ColdFusion so I apologize for my lack of knowledge in some areas.  
>
> thanks in advance for your help on this issue.  Hopefully we will be able to get this up and working consistently.
>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6697
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Mark Ireland-2]

This is where CF8 Alerts are so cool.

> Subject: Re: Cluster stops responding, even after server reboot
> From: lists@...
> To: cf-server@...
> Date: Tue, 9 Dec 2008 17:15:37 -0800
>
> Are the individual instances responsive when the cluster hangs?
>
> Do you have metrics logging turned on?  What do they show?
>
> Usually, the instances stop responding because they fill up with active
> threads, and the instances seem to have hanged, but they are actually
> still accepting requests, just that they go straight into a queue.
>
> Try not rebooting the server, just shutting down ALL instances, and
> turning them all on.  Make sure to SHUT DOWN ALL INSTANCES FIRST.
>
> Also, when you reboot the servers, not all instances might be up, as
> JRUN takes a while to start up, especially with multiple instances, so
> it may seem that the cluster has died, but is in fact, still starting up.
>
> Jimmy Gilley wrote:
> > We are using MX7 and have an issue with a cluster we made.  
> >
> > It has happen approx. 4 times in two weeks.
> >
> > What it does is the cluster stops responding and when you reboot the server the cluster still isn't responding.  You have to remove the cluster, and then re-add the instances into the cluster for it to start working again.  
> >
> > i am fairly new to ColdFusion so I apologize for my lack of knowledge in some areas.  
> >
> > thanks in advance for your help on this issue.  Hopefully we will be able to get this up and working consistently.
> >
> >
> >
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6698
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jochem van Dieten]

On Tue, Dec 9, 2008 at 10:20 PM, Jimmy Gilley wrote:
> What it does is the cluster stops responding and when you reboot the server the cluster still isn't responding.  You have to remove the cluster, and then re-add the instances into the cluster for it to start working again.

Are your webserver and CF instances on the same server? Have you tried
restarting the webserver instead of the CF instances?

If this isn't resolved yet and happens again, could you try the following:
- shut down the webserver and all CF instances
- start your CF instances
- wait until they are fully started (test this through the builtin webserver)
- then start the webserver with the real site.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6699
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

The problem is that sometimes the server crashes and so a pristine shut down is not an option.  I can't believe that the clusters not working after it comes back up is something it is supposed to do.  

Also those testing things are things I will try but it will be tough because these sites have heavy traffic on them.  This being the reseason if the server goes down and we reboot it and the clusters do not respond is an issue.  This happens sometimes at 2-3 in the mourning where it is impossible to remove the cluster and re-add it to make the webserver work again.

any thoughts?

Jochem van Dieten wrote:

On Tue, Dec 9, 2008 at 10:20 PM, Jimmy Gilley wrote:
> What it does is the cluster stops responding and when you reboot the server the cluster still isn't responding.  You have to remove the cluster, and then re-add the instances into the cluster for it to start working again.

Are your webserver and CF instances on the same server? Have you tried
restarting the webserver instead of the CF instances?

If this isn't resolved yet and happens again, could you try the following:
- shut down the webserver and all CF instances
- start your CF instances
- wait until they are fully started (test this through the builtin webserver)
- then start the webserver with the real site.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6699
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jochem van Dieten]

On Tue, Jan 6, 2009 at 3:56 PM, Jimmy Gilley wrote:
> The problem is that sometimes the server crashes and so a pristine shut down
> is not an option.  I can't believe that the clusters not working after it
> comes back up is something it is supposed to do.

If the server comes back up from a hard reboot all services are
starting up at the same time. That startup process has two
communicating sides to it: JRun and the webserver. Part of the
webserver is the JRun webserver connector, a library that is loaded
into the webserver. When that library is initialized it tries to
connect back to JRun retrieve some configuration information. This
bootstrap will only work if JRun starts before the webserver. So the
problem of the cluster not working may be alleviated by configuring
the webserver to start after JRun.
This also explains why in some cases you need to reset the webserver
to propagate changes instead of resetting CF/JRun.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6719
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

Awesome!  Now that sounds more like we are heading somewhere.  Again I am new to coldfusion, how do I set up the webserver to start after the JRun?

Jochem van Dieten wrote:

On Tue, Jan 6, 2009 at 3:56 PM, Jimmy Gilley wrote:
> The problem is that sometimes the server crashes and so a pristine shut down
> is not an option.  I can't believe that the clusters not working after it
> comes back up is something it is supposed to do.

If the server comes back up from a hard reboot all services are
starting up at the same time. That startup process has two
communicating sides to it: JRun and the webserver. Part of the
webserver is the JRun webserver connector, a library that is loaded
into the webserver. When that library is initialized it tries to
connect back to JRun retrieve some configuration information. This
bootstrap will only work if JRun starts before the webserver. So the
problem of the cluster not working may be alleviated by configuring
the webserver to start after JRun.
This also explains why in some cases you need to reset the webserver
to propagate changes instead of resetting CF/JRun.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6719
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Dave Watts]

> Awesome!  Now that sounds more like we are heading somewhere.  Again I am new
> to coldfusion, how do I set up the webserver to start after the JRun?

If you're using Windows, you can set a registry key called
DependOnService to make the web server require JRun to start first.
Google DependOnService for more information. If you're using Unix, you
can specify the startup order of daemons by changing the numbers
within the names of the startup scripts in the appropriate runlevel
directory as described here:

http://www.cis.ufl.edu/~jnw/SysAdminfa04/Lectures/03.html

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6721
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

I am using Windows.  So I just set that to dependonservice to the JRun CFusion server?  or do I also have to have it wait on the jrun admin server?

Sorry to be a bother, I REALLY do appreciate this!

Maybe give the exact register layout here so that others can search it.

service name/DependOnService --->service name

Dave Watts wrote:

> Awesome!  Now that sounds more like we are heading somewhere.  Again I am new
> to coldfusion, how do I set up the webserver to start after the JRun?

If you're using Windows, you can set a registry key called
DependOnService to make the web server require JRun to start first.
Google DependOnService for more information. If you're using Unix, you
can specify the startup order of daemons by changing the numbers
within the names of the startup scripts in the appropriate runlevel
directory as described here:

http://www.cis.ufl.edu/~jnw/SysAdminfa04/Lectures/03.html

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6721
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Dave Watts]

> I am using Windows.  So I just set that to dependonservice to the JRun
> CFusion server?  or do I also have to have it wait on the jrun admin server?

You don't even need to run the JRun Admin server. It's just used to
manage your JRun servers. You will need to set DependOnService for the
web server so that it starts after any cluster members on that
machine. If you just have a default multiserver install of CF
Enterprise without any additional CF instances, that would mean just
the JRun CFusion Server.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6723
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

That was what I thought.  So I have multiserver set up with 3 instances.  So I would set the DependOnService for W3SVC (since I am Windows) to the three instances plus the main cold fusion instance.

Dave Watts wrote:

> I am using Windows.  So I just set that to dependonservice to the JRun
> CFusion server?  or do I also have to have it wait on the jrun admin server?

You don't even need to run the JRun Admin server. It's just used to
manage your JRun servers. You will need to set DependOnService for the
web server so that it starts after any cluster members on that
machine. If you just have a default multiserver install of CF
Enterprise without any additional CF instances, that would mean just
the JRun CFusion Server.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6723
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Dave Watts]

> That was what I thought.  So I have multiserver set up with 3 instances.  So
> I would set the DependOnService for W3SVC (since I am Windows) to the three
> instances plus the main cold fusion instance.

Yes, that would work, although you don't even really need to include
the main CF instance, since it's not part of the cluster. In fact, you
can turn that instance off if it's not serving any applications, and
just turn it on when you need to use Instance Manager or Cluster
Manager.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6725
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jochem van Dieten]

On Tue, Jan 6, 2009 at 5:11 PM, Jimmy Gilley wrote:
> I am using Windows.  So I just set that to dependonservice to the JRun
> CFusion server?

You set IIS to wait for the JRun instances.


> or do I also have to have it wait on the jrun admin server?

You should consider disabling the JRun Admin Server and never touching
it again. You shouldn't need it any more once you have configured JRun
and a common problem is that people use both the JRun Admin and the
Windows Services Control Panel and they start interfering with each
other.

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6726
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

Jochem van Dieten wrote:

You set IIS to wait for the JRun instances.

So you set teh IISAdmin or W3SVC?

[Dave Watts]

> So you set teh IISAdmin or W3SVC?

W3SVC.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6728
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

Dave Watts wrote:

W3SVC.

Ok I have done this, and I am still getting:

 Server Error
Either the Macromedia application server is unreachable or it does not have a mapping to process this request.

After a reboot.  The instances are started, but it still will not run the website.

I then removed it using web configuration tool and re-added it and now the site is working fine...

[Jochem van Dieten]

On Tue, Jan 6, 2009 at 6:16 PM, Jimmy Gilley wrote:
>
> Ok I have done this, and I am still getting:
>
>  Server Error
> Either the Macromedia application server is unreachable or it does not have
> a mapping to process this request.

Have you confirmed (through the builtin webserver) that the JRun
instances are running?

Jochem

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6731
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Dave Watts]

> Ok I have done this, and I am still getting:
>
>  Server Error
> Either the Macromedia application server is unreachable or it does not have
> a mapping to process this request.
>
> After a reboot.  The instances are started, but it still will not run the
> website.

It takes a while for the CF services to be responsive after they've
been started. However, the clustering should work fine once they are
responsive.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/

Fig Leaf Software provides the highest caliber vendor-authorized
instruction at our training centers in Washington DC, Atlanta,
Chicago, Baltimore, Northern Virginia, or on-site at your location.
Visit http://training.figleaf.com/ for more information!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date
Get the Free Trial
http://ad.doubleclick.net/clk;207172674;29440083;f

Archive: http://www.houseoffusion.com/groups/cf-server/message.cfm/messageid:6730
Subscription: http://www.houseoffusion.com/groups/cf-server/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=17837.14401.10

[Jimmy Gilley]

Dave Watts wrote:

It takes a while for the CF services to be responsive after they've
been started. However, the clustering should work fine once they are
responsive.

How much time would you expect?  Once the server came up I waited about 5 minutes or so before I finally removed and readded the cluster.  which immediately solved the problem.

[Jimmy Gilley]

We are still having this issue.

A little more information I have dug up on it.

It appears we have a problem with the W3SVC service restarting due to an Application Pool hang.  It goes through and automatically restarts.  But when it does we still are unable to hit any of our coldfusion websites.

So we try to remove the cluster, and re add it and we get nothing.  

We then have to restart the server, then re add the the cluster and it starts working fine.  

I can't seem to find anything in the logs about it.

Please help!!  This is horrible because we have probes on the server but they seem to be ok, because they never e-mail us letting us know anything is down so we have a significant amount of downtime.