Artifactory Shutdown in Tomcat Container Causes Server Hang

View: New views
5 Messages — Rating Filter:   Alert me  

Artifactory Shutdown in Tomcat Container Causes Server Hang

by ddenton :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hello all. We have an artifactory instance running in a tomcat container, nested in another parent tomcat container. We can start the instance no problem, but upon shutdown (using a tweaked shutdown.sh supplied with tomcat 6), the entire host server becomes unresponsive and has to be rebooted.

Nothing pertinent in the system logs, but here's the last few entries in my catalina.out file before the hang:

2009-08-06 16:10:21,462 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.repo.index.IndexerService
2009-08-06 16:10:21,463 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.backup.InternalBackupService
2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.security.SecurityServiceInternal
2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.security.AclManager
2009-08-06 16:10:21,471 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.security.UserGroupManager
2009-08-06 16:10:21,472 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.schedule.TaskService
2009-08-06 16:10:21,473 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.worker.WorkerService
2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.engine.InternalDownloadService
2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.repo.service.InternalRepositoryService
2009-08-06 16:10:21,479 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.jcr.md.MetadataService
2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.cache.InternalCacheService
2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.jcr.md.MetadataDefinitionService
2009-08-06 16:10:21,481 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) - Destroying org.artifactory.jcr.JcrService

I'm hoping someone could shed some light on why shutting down a simple tomcat container would hang the server. As far as I can tell, the TCP port designations for the tomcat instance are unique, so I can't see that being the cause. Any help is greatly appreciated!

P.S., I know this might end up being a tomcat issue, but we have other tomcat instances on this server that have been running apps for a while without issue.

Re: Artifactory Shutdown in Tomcat Container Causes Server Hang

by Yoav Landman-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Are you talking about a shutdown of the Tomcat service as part of a system shutdown, or does it also happen when running shutdown.sh manually?
Is it reproducible if you remove artifactory from the deployed webapps?
Do other Tomcat instances on the server use the same shutdown.sh?
What do you mean by "nested in another parent tomcat container"?

On Fri, Aug 7, 2009 at 12:25 AM, ddenton <ddenton@...> wrote:

Hello all. We have an artifactory instance running in a tomcat container,
nested in another parent tomcat container. We can start the instance no
problem, but upon shutdown (using a tweaked shutdown.sh supplied with tomcat
6), the entire host server becomes unresponsive and has to be rebooted.

Nothing pertinent in the system logs, but here's the last few entries in my
catalina.out file before the hang:

2009-08-06 16:10:21,462 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.repo.index.IndexerService
2009-08-06 16:10:21,463 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.backup.InternalBackupService
2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.security.SecurityServiceInternal
2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.security.AclManager
2009-08-06 16:10:21,471 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.security.UserGroupManager
2009-08-06 16:10:21,472 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.schedule.TaskService
2009-08-06 16:10:21,473 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.worker.WorkerService
2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.engine.InternalDownloadService
2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.repo.service.InternalRepositoryService
2009-08-06 16:10:21,479 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.jcr.md.MetadataService
2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.cache.InternalCacheService
2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.jcr.md.MetadataDefinitionService
2009-08-06 16:10:21,481 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
Destroying org.artifactory.jcr.JcrService

I'm hoping someone could shed some light on why shutting down a simple
tomcat container would hang the server. As far as I can tell, the TCP port
designations for the tomcat instance are unique, so I can't see that being
the cause. Any help is greatly appreciated!

P.S., I know this might end up being a tomcat issue, but we have other
tomcat instances on this server that have been running apps for a while
without issue.
--
View this message in context: http://www.nabble.com/Artifactory-Shutdown-in-Tomcat-Container-Causes-Server-Hang-tp24854735p24854735.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory Shutdown in Tomcat Container Causes Server Hang

by ddenton :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Sorry if I wasn't clear in my first post. We have multiple tomcat containers in a subdirectory of another tomcat container. Each has its own shutdown and startup scripts that reference the shutdown and startup scripts in the parent directory. We do it this way instead of having multiple webapps in one container to avoid taking down all of them all if we shut down one. So in this case, artifactory is the only webapp in the container in question.

The hang occurs when shutting down the individual tomcat instance for artifactory. Sometimes, the whole server hangs within seconds of shutting down the artifactory instance, and sometimes it takes as much as 30 seconds. I've found that killing the process with -9 avoids the hang, but obviously that's not best practice. Even a kill without the -9 results in a hang.

Thanks for your reply...

Yoav Landman-3 wrote:
Are you talking about a shutdown of the Tomcat service as part of a system
shutdown, or does it also happen when running shutdown.sh manually?
Is it reproducible if you remove artifactory from the deployed webapps?
Do other Tomcat instances on the server use the same shutdown.sh?
What do you mean by "nested in another parent tomcat container"?

On Fri, Aug 7, 2009 at 12:25 AM, ddenton <ddenton@remitpro.com> wrote:

>
> Hello all. We have an artifactory instance running in a tomcat container,
> nested in another parent tomcat container. We can start the instance no
> problem, but upon shutdown (using a tweaked shutdown.sh supplied with
> tomcat
> 6), the entire host server becomes unresponsive and has to be rebooted.
>
> Nothing pertinent in the system logs, but here's the last few entries in my
> catalina.out file before the hang:
>
> 2009-08-06 16:10:21,462 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.repo.index.IndexerService
> 2009-08-06 16:10:21,463 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.backup.InternalBackupService
> 2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.security.SecurityServiceInternal
> 2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.security.AclManager
> 2009-08-06 16:10:21,471 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.security.UserGroupManager
> 2009-08-06 16:10:21,472 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.schedule.TaskService
> 2009-08-06 16:10:21,473 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.worker.WorkerService
> 2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.engine.InternalDownloadService
> 2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.repo.service.InternalRepositoryService
> 2009-08-06 16:10:21,479 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.jcr.md.MetadataService
> 2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.cache.InternalCacheService
> 2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.jcr.md.MetadataDefinitionService
> 2009-08-06 16:10:21,481 [INFO ] (o.a.s.ArtifactoryApplicationContext:156) -
> Destroying org.artifactory.jcr.JcrService
>
> I'm hoping someone could shed some light on why shutting down a simple
> tomcat container would hang the server. As far as I can tell, the TCP port
> designations for the tomcat instance are unique, so I can't see that being
> the cause. Any help is greatly appreciated!
>
> P.S., I know this might end up being a tomcat issue, but we have other
> tomcat instances on this server that have been running apps for a while
> without issue.
> --
> View this message in context:
> http://www.nabble.com/Artifactory-Shutdown-in-Tomcat-Container-Causes-Server-Hang-tp24854735p24854735.html
> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory Shutdown in Tomcat Container Causes Server Hang

by Yoav Landman-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Sounds like something very specific to your configuration.
Can you take a thread dump (use jstack) and post it or send it privately?

On Fri, Aug 7, 2009 at 5:04 PM, ddenton <ddenton@...> wrote:

Sorry if I wasn't clear in my first post. We have multiple tomcat containers
in a subdirectory of another tomcat container. Each has its own shutdown and
startup scripts that reference the shutdown and startup scripts in the
parent directory. We do it this way instead of having multiple webapps in
one container to avoid taking down all of them all if we shut down one. So
in this case, artifactory is the only webapp in the container in question.

The hang occurs when shutting down the individual tomcat instance for
artifactory. Sometimes, the whole server hangs within seconds of shutting
down the artifactory instance, and sometimes it takes as much as 30 seconds.
I've found that killing the process with -9 avoids the hang, but obviously
that's not best practice. Even a kill without the -9 results in a hang.

Thanks for your reply...


Yoav Landman-3 wrote:
>
> Are you talking about a shutdown of the Tomcat service as part of a system
> shutdown, or does it also happen when running shutdown.sh manually?
> Is it reproducible if you remove artifactory from the deployed webapps?
> Do other Tomcat instances on the server use the same shutdown.sh?
> What do you mean by "nested in another parent tomcat container"?
>
> On Fri, Aug 7, 2009 at 12:25 AM, ddenton <ddenton@...> wrote:
>
>>
>> Hello all. We have an artifactory instance running in a tomcat container,
>> nested in another parent tomcat container. We can start the instance no
>> problem, but upon shutdown (using a tweaked shutdown.sh supplied with
>> tomcat
>> 6), the entire host server becomes unresponsive and has to be rebooted.
>>
>> Nothing pertinent in the system logs, but here's the last few entries in
>> my
>> catalina.out file before the hang:
>>
>> 2009-08-06 16:10:21,462 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.repo.index.IndexerService
>> 2009-08-06 16:10:21,463 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.backup.InternalBackupService
>> 2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.security.SecurityServiceInternal
>> 2009-08-06 16:10:21,470 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.security.AclManager
>> 2009-08-06 16:10:21,471 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.security.UserGroupManager
>> 2009-08-06 16:10:21,472 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.schedule.TaskService
>> 2009-08-06 16:10:21,473 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.worker.WorkerService
>> 2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.engine.InternalDownloadService
>> 2009-08-06 16:10:21,478 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.repo.service.InternalRepositoryService
>> 2009-08-06 16:10:21,479 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.jcr.md.MetadataService
>> 2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.cache.InternalCacheService
>> 2009-08-06 16:10:21,480 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.jcr.md.MetadataDefinitionService
>> 2009-08-06 16:10:21,481 [INFO ] (o.a.s.ArtifactoryApplicationContext:156)
>> -
>> Destroying org.artifactory.jcr.JcrService
>>
>> I'm hoping someone could shed some light on why shutting down a simple
>> tomcat container would hang the server. As far as I can tell, the TCP
>> port
>> designations for the tomcat instance are unique, so I can't see that
>> being
>> the cause. Any help is greatly appreciated!
>>
>> P.S., I know this might end up being a tomcat issue, but we have other
>> tomcat instances on this server that have been running apps for a while
>> without issue.
>> --
>> View this message in context:
>> http://www.nabble.com/Artifactory-Shutdown-in-Tomcat-Container-Causes-Server-Hang-tp24854735p24854735.html
>> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
>> 30-Day
>> trial. Simplify your report design, integration and deployment - and
>> focus
>> on
>> what you do best, core application coding. Discover what's new with
>> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
>> _______________________________________________
>> Artifactory-users mailing list
>> Artifactory-users@...
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@...
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

--
View this message in context: http://www.nabble.com/Artifactory-Shutdown-in-Tomcat-Container-Causes-Server-Hang-tp24854735p24865463.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: Artifactory Shutdown in Tomcat Container Causes Server Hang

by ddenton :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Well, it appears I have a tomcat6/JDK 1.6.0_20 issue, not one with artifactory. My server hangs when I attempt to stop any tomcat container as long as I'm using JDK 1.6.0_20. If I roll back to JDK 1.5.0_12, then everything works fine.

Thanks to all who replied, and while I understand it's off topic for this mailing list, if anyone could share any similar experiences with tomcat6 and JDK6 doing this to them, I'd appreciate it.

Yoav Landman-3 wrote:
Sounds like something very specific to your configuration.
Can you take a thread dump (use jstack) and post it or send it privately?