Occasional NoClassDefFoundError, but disappears after couple of restarts

[arjan.tijms]

Hi,

I'm running Tomcat 6.0.16 on a 32 bit Debian Etch using Java 6 update
12. The problem I have is that occasionally after I restart Tomcat, the
startup fails with NoClassDefFoundError or ClassNotFoundException for
every servlet listener that's being added from web.xml.

If I restart Tomcat a couple of times, the  exceptions mysteriously
vanish and startup succeeds. I've seen the exact same exceptions
happening when starting up another Tomcat instance I manage, which is a
Tomcat 6.0.18 on a 64 bit Debian Lenny also using Java 6 update 12.

The Tomcat 6.0.16 instance produces the following strack trace when
starting up:

Jul 23, 2009 11:32:41 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path:
/opt/jdk1.6.0_12/jre/lib/i386/server:/opt/jdk1.6.0_12/jre/lib/i386:/opt/jdk1.6.0_12/jre/../lib/i386:/usr/java/packages/l

ib/i386:/lib:/usr/lib
Jul 23, 2009 11:32:41 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 506 ms
Jul 23, 2009 11:32:41 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 23, 2009 11:32:41 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener
"com.xmx.cyberdyme.system.settings.ApplicationSettings" is already
configured for this context. The duplicate definition has been ignored.
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext
addApplicationListener
INFO: The listener "org.quartz.ee.servlet.QuartzInitializerListener" is
already configured for this context. The duplicate definition has been
ignored.
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext
listenerStart
SEVERE: Exception sending context initialized event to listener instance
of class com.xmx.cyberdyme.system.settings.ApplicationSettings
java.lang.NoClassDefFoundError:
com/xmx/cyberdyme/system/settings/AbstractWebApplicationSettings
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1275)

       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1206)

       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       at java.lang.Class.forName0(Native Method)
       at java.lang.Class.forName(Class.java:169)
       at
com.xmx.cyberdyme.system.settings.ApplicationSettings.contextInitialized(ApplicationSettings.java:52)

       at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)

       at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4350)
       at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)

       at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
       at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
       at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:924)
       at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:887)

       at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
       at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1147)
       at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
       at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)

       at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
       at
org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
       at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
       at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
       at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
       at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
       at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
       at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.lang.ClassNotFoundException:
com.xmx.cyberdyme.system.settings.AbstractWebApplicationSettings
       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       ... 41 more
Jul 23, 2009 11:32:42 PM org.quartz.simpl.SimpleThreadPool initialize
INFO: Job execution threads will use class loader of thread: main
Jul 23, 2009 11:32:42 PM org.quartz.core.QuartzScheduler <init>
INFO: Quartz Scheduler v.1.5.2 created.
Jul 23, 2009 11:32:42 PM org.quartz.plugins.xml.JobInitializationPlugin
initialize
INFO: Registering Quartz Job Initialization Plug-in.
Jul 23, 2009 11:32:42 PM org.quartz.simpl.RAMJobStore initialize
INFO: RAMJobStore initialized.
Jul 23, 2009 11:32:42 PM org.quartz.impl.StdSchedulerFactory instantiate
INFO: Quartz scheduler 'DefaultQuartzScheduler' initialized from the
specified file : 'quartz.properties' from the class resource path.
Jul 23, 2009 11:32:42 PM org.quartz.impl.StdSchedulerFactory instantiate
INFO: Quartz scheduler version: 1.5.2
Jul 23, 2009 11:32:42 PM org.quartz.xml.JobSchedulingDataProcessor
processFile
INFO: Parsing XML file: com/xmx/cyberdyme/resources/jobs/systemJobs.xml
with systemId: com/xmx/cyberdyme/resources/jobs/systemJobs.xml
validating: false validating schema:
jar:file:/cyberdyme/www/moon_2.0.4_u4/WEB-INF/lib/quartz-1.5.2.jar!/org/quartz/xml/job_scheduling_data_1_5

.xsd
Jul 23, 2009 11:32:42 PM org.apache.commons.digester.Digester endElement
SEVERE: End event threw error
java.lang.NoClassDefFoundError:
com/xmx/cyberdyme/system/scheduling/AbstractSimpleJob
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1275)

       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1206)

       at
org.apache.commons.beanutils.converters.ClassConverter.convert(ClassConverter.java:118)

       at
org.apache.commons.beanutils.ConvertUtilsBean.convert(ConvertUtilsBean.java:428)

       at
org.apache.commons.beanutils.BeanUtilsBean.setProperty(BeanUtilsBean.java:1002)

       at
org.apache.commons.beanutils.BeanUtils.setProperty(BeanUtils.java:313)
       at
org.apache.commons.digester.BeanPropertySetterRule.end(BeanPropertySetterRule.java:197)

       at
org.apache.commons.digester.Digester.endElement(Digester.java:1067)
       at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1774)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2930)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)

       at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)

       at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)

       at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)

       at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)

       at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)

       at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)

       at org.apache.commons.digester.Digester.parse(Digester.java:1572)
       at
org.quartz.xml.JobSchedulingDataProcessor.processFile(JobSchedulingDataProcessor.java:430)

       at
org.quartz.xml.JobSchedulingDataProcessor.processFileAndScheduleJobs(JobSchedulingDataProcessor.java:499)

       at
org.quartz.xml.JobSchedulingDataProcessor.processFileAndScheduleJobs(JobSchedulingDataProcessor.java:485)

       at
org.quartz.plugins.xml.JobInitializationPlugin.processFile(JobInitializationPlugin.java:401)

       at
org.quartz.plugins.xml.JobInitializationPlugin.start(JobInitializationPlugin.java:375)

       at
org.quartz.core.QuartzScheduler.startPlugins(QuartzScheduler.java:1919)
       at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:397)
       at org.quartz.impl.StdScheduler.start(StdScheduler.java:147)
       at
org.quartz.ee.servlet.QuartzInitializerListener.contextInitialized(QuartzInitializerListener.java:145)

       at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)

       at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4350)
       at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)

       at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
       at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
       at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:924)
       at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:887)

       at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
       at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1147)
       at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
       at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)

       at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
Caused by: java.lang.ClassNotFoundException:
com.xmx.cyberdyme.system.scheduling.AbstractSimpleJob
       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)

Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext
listenerStart
SEVERE: Exception sending context initialized event to listener instance
of class org.quartz.ee.servlet.QuartzInitializerListener
java.lang.NoClassDefFoundError:
com/xmx/cyberdyme/system/scheduling/AbstractSimpleJob
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1275)

       at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1206)

       at
org.apache.commons.beanutils.converters.ClassConverter.convert(ClassConverter.java:118)

       at
org.apache.commons.beanutils.ConvertUtilsBean.convert(ConvertUtilsBean.java:428)

       at
org.apache.commons.beanutils.BeanUtilsBean.setProperty(BeanUtilsBean.java:1002)

       at
org.apache.commons.beanutils.BeanUtils.setProperty(BeanUtils.java:313)
       at
org.apache.commons.digester.BeanPropertySetterRule.end(BeanPropertySetterRule.java:197)

       at
org.apache.commons.digester.Digester.endElement(Digester.java:1067)
       at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:601)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1774)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2930)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)

       at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)

       at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)

       at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)

       at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)

       at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)

       at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)

       at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)

       at org.apache.commons.digester.Digester.parse(Digester.java:1572)
       at
org.quartz.xml.JobSchedulingDataProcessor.processFile(JobSchedulingDataProcessor.java:430)

       at
org.quartz.xml.JobSchedulingDataProcessor.processFileAndScheduleJobs(JobSchedulingDataProcessor.java:499)

       at
org.quartz.xml.JobSchedulingDataProcessor.processFileAndScheduleJobs(JobSchedulingDataProcessor.java:485)

       at
org.quartz.plugins.xml.JobInitializationPlugin.processFile(JobInitializationPlugin.java:401)

       at
org.quartz.plugins.xml.JobInitializationPlugin.start(JobInitializationPlugin.java:375)

       at
org.quartz.core.QuartzScheduler.startPlugins(QuartzScheduler.java:1919)
       at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:397)
       at org.quartz.impl.StdScheduler.start(StdScheduler.java:147)
       at
org.quartz.ee.servlet.QuartzInitializerListener.contextInitialized(QuartzInitializerListener.java:145)

       at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)

       at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4350)
       at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)

       at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
       at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
       at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:924)
       at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:887)

       at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
       at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1147)
       at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
       at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)

       at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
       at
org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
       at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
       at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
       at
org.apache.catalina.core.StandardService.start(StandardService.java:516)
       at
org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
       at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
       at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.lang.ClassNotFoundException:
com.xmx.cyberdyme.system.scheduling.AbstractSimpleJob
       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       ... 64 more
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [] startup failed due to previous errors
Jul 23, 2009 11:32:42 PM org.quartz.core.QuartzScheduler shutdown
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
Jul 23, 2009 11:32:42 PM org.quartz.core.QuartzScheduler standby
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
Jul 23, 2009 11:32:42 PM org.quartz.core.QuartzScheduler shutdown
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext
listenerStop

As mentioned, the exceptions go away after a couple of restarts, which
may indicate a race condition of some sort. The exceptions seem to
happen exclusively on the production servers and never on my localhost
(where I start up Tomcat via Eclipse).The main difference between my
localhost and the production server is that my localhost uses a near
stock Tomcat, while the production servers have a modified server.xml,
and an Apache web server in front of it. The server.xml of one of those
production servers is as follows:

<Server port="8005" shutdown="SHUTDOWN">
 <!--APR library loader. Documentation at /docs/apr.html -->
 <Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
 <!--Initialize Jasper prior to webapps are loaded. Documentation at
/docs/jasper-howto.html -->
 <Listener className="org.apache.catalina.core.JasperListener" />
 <!-- JMX Support for the Tomcat server. Documentation at
/docs/non-existent.html -->
 <Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
 <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

 <GlobalNamingResources>
   <Resource name="UserDatabase" auth="Container"
             type="org.apache.catalina.UserDatabase"
             description="User database that can be updated and saved"
             factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
             pathname="conf/tomcat-users.xml" />
 </GlobalNamingResources>

 <Service name="Catalina">
   <Connector      port="8009"
                   protocol="AJP/1.3"
                   redirectPort="443"
                   backlog="200"
                   maxSpareThreads="50"
                   maxThreads="800"
                   minSpareThreads="10"
                   connectionTimeout="20000"
              />
   <Engine name="Catalina" defaultHost="localhost">
     <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
     <Host name="localhost"  appBase="webapps"
workDir="work/Catalina/cyberdyme"
           unpackWARs="true" autoDeploy="true"
           xmlValidation="false" xmlNamespaceAware="false">

       <Valve  className="org.apache.catalina.valves.AccessLogValve"
               directory="logs"
               pattern="combined"
               prefix="cybername-access"
               resolveHosts="false"
               suffix=".log"
               rotatable="true"
               fileDateFormat="-yyyy.MM.dd.HH"/>
     </Host>
   </Engine>
 </Service>
</Server>

Does anyone have any ideas how to get rid of these exceptions? I've come
to the point where I barely dare to restart my Tomcat instances anymore,
for fear of them not coming back up.

Thanks in advance for all help.

--
It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.

[Caldarale, Charles R]

> From: M4N - Arjan Tijms [mailto:arjan.tijms@...]
> Subject: Occasional NoClassDefFoundError, but disappears after couple
> of restarts
>
> The problem I have is that occasionally after I restart Tomcat, the
> startup fails with NoClassDefFoundError or ClassNotFoundException for
> every servlet listener that's being added from web.xml.

This kind of error can be caused by having the same class(es) in multiple locations in a given branch of the classloader tree, such as in both a webapp's WEB-INF/lib and in Tomcat's lib directory (or on the system classpath).  Have you checked that?

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[arjan.tijms]

Hi,
> This kind of error can be caused by having the same class(es) in multiple locations in a given
> branch of the classloader tree, such as in both a webapp's WEB-INF/lib and in Tomcat's lib
> directory (or on the system classpath).  Have you checked that?
Well, that's indeed a good suggestion. We added 3 libraries to [tomcat
install dir]/lib, namely activation.jar, mail.jar and
postgresql-8.3-603.jdbc3.jar. As it happens the web application's
WEB-INF/lib does contain a postgresql-8.3-603.jdbc4.jar. This last .jar
should of course not be there.

I have indeed seen applications on Jboss AS fail due to duplicate class
definitions. The odd thing is that on Jboss AS it would fail both on
localhost and on the production server, but in the case of Tomcat it
only fails on the production server. Also, these (nearly) duplicate jar
files seem to have been there for some time and restarts only started
recently to fail on a regular basis. A recent upgrade from JDK 6 update
6 to update 12 seems to be most likely candidate for causing the sudden
change in behavior.

I'll remove the duplicate .jar, and hopefully the next restarts will
happen smoothly again.

Thanks again for your reply.

Arjan

--
It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[arjan.tijms]

Hi,

Caldarale, Charles R wrote:
>> The problem I have is that occasionally after I restart Tomcat, the
>> startup fails with NoClassDefFoundError or ClassNotFoundException for
>> every servlet listener that's being added from web.xml.
>>    
>
> This kind of error can be caused by having the same class(es) in multiple locations in a given branch of the classloader tree, such as in both a webapp's WEB-INF/lib and in Tomcat's lib directory (or on the system classpath).  Have you checked that?
>  
I removed the obvious suspect, a duplicate postgresql driver that
appeared in both WEB-INF/lib  and [tomcat install dir]/lib.
Unfortunately this did not seem to help. Last night I needed to restart
the production server for a configuration change and I needed 22(!)
restarts for Tomcat to start up correctly.

It really seems that there's a race condition going on somehow. I mean,
if it was a structural problem then Tomcat would simply never start up,
wouldn't it?

These are the libraries the web app currently uses:

In WEB-INF/lib
slf4j-jdk14-1.5.2.jar
slf4j-api-1.5.2.jar
regexp.jar
quartz-1.5.2.jar
jnp-client.jar
jbosssx-client.jar
jbosssx-as-client.jar
jboss-security-spi.jar
jboss-remoting.jar
jboss-logging-spi.jar
jboss-j2ee.jar
jboss-integration.jar
jboss-ejb3-security-client.jar
jboss-ejb3-proxy-client.jar
jboss-ejb3-core-client.jar
jboss-ejb3-common-client.jar
jboss-common-core.jar
jboss-client.jar
jboss-aspect-jdk50-client.jar
jboss-aop-client.jar
hibernate-annotations.jar
concurrent.jar
commons-logging-1.0.4.jar
commons-digester-1.6.jar
commons-collections-3.1.jar
commons-beanutils-1.7.0.jar

In [tomcat install dir]/lib
mail.jar
activation.jar
postgresql-8.3-603.jdbc3.jar

The Jboss libraries are the Jboss client libraries and are needed to
consume EJBs from a remote Jboss AS instance. Maybe there are some
classes somewhere that conflict with those of Tomcat, but if that would
be true it's still hard to understand why starting up fails in
production but never on a local workstation.

Anyone has any idea?

Grtz,
Arjan

--
It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.

[markt-2]

M4N - Arjan Tijms wrote:
> I removed the obvious suspect, a duplicate postgresql driver that
> appeared in both WEB-INF/lib  and [tomcat install dir]/lib.
> Unfortunately this did not seem to help. Last night I needed to restart
> the production server for a configuration change and I needed 22(!)
> restarts for Tomcat to start up correctly.
>
> It really seems that there's a race condition going on somehow. I mean,
> if it was a structural problem then Tomcat would simply never start up,
> wouldn't it?

A reasonable conclusion.

> The Jboss libraries are the Jboss client libraries and are needed to
> consume EJBs from a remote Jboss AS instance. Maybe there are some
> classes somewhere that conflict with those of Tomcat, but if that would
> be true it's still hard to understand why starting up fails in
> production but never on a local workstation.

Does the production box have more cores than the dev box? If there was a
race condition you would be more likely to see it with more cores.

> Anyone has any idea?

There was a sync issue in the webappclassloader fixed in 6.0.19.
Although the symptoms were different, it might be worth trying 6.0.20.

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[A. Rothman]

A couple months ago I had a slightly different setting but with the same
strange result - NoClassDefFoundErrors which occurred on a particular
jsp page in my app, but only if it was the first jsp accessed following
a restart (and remained in the error state until the next restart). If
it was not the first jsp accessed, then everything came up fine (with
nothing in the configuration or app changing in between). I also figured
it's some classloader bug in Tomcat, not entirely a race condition
(since it did happen consistently), but something that manifests itself
depending on the order of class loading.


I never did find out for sure what the problem was (maybe it's the sync
issue Mark refers to below), but I stumbled upon a workaround, which I
do not quite understand, which fixed this for good: I changed the order
of the first two statements in the jsp, the first was one that accessed
java.net.URI (a system class) and the second the session bean
declaration (a custom app-specific class). Although the two statements
were unrelated to each other and their order should not have made any
difference, this solved the problem (switching them back did reintroduce
the error, consistently). I still suspect the problem was with the order
of classloading.


So, not sure if this will help, but in the off chance that your issue is
related to the one I was experiencing, you might want to try fiddling
with the order in which classes are accessed/loaded by Tomcat (either
via their xml reference, or their actual content's first statements) and
hopefully something will work. Yes, it sounds strange, but it worked for
me in a reproducible manner.


Good luck :-)


Amichai


Mark Thomas wrote:

> M4N - Arjan Tijms wrote:
>  
>> I removed the obvious suspect, a duplicate postgresql driver that
>> appeared in both WEB-INF/lib  and [tomcat install dir]/lib.
>> Unfortunately this did not seem to help. Last night I needed to restart
>> the production server for a configuration change and I needed 22(!)
>> restarts for Tomcat to start up correctly.
>>
>> It really seems that there's a race condition going on somehow. I mean,
>> if it was a structural problem then Tomcat would simply never start up,
>> wouldn't it?
>>    
>
> A reasonable conclusion.
>
>  
>> The Jboss libraries are the Jboss client libraries and are needed to
>> consume EJBs from a remote Jboss AS instance. Maybe there are some
>> classes somewhere that conflict with those of Tomcat, but if that would
>> be true it's still hard to understand why starting up fails in
>> production but never on a local workstation.
>>    
>
> Does the production box have more cores than the dev box? If there was a
> race condition you would be more likely to see it with more cores.
>
>  
>> Anyone has any idea?
>>    
>
> There was a sync issue in the webappclassloader fixed in 6.0.19.
> Although the symptoms were different, it might be worth trying 6.0.20.
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>  

[markt-2]

A. Rothman wrote:
> A couple months ago I had a slightly different setting but with the same
> strange result - NoClassDefFoundErrors which occurred on a particular
> jsp page in my app, but only if it was the first jsp accessed following
> a restart (and remained in the error state until the next restart). If
> it was not the first jsp accessed, then everything came up fine (with
> nothing in the configuration or app changing in between). I also figured
> it's some classloader bug in Tomcat, not entirely a race condition
> (since it did happen consistently), but something that manifests itself
> depending on the order of class loading.

If you can provide a cut-down copy of that JSP that reproduces the
problem on a clean Tomcat install, I'd be happy to take a look. Just
create a new bugzilla entry and provide the JSP as an attachment.

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[A. Rothman]

I've opened https://issues.apache.org/bugzilla/show_bug.cgi?id=48097 
with a sample jsp that recreates the error.

Mark Thomas wrote:

> A. Rothman wrote:
>  
>> A couple months ago I had a slightly different setting but with the same
>> strange result - NoClassDefFoundErrors which occurred on a particular
>> jsp page in my app, but only if it was the first jsp accessed following
>> a restart (and remained in the error state until the next restart). If
>> it was not the first jsp accessed, then everything came up fine (with
>> nothing in the configuration or app changing in between). I also figured
>> it's some classloader bug in Tomcat, not entirely a race condition
>> (since it did happen consistently), but something that manifests itself
>> depending on the order of class loading.
>>    
>
> If you can provide a cut-down copy of that JSP that reproduces the
> problem on a clean Tomcat install, I'd be happy to take a look. Just
> create a new bugzilla entry and provide the JSP as an attachment.
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>
>  

[A. Rothman]

To anyone who may stumble upon this thread looking for solutions - this
bug has been investigated and fixed by Konstantin Kolinko
(https://issues.apache.org/bugzilla/show_bug.cgi?id=48097). The fix
should be available in Tomcat 6.0.21 and later.
<mailto:knst.kolinko@...>

A. Rothman wrote:

> I've opened https://issues.apache.org/bugzilla/show_bug.cgi?id=48097 
> with a sample jsp that recreates the error.
>
> Mark Thomas wrote:
>
>> A. Rothman wrote:
>>  
>>> A couple months ago I had a slightly different setting but with the
>>> same
>>> strange result - NoClassDefFoundErrors which occurred on a particular
>>> jsp page in my app, but only if it was the first jsp accessed following
>>> a restart (and remained in the error state until the next restart). If
>>> it was not the first jsp accessed, then everything came up fine (with
>>> nothing in the configuration or app changing in between). I also
>>> figured
>>> it's some classloader bug in Tomcat, not entirely a race condition
>>> (since it did happen consistently), but something that manifests itself
>>> depending on the order of class loading.
>>>    
>>
>> If you can provide a cut-down copy of that JSP that reproduces the
>> problem on a clean Tomcat install, I'd be happy to take a look. Just
>> create a new bugzilla entry and provide the JSP as an attachment.
>>
>> Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@...
>> For additional commands, e-mail: users-help@...
>>
>>
>>  
>