« Return to Thread: Disappearing tasks and OOMex.

RE: Disappearing tasks and OOMex.

by Nord, James-2 :: Rate this Message:

Reply to Author | View in Thread

Upgraded to 1.3.4 but IE seems to have massive issues with this version :-(
 
Things are timing out, some icons/styles etc are missing - repository lists not shown
Errors like
"Nexus returned an error: ERROR -1: transaction aborted"
 
FF3.5 is fine, as is Opera 10, however IE7 & IE8 have the same issue.
 
I have deleted my IE cache, and restart IE several times and unfortunately this is reproducible.
 
The following will reproduce the issue in our environment.
 
clear IE cache.
navigate to a 1.3.4 nexus instance
browser repositories
navigate to a 1.3.1.1 nexus instance
browser repositories
navigate to 1.3.4 nexus instance
browse repository list
hit refresh (in the nexus UI not in IE)
the operation sits waiting and will eventually fail...
 
Nexus is currently re-indexing so finding the following may not be the correct excerpt log but others looked like the re-indexing.
 
2009-07-07 14:25:19 ERROR [qtp0-35        ] - o.r.C.Server                  - An exception occured writing the response entity
org.mortbay.jetty.EofException
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
 at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
 at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
 at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
 at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
 at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
 at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
 at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
 at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
 at org.restlet.util.ByteUtils.write(ByteUtils.java:974)
 at org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
 at com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
 at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
 at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
 at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
 at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
 at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
 at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
 at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:324)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
 at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Connection reset by peer
 at sun.nio.ch.FileDispatcher.write0(Native Method)
 at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
 at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
 at sun.nio.ch.IOUtil.write(IOUtil.java:60)
 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
 at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
 at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
 at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
 ... 34 more
2009-07-07 14:25:19 WARN  [qtp0-35        ] - org.mortbay.log               - Committed before 500 An exception occured writing the response entity
2009-07-07 14:25:19 WARN  [qtp0-35        ] - o.r.C.Server                  - Error while handling an HTTP server call:
/James
 


From: Nord, James [mailto:JNord@...]
Sent: 07 July 2009 09:56
To: nexus-user@...
Subject: RE: [nexus-user] Disappearing tasks and OOMex.

I'll upgrade to 1.3.4 for the config issue.
 
As for the OOM, I have 14 repositories defined (all proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3 groups.
I also wouldn't say this instance was being hit _that_ hard.
 
The (virtual) machine it's running on has a single virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments to the wrapper are commented out - so it seems to be using the default heap size (64MB)  - should this be enough - and would it be helpful if I enabled the heap dump on OOM?
 
/James
 
 


From: Damian Bradicich [mailto:dbradicich@...]
Sent: 06 July 2009 21:14
To: nexus-user@...
Subject: Re: [nexus-user] Disappearing tasks and OOMex.

Hi James, there were problems in the past where failed tasks would be removed from config.  This has been fixed in a newer release (currently 1.3.4 is available). 
 
The OOM exception is a bit more concerning, haven't seen this particular issue in the past, although if you have a large number of repositories, our NFC (not found cache) can get rather large, and consume a large amount of memory.  This is being reworked for 1.4.0 to use a single cache (rather than a cache per repository) to help ease that unbounded memory usage.  Also, in 1.3.5 (release due out shortly) the default caching will be configured to hold a smaller number of records, which has helped out public server (repository.sonatype.org) quite a bit.
 

 Damian

On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...> wrote:
Hi all,
 
Using nexus 1.3.1.1,
    CentOS 5.2
    Sun java version "1.6.0_07"
        Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
        Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)
I've come across an issue with scheduled tasks disappearing which has happened on a number of occasions now...
 
Most recently nightly re-index disappeared from the tasks view.  There was nothing worthy in the logs (as I only just noticed the task was missing) but the system rss feed shows the following after which it appears the task never runs again and is probably lost..
 
(note even though we have an OOMEx nexus is still serving requests :-) )
 
 
Reindexing all registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:2786)
 at java.lang.StringCoding.safeTrim(StringCoding.java:64)
 at java.lang.StringCoding.access$300(StringCoding.java:34)
 at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
 at java.lang.StringCoding.encode(StringCoding.java:272)
 at java.lang.String.getBytes(String.java:947)
 at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
 at org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
 at org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
 at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
 at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
 at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
 at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
 at org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
 at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
 at org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
 at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
 at org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
 at org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
 at org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
 at org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
 at org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
 at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
 at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:619)


**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

 « Return to Thread: Disappearing tasks and OOMex.