Re: Disappearing tasks and OOMex.

by Damian Bradicich :: Rate this Message:

Oh, could you also include the full log file as well ?

Thanks

On Wed, Jul 22, 2009 at 8:53 AM, Damian Bradicich <dbradicich@...> wrote:

James, how much memory are you allocating to the jvm (if you changed from the default) ?

Also, 1.3.6 is released, which contains the memory fixes that i had mentioned earlier, although not directly related to the issue at hand, could very well be contributing to the problem.

Could you pass along your nexus.xml file to us as well, i haven't had any luck reproducing this problem yet, and am hoping something will catch my eye

Thanks

Damian

On Wed, Jul 22, 2009 at 7:17 AM, Nord, James <JNord@...> wrote:

Still getting the OOMEX and my scheduled task is *still* disappearing :-(

INFO   | jvm 1    | 2009/07/17 02:29:22 | 2009-07-17 02:29:22 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository central
INFO   | jvm 1    | 2009/07/17 02:32:35 | 2009-07-17 02:32:35 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository all
INFO   | jvm 1    | 2009/07/17 02:34:25 | 2009-07-17 02:34:25 INFO - o.s.n.f.FeedRecorder:default - Reindexing all registered repositories : Process started on Fri Jul 17 02:00:00 BST 2009, finished on Fri Jul 17 02:34:25 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2009/07/17 02:34:25 |       at java.util.Arrays.copyOf(Arrays.java:2882)
INFO   | jvm 1    | 2009/07/17 02:34:25 |       at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
INFO   | jvm 1    | 2009/07/17 02:34:25 |       at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)

/James

From: Nord, James [mailto:JNord@...]
Sent: 07 July 2009 14:38

To: nexus-user@...
Subject: RE: [nexus-user] Disappearing tasks and OOMex.

Upgraded to 1.3.4 but IE seems to have massive issues with this version :-(

Things are timing out, some icons/styles etc are missing - repository lists not shown

Errors like

"Nexus returned an error: ERROR -1: transaction aborted"

FF3.5 is fine, as is Opera 10, however IE7 & IE8 have the same issue.

I have deleted my IE cache, and restart IE several times and unfortunately this is reproducible.

The following will reproduce the issue in our environment.

clear IE cache.

navigate to a 1.3.4 nexus instance

browser repositories

navigate to a 1.3.1.1 nexus instance

browser repositories

navigate to 1.3.4 nexus instance

browse repository list

hit refresh (in the nexus UI not in IE)

the operation sits waiting and will eventually fail...

Nexus is currently re-indexing so finding the following may not be the correct excerpt log but others looked like the re-indexing.

2009-07-07 14:25:19 ERROR [qtp0-35        ] - o.r.C.Server                  - An exception occured writing the response entity
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
at org.restlet.util.ByteUtils.write(ByteUtils.java:974)
at org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
at com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 34 more
2009-07-07 14:25:19 WARN [qtp0-35        ] - org.mortbay.log               - Committed before 500 An exception occured writing the response entity
2009-07-07 14:25:19 WARN [qtp0-35        ] - o.r.C.Server                  - Error while handling an HTTP server call:

/James

From: Nord, James [mailto:JNord@...]
Sent: 07 July 2009 09:56
To: nexus-user@...
Subject: RE: [nexus-user] Disappearing tasks and OOMex.

I'll upgrade to 1.3.4 for the config issue.

As for the OOM, I have 14 repositories defined (all proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3 groups.

I also wouldn't say this instance was being hit _that_ hard.

The (virtual) machine it's running on has a single virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments to the wrapper are commented out - so it seems to be using the default heap size (64MB) - should this be enough - and would it be helpful if I enabled the heap dump on OOM?

/James

From: Damian Bradicich [mailto:dbradicich@...]
Sent: 06 July 2009 21:14
To: nexus-user@...
Subject: Re: [nexus-user] Disappearing tasks and OOMex.

Hi James, there were problems in the past where failed tasks would be removed from config. This has been fixed in a newer release (currently 1.3.4 is available).

The OOM exception is a bit more concerning, haven't seen this particular issue in the past, although if you have a large number of repositories, our NFC (not found cache) can get rather large, and consume a large amount of memory. This is being reworked for 1.4.0 to use a single cache (rather than a cache per repository) to help ease that unbounded memory usage. Also, in 1.3.5 (release due out shortly) the default caching will be configured to hold a smaller number of records, which has helped out public server (repository.sonatype.org) quite a bit.

Damian

On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...> wrote:

Hi all,

Using nexus 1.3.1.1,

    CentOS 5.2

    Sun java version "1.6.0_07"
        Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
        Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

I've come across an issue with scheduled tasks disappearing which has happened on a number of occasions now...

Most recently nightly re-index disappeared from the tasks view. There was nothing worthy in the logs (as I only just noticed the task was missing) but the system rss feed shows the following after which it appears the task never runs again and is probably lost..

(note even though we have an OOMEx nexus is still serving requests :-) )

Reindexing all registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.lang.StringCoding.safeTrim(StringCoding.java:64)
at java.lang.StringCoding.access$300(StringCoding.java:34)
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************
**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

« Return to Thread: Disappearing tasks and OOMex.