|
Disappearing tasks and OOMex.
Hi
all,
Using nexus
1.3.1.1,
CentOS 5.2
Sun java version "1.6.0_07"
Java(TM) SE Runtime Environment (build
1.6.0_07-b06)
Java HotSpot(TM)
Client VM (build 10.0-b23, mixed mode, sharing)
I've
come across an issue with scheduled tasks disappearing which has happened on a
number of occasions now...
Most recently
nightly re-index disappeared from the tasks view. There was nothing worthy
in the logs (as I only just noticed the task was missing) but the system rss
feed shows the following after which it appears the task never runs again and is
probably lost..
(note even though we
have an OOMEx nexus is still serving requests :-) )
Reindexing all
registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009,
finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is:
java.lang.OutOfMemoryError, Java heap space Strack trace:
java.lang.OutOfMemoryError: Java heap space
at
java.util.Arrays.copyOf(Arrays.java:2786)
at
java.lang.StringCoding.safeTrim(StringCoding.java:64)
at
java.lang.StringCoding.access$300(StringCoding.java:34)
at
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at
java.lang.StringCoding.encode(StringCoding.java:272)
at
java.lang.String.getBytes(String.java:947)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at
org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at
org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at
org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at
org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at
org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at
org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at
org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
|
Re: Disappearing tasks and OOMex.
Hi James, there were problems in the past where failed tasks would be removed from config. This has been fixed in a newer release (currently 1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this particular issue in the past, although if you have a large number of repositories, our NFC (not found cache) can get rather large, and consume a large amount of memory. This is being reworked for 1.4.0 to use a single cache (rather than a cache per repository) to help ease that unbounded memory usage. Also, in 1.3.5 (release due out shortly) the default caching will be configured to hold a smaller number of records, which has helped out public server ( repository.sonatype.org) quite a bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...> wrote:
Hi all,
Using nexus 1.3.1.1,
CentOS 5.2
Sun java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)
I've come across an issue with scheduled tasks disappearing which has happened on a number of occasions now...
Most recently nightly re-index disappeared from the tasks view. There was nothing worthy in the logs (as I only just noticed the task was missing) but the system rss feed shows the following after which it appears the task never runs again and is probably lost..
(note even though we have an OOMEx nexus is still serving requests :-) )
Reindexing all registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.lang.StringCoding.safeTrim(StringCoding.java:64)
at java.lang.StringCoding.access$300(StringCoding.java:34)
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************
|
|
RE: Disappearing tasks and OOMex.
I'll upgrade to 1.3.4 for the config
issue.
As for the OOM, I have 14 repositories defined (all
proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3
groups.
I also wouldn't say this instance was being hit _that_
hard.
The (virtual) machine it's running on has a single
virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments to
the wrapper are commented out - so it seems to be using the default heap size
(64MB) - should this be enough - and would it be helpful if I enabled the
heap dump on OOM?
/James
Hi James, there were problems in the past where failed tasks would be
removed from config. This has been fixed in a newer release (currently
1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this particular
issue in the past, although if you have a large number of repositories, our
NFC (not found cache) can get rather large, and consume a large amount of
memory. This is being reworked for 1.4.0 to use a single cache (rather
than a cache per repository) to help ease that unbounded memory usage.
Also, in 1.3.5 (release due out shortly) the default caching will be
configured to hold a smaller number of records, which has helped out public
server ( repository.sonatype.org)
quite a bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...>
wrote:
Hi all,
Using nexus 1.3.1.1,
CentOS
5.2
Sun java version
"1.6.0_07"
Java(TM) SE Runtime
Environment (build
1.6.0_07-b06)
Java HotSpot(TM)
Client VM (build 10.0-b23, mixed mode, sharing)
I've come across an
issue with scheduled tasks disappearing which has happened on a number of
occasions now...
Most recently nightly re-index
disappeared from the tasks view. There was nothing worthy in the logs
(as I only just noticed the task was missing) but the system rss feed shows
the following after which it appears the task never runs again and is
probably lost..
(note even though we have an OOMEx nexus
is still serving requests :-) )
Reindexing all registered repositories :
Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18
02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError,
Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap
space
at java.util.Arrays.copyOf(Arrays.java:2786)
at
java.lang.StringCoding.safeTrim(StringCoding.java:64)
at
java.lang.StringCoding.access$300(StringCoding.java:34)
at
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at
java.lang.StringCoding.encode(StringCoding.java:272)
at
java.lang.String.getBytes(String.java:947)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at
org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at
org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at
org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at
org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at
org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at
org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at
org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
**************************************************************************************
This
message is confidential and intended only for the addressee. If you have
received this message in error, please immediately notify the postmaster@... and
delete it from your system as well as any copies. The content of e-mails as
well as traffic data may be monitored by NDS for employment and security
purposes. To protect the environment please do not print this e-mail unless
necessary.
NDS Limited. Registered Office: One London Road, Staines,
Middlesex, TW18 4EX, United Kingdom. A company registered in England and
Wales. Registered no. 3080780. VAT no. GB 603 8808
40-00
**************************************************************************************
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
|
RE: Disappearing tasks and OOMex.
Upgraded to 1.3.4 but IE seems to have massive issues with
this version :-(
Things are timing out, some icons/styles etc are missing -
repository lists not shown
Errors like
"Nexus returned an error: ERROR -1: transaction
aborted"
FF3.5 is fine, as is Opera 10, however IE7 & IE8 have
the same issue.
I have deleted my
IE cache, and restart IE several times and unfortunately this is
reproducible.
The following will reproduce the issue in our
environment.
clear IE cache.
navigate to a 1.3.4 nexus instance
browser repositories
navigate to a 1.3.1.1 nexus instance
browser repositories
navigate to 1.3.4 nexus instance
browse repository list
hit refresh (in the nexus UI not in IE)
the operation sits waiting and will eventually
fail...
Nexus is currently re-indexing so finding the following may
not be the correct excerpt log but others looked like the
re-indexing.
2009-07-07 14:25:19 ERROR
[qtp0-35 ] -
o.r.C.Server
- An exception occured writing the response
entity
org.mortbay.jetty.EofException
at
org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
at
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at
java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
at
org.restlet.util.ByteUtils.write(ByteUtils.java:974)
at
org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
at
com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
at
com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at
com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at
com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
at
com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
at
com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at
com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at
org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused
by: java.io.IOException: Connection reset by peer
at
sun.nio.ch.FileDispatcher.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at
sun.nio.ch.IOUtil.write(IOUtil.java:60)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at
org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 34
more
2009-07-07 14:25:19 WARN
[qtp0-35 ] -
org.mortbay.log
- Committed before 500 An exception occured writing the response
entity
2009-07-07 14:25:19 WARN
[qtp0-35 ] -
o.r.C.Server
- Error while handling an HTTP server call:
/James
I'll upgrade to 1.3.4 for the config
issue.
As for the OOM, I have 14 repositories defined (all
proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3
groups.
I also wouldn't say this instance was being hit
_that_ hard.
The (virtual) machine it's running on has a single
virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments
to the wrapper are commented out - so it seems to be using the default heap
size (64MB) - should this be enough - and would it be helpful if I
enabled the heap dump on OOM?
/James
Hi James, there were problems in the past where failed tasks would be
removed from config. This has been fixed in a newer release (currently
1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this
particular issue in the past, although if you have a large number of
repositories, our NFC (not found cache) can get rather large, and consume a
large amount of memory. This is being reworked for 1.4.0 to use a
single cache (rather than a cache per repository) to help ease that
unbounded memory usage. Also, in 1.3.5 (release due out shortly) the
default caching will be configured to hold a smaller number of records,
which has helped out public server ( repository.sonatype.org) quite a
bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...>
wrote:
Hi all,
Using nexus
1.3.1.1,
CentOS
5.2
Sun java version
"1.6.0_07"
Java(TM) SE
Runtime Environment (build
1.6.0_07-b06)
Java
HotSpot(TM) Client VM (build 10.0-b23, mixed mode,
sharing)
I've come across an
issue with scheduled tasks disappearing which has happened on a number of
occasions now...
Most recently nightly re-index
disappeared from the tasks view. There was nothing worthy in the
logs (as I only just noticed the task was missing) but the system rss feed
shows the following after which it appears the task never runs again and
is probably lost..
(note even though we have an OOMEx
nexus is still serving requests :-) )
Reindexing all registered repositories
: Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18
02:20:53 BST 2009 with error. Error message is:
java.lang.OutOfMemoryError, Java heap space Strack trace:
java.lang.OutOfMemoryError: Java heap space
at
java.util.Arrays.copyOf(Arrays.java:2786)
at
java.lang.StringCoding.safeTrim(StringCoding.java:64)
at
java.lang.StringCoding.access$300(StringCoding.java:34)
at
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at
java.lang.StringCoding.encode(StringCoding.java:272)
at
java.lang.String.getBytes(String.java:947)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at
org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at
org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at
org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at
org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at
org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at
org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at
org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
|
RE: Disappearing tasks and OOMex.
Still getting the OOMEX and my scheduled task is *still*
disappearing :-(
INFO | jvm 1 | 2009/07/17
02:29:22 | 2009-07-17 02:29:22 INFO - o.s.n.i.IndexerMana~:default -
Publishing best index for repository central
INFO | jvm
1 | 2009/07/17 02:32:35 | 2009-07-17 02:32:35 INFO -
o.s.n.i.IndexerMana~:default - Publishing best index for repository
all
INFO | jvm 1 | 2009/07/17 02:34:25 |
2009-07-17 02:34:25 INFO - o.s.n.f.FeedRecorder:default - Reindexing
all registered repositories : Process started on Fri Jul 17 02:00:00 BST 2009,
finished on Fri Jul 17 02:34:25 BST 2009 with error. Error message is:
java.lang.OutOfMemoryError, Java heap space Strack trace:
java.lang.OutOfMemoryError: Java heap space
INFO | jvm
1 | 2009/07/17 02:34:25 |
at java.util.Arrays.copyOf(Arrays.java:2882)
INFO | jvm
1 | 2009/07/17 02:34:25 |
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
INFO
| jvm 1 | 2009/07/17 02:34:25
| at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
/James
Upgraded to 1.3.4 but IE seems to have massive issues
with this version :-(
Things are timing out, some icons/styles etc are missing
- repository lists not shown
Errors like
"Nexus returned an error: ERROR -1: transaction
aborted"
FF3.5 is fine, as is Opera 10, however IE7 & IE8 have
the same issue.
I have deleted
my IE cache, and restart IE several times and unfortunately this is
reproducible.
The following will reproduce the issue in our
environment.
clear IE cache.
navigate to a 1.3.4 nexus instance
browser repositories
navigate to a 1.3.1.1 nexus instance
browser repositories
navigate to 1.3.4 nexus instance
browse repository list
hit refresh (in the nexus UI not in
IE)
the operation sits waiting and will eventually
fail...
Nexus is currently re-indexing so finding the following
may not be the correct excerpt log but others looked like the
re-indexing.
2009-07-07 14:25:19 ERROR
[qtp0-35 ] -
o.r.C.Server
- An exception occured writing the response
entity
org.mortbay.jetty.EofException
at
org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
at
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at
java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
at
org.restlet.util.ByteUtils.write(ByteUtils.java:974)
at
org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
at
com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
at
com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at
com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at
com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
at
com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
at
com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at
com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at
org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused
by: java.io.IOException: Connection reset by peer
at
sun.nio.ch.FileDispatcher.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at
sun.nio.ch.IOUtil.write(IOUtil.java:60)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at
org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 34
more
2009-07-07 14:25:19 WARN
[qtp0-35 ] -
org.mortbay.log
- Committed before 500 An exception occured writing the response
entity
2009-07-07 14:25:19 WARN
[qtp0-35 ] -
o.r.C.Server
- Error while handling an HTTP server call:
/James
I'll upgrade to 1.3.4 for the config
issue.
As for the OOM, I have 14 repositories defined (all
proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3
groups.
I also wouldn't say this instance was being hit
_that_ hard.
The (virtual) machine it's running on has a single
virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments
to the wrapper are commented out - so it seems to be using the default heap
size (64MB) - should this be enough - and would it be helpful if I
enabled the heap dump on OOM?
/James
Hi James, there were problems in the past where failed tasks would be
removed from config. This has been fixed in a newer release
(currently 1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this
particular issue in the past, although if you have a large number of
repositories, our NFC (not found cache) can get rather large, and consume
a large amount of memory. This is being reworked for 1.4.0 to use a
single cache (rather than a cache per repository) to help ease that
unbounded memory usage. Also, in 1.3.5 (release due out shortly) the
default caching will be configured to hold a smaller number of records,
which has helped out public server ( repository.sonatype.org) quite a
bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...>
wrote:
Hi all,
Using nexus
1.3.1.1,
CentOS
5.2
Sun java version
"1.6.0_07"
Java(TM) SE
Runtime Environment (build
1.6.0_07-b06)
Java
HotSpot(TM) Client VM (build 10.0-b23, mixed mode,
sharing)
I've come across
an issue with scheduled tasks disappearing which has happened on a
number of occasions now...
Most recently nightly re-index
disappeared from the tasks view. There was nothing worthy in the
logs (as I only just noticed the task was missing) but the system rss
feed shows the following after which it appears the task never runs
again and is probably lost..
(note even though we have an OOMEx
nexus is still serving requests :-) )
Reindexing all registered
repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished
on Thu Jun 18 02:20:53 BST 2009 with error. Error message is:
java.lang.OutOfMemoryError, Java heap space Strack trace:
java.lang.OutOfMemoryError: Java heap space
at
java.util.Arrays.copyOf(Arrays.java:2786)
at
java.lang.StringCoding.safeTrim(StringCoding.java:64)
at
java.lang.StringCoding.access$300(StringCoding.java:34)
at
java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at
java.lang.StringCoding.encode(StringCoding.java:272)
at
java.lang.String.getBytes(String.java:947)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at
org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at
org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at
org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at
org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at
org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at
org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at
org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at
org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at
org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at
java.lang.Thread.run(Thread.java:619)
**************************************************************************************
This
message is confidential and intended only for the addressee. If you have
received this message in error, please immediately notify the
postmaster@... and delete it from your system as well as any copies. The
content of e-mails as well as traffic data may be monitored by NDS for
employment and security purposes. To protect the environment please do not
print this e-mail unless necessary.
NDS Limited. Registered Office: One
London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company
registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808
40-00
**************************************************************************************
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
|
Re: Disappearing tasks and OOMex.
James, how much memory are you allocating to the jvm (if you changed from the default) ?
Also, 1.3.6 is released, which contains the memory fixes that i had mentioned earlier, although not directly related to the issue at hand, could very well be contributing to the problem.
Could you pass along your nexus.xml file to us as well, i haven't had any luck reproducing this problem yet, and am hoping something will catch my eye
Thanks
Damian
On Wed, Jul 22, 2009 at 7:17 AM, Nord, James <JNord@...> wrote:
Still getting the OOMEX and my scheduled task is *still* disappearing :-(
INFO | jvm 1 | 2009/07/17 02:29:22 | 2009-07-17 02:29:22 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository central
INFO | jvm 1 | 2009/07/17 02:32:35 | 2009-07-17 02:32:35 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository all
INFO | jvm 1 | 2009/07/17 02:34:25 | 2009-07-17 02:34:25 INFO - o.s.n.f.FeedRecorder:default - Reindexing all registered repositories : Process started on Fri Jul 17 02:00:00 BST 2009, finished on Fri Jul 17 02:34:25 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.util.Arrays.copyOf(Arrays.java:2882)
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
/James
Sent: 07 July 2009 14:38
To: nexus-user@...Subject: RE: [nexus-user] Disappearing tasks and OOMex.
Upgraded to 1.3.4 but IE seems to have massive issues with this version :-(
Things are timing out, some icons/styles etc are missing - repository lists not shown
Errors like
"Nexus returned an error: ERROR -1: transaction aborted"
FF3.5 is fine, as is Opera 10, however IE7 & IE8 have the same issue.
I have deleted my IE cache, and restart IE several times and unfortunately this is reproducible.
The following will reproduce the issue in our environment.
clear IE cache.
navigate to a 1.3.4 nexus instance
browser repositories
navigate to a 1.3.1.1 nexus instance
browser repositories
navigate to 1.3.4 nexus instance
browse repository list
hit refresh (in the nexus UI not in IE)
the operation sits waiting and will eventually fail...
Nexus is currently re-indexing so finding the following may not be the correct excerpt log but others looked like the re-indexing.
2009-07-07 14:25:19 ERROR [qtp0-35 ] - o.r.C.Server - An exception occured writing the response entity
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
at org.restlet.util.ByteUtils.write(ByteUtils.java:974)
at org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
at com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 34 more
2009-07-07 14:25:19 WARN [qtp0-35 ] - org.mortbay.log - Committed before 500 An exception occured writing the response entity
2009-07-07 14:25:19 WARN [qtp0-35 ] - o.r.C.Server - Error while handling an HTTP server call:
/James
From: Nord, James [mailto:JNord@...]
Sent: 07 July 2009 09:56
To: nexus-user@...
Subject: RE: [nexus-user] Disappearing tasks and OOMex.
I'll upgrade to 1.3.4 for the config issue.
As for the OOM, I have 14 repositories defined (all proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3 groups.
I also wouldn't say this instance was being hit _that_ hard.
The (virtual) machine it's running on has a single virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments to the wrapper are commented out - so it seems to be using the default heap size (64MB) - should this be enough - and would it be helpful if I enabled the heap dump on OOM?
/James
From: Damian Bradicich [mailto:dbradicich@...]
Sent: 06 July 2009 21:14
To: nexus-user@...
Subject: Re: [nexus-user] Disappearing tasks and OOMex.
Hi James, there were problems in the past where failed tasks would be removed from config. This has been fixed in a newer release (currently 1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this particular issue in the past, although if you have a large number of repositories, our NFC (not found cache) can get rather large, and consume a large amount of memory. This is being reworked for 1.4.0 to use a single cache (rather than a cache per repository) to help ease that unbounded memory usage. Also, in 1.3.5 (release due out shortly) the default caching will be configured to hold a smaller number of records, which has helped out public server ( repository.sonatype.org) quite a bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...> wrote:
Hi all,
Using nexus 1.3.1.1,
CentOS 5.2
Sun java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)
I've come across an issue with scheduled tasks disappearing which has happened on a number of occasions now...
Most recently nightly re-index disappeared from the tasks view. There was nothing worthy in the logs (as I only just noticed the task was missing) but the system rss feed shows the following after which it appears the task never runs again and is probably lost..
(note even though we have an OOMEx nexus is still serving requests :-) )
Reindexing all registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.lang.StringCoding.safeTrim(StringCoding.java:64)
at java.lang.StringCoding.access$300(StringCoding.java:34)
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
|
Re: Disappearing tasks and OOMex.
Oh, could you also include the full log file as well ?
Thanks
On Wed, Jul 22, 2009 at 8:53 AM, Damian Bradicich <dbradicich@...> wrote:
James, how much memory are you allocating to the jvm (if you changed from the default) ?
Also, 1.3.6 is released, which contains the memory fixes that i had mentioned earlier, although not directly related to the issue at hand, could very well be contributing to the problem.
Could you pass along your nexus.xml file to us as well, i haven't had any luck reproducing this problem yet, and am hoping something will catch my eye
Thanks
Damian
On Wed, Jul 22, 2009 at 7:17 AM, Nord, James <JNord@...> wrote:
Still getting the OOMEX and my scheduled task is *still* disappearing :-(
INFO | jvm 1 | 2009/07/17 02:29:22 | 2009-07-17 02:29:22 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository central
INFO | jvm 1 | 2009/07/17 02:32:35 | 2009-07-17 02:32:35 INFO - o.s.n.i.IndexerMana~:default - Publishing best index for repository all
INFO | jvm 1 | 2009/07/17 02:34:25 | 2009-07-17 02:34:25 INFO - o.s.n.f.FeedRecorder:default - Reindexing all registered repositories : Process started on Fri Jul 17 02:00:00 BST 2009, finished on Fri Jul 17 02:34:25 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.util.Arrays.copyOf(Arrays.java:2882)
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
INFO | jvm 1 | 2009/07/17 02:34:25 | at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
/James
Sent: 07 July 2009 14:38
To: nexus-user@...Subject: RE: [nexus-user] Disappearing tasks and OOMex.
Upgraded to 1.3.4 but IE seems to have massive issues with this version :-(
Things are timing out, some icons/styles etc are missing - repository lists not shown
Errors like
"Nexus returned an error: ERROR -1: transaction aborted"
FF3.5 is fine, as is Opera 10, however IE7 & IE8 have the same issue.
I have deleted my IE cache, and restart IE several times and unfortunately this is reproducible.
The following will reproduce the issue in our environment.
clear IE cache.
navigate to a 1.3.4 nexus instance
browser repositories
navigate to a 1.3.1.1 nexus instance
browser repositories
navigate to 1.3.4 nexus instance
browse repository list
hit refresh (in the nexus UI not in IE)
the operation sits waiting and will eventually fail...
Nexus is currently re-indexing so finding the following may not be the correct excerpt log but others looked like the re-indexing.
2009-07-07 14:25:19 ERROR [qtp0-35 ] - o.r.C.Server - An exception occured writing the response entity
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
at org.restlet.util.ByteUtils.write(ByteUtils.java:974)
at org.restlet.resource.InputRepresentation.write(InputRepresentation.java:121)
at com.noelios.restlet.application.EncodeRepresentation.write(EncodeRepresentation.java:273)
at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:407)
at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:388)
at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:884)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
... 34 more
2009-07-07 14:25:19 WARN [qtp0-35 ] - org.mortbay.log - Committed before 500 An exception occured writing the response entity
2009-07-07 14:25:19 WARN [qtp0-35 ] - o.r.C.Server - Error while handling an HTTP server call:
/James
From: Nord, James [mailto:JNord@...]
Sent: 07 July 2009 09:56
To: nexus-user@...
Subject: RE: [nexus-user] Disappearing tasks and OOMex.
I'll upgrade to 1.3.4 for the config issue.
As for the OOM, I have 14 repositories defined (all proxy Maven2 types, a mixture of release and snapshot - roughly 50/50) and 3 groups.
I also wouldn't say this instance was being hit _that_ hard.
The (virtual) machine it's running on has a single virtual CPU and 1GB of RAM (so client VM), and it seems any memory arguments to the wrapper are commented out - so it seems to be using the default heap size (64MB) - should this be enough - and would it be helpful if I enabled the heap dump on OOM?
/James
From: Damian Bradicich [mailto:dbradicich@...]
Sent: 06 July 2009 21:14
To: nexus-user@...
Subject: Re: [nexus-user] Disappearing tasks and OOMex.
Hi James, there were problems in the past where failed tasks would be removed from config. This has been fixed in a newer release (currently 1.3.4 is available).
The OOM exception is a bit more concerning, haven't seen this particular issue in the past, although if you have a large number of repositories, our NFC (not found cache) can get rather large, and consume a large amount of memory. This is being reworked for 1.4.0 to use a single cache (rather than a cache per repository) to help ease that unbounded memory usage. Also, in 1.3.5 (release due out shortly) the default caching will be configured to hold a smaller number of records, which has helped out public server ( repository.sonatype.org) quite a bit.
Damian
On Mon, Jul 6, 2009 at 10:11 AM, Nord, James <JNord@...> wrote:
Hi all,
Using nexus 1.3.1.1,
CentOS 5.2
Sun java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)
I've come across an issue with scheduled tasks disappearing which has happened on a number of occasions now...
Most recently nightly re-index disappeared from the tasks view. There was nothing worthy in the logs (as I only just noticed the task was missing) but the system rss feed shows the following after which it appears the task never runs again and is probably lost..
(note even though we have an OOMEx nexus is still serving requests :-) )
Reindexing all registered repositories : Process started on Thu Jun 18 02:00:01 BST 2009, finished on Thu Jun 18 02:20:53 BST 2009 with error. Error message is: java.lang.OutOfMemoryError, Java heap space Strack trace: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.lang.StringCoding.safeTrim(StringCoding.java:64)
at java.lang.StringCoding.access$300(StringCoding.java:34)
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:251)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:111)
at org.apache.lucene.index.DocumentsWriter$ThreadState$FieldData.processField(DocumentsWriter.java:1418)
at org.apache.lucene.index.DocumentsWriter$ThreadState.processDocument(DocumentsWriter.java:1121)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:2442)
at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:2424)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1442)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.copyLegacyDocuments(DefaultIndexPacker.java:254)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndexArchive(DefaultIndexPacker.java:229)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.writeIndexArchive(DefaultIndexPacker.java:201)
at org.sonatype.nexus.index.packer.DefaultIndexPacker.packIndex(DefaultIndexPacker.java:99)
at org.sonatype.nexus.index.DefaultIndexerManager.publishRepositoryIndex(DefaultIndexerManager.java:888)
at org.sonatype.nexus.index.DefaultIndexerManager.publishAllIndex(DefaultIndexerManager.java:809)
at org.sonatype.nexus.index.DefaultIndexerManager.reindexAllRepositories(DefaultIndexerManager.java:433)
at org.sonatype.nexus.DefaultNexus.reindexAllRepositories(DefaultNexus.java:1364)
at org.sonatype.nexus.tasks.ReindexTask.doRun(ReindexTask.java:44)
at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:139)
at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:342)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
************************************************************************************** This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmaster@... and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00 **************************************************************************************
|
