eXist forcing full GC before heap full?

[Alessandro Vernet]

I am importing a very large XML document, and have set the VM to -Xms and -Xmx to 1 GB. Running the VM with -verbosegc, I notice that VM is doing repetitive garbage collections when the heap reaches 800 MB, 200 MB short of the 1 GB I set. Surprised by this behavior, I did a few thread dumps while the import was running, and it seems that those full GCs are forced by eXist:

"http-8080-Processor23" daemon prio=5 tid=0x00235910 nid=0x845e00 runnable [0xb20b2000..0xb20b2d90]
        at java.lang.Runtime.gc(Native Method)
        at java.lang.System.gc(System.java:891)
        at org.exist.storage.NativeBroker.checkAvailableMemory(NativeBroker.java:2932)
        at org.exist.storage.NativeBroker.storeNode(NativeBroker.java:2381)
        at org.exist.storage.DBBroker.storeNode(DBBroker.java:503)

Is this an indication that I need to increase the cacheSize setting (which I still have at 48 MB)? Can increasing it too much lead to out of memory errors? Since eXist is forcing a full GC short of 200 MB before reaching the 1 GB max heap size, should I consider increasing cacheSize by close to 200 MB (say 150 MB, to be on the safe side)?

Ideally, I'd like eXist to automagically use as much memory as possible for the cache, up to the max heap size, and free some of the memory used by the cache on full garbage collection (sort of what weak references are promised to offer). But maybe this isn't realistic?

Alex

[James Fuller-4]

are u using java client ? can u clarify how you are importing ?

Jim Fuller


On Tue, Jun 30, 2009 at 7:38 PM, Alessandro Vernet<avernet@...> wrote:

>
> I am importing a very large XML document, and have set the VM to -Xms and
> -Xmx to 1 GB. Running the VM with -verbosegc, I notice that VM is doing
> repetitive garbage collections when the heap reaches 800 MB, 200 MB short of
> the 1 GB I set. Surprised by this behavior, I did a few thread dumps while
> the import was running, and it seems that those full GCs are forced by
> eXist:
>
> "http-8080-Processor23" daemon prio=5 tid=0x00235910 nid=0x845e00 runnable
> [0xb20b2000..0xb20b2d90]
>        at java.lang.Runtime.gc(Native Method)
>        at java.lang.System.gc(System.java:891)
>        at
> org.exist.storage.NativeBroker.checkAvailableMemory(NativeBroker.java:2932)
>        at org.exist.storage.NativeBroker.storeNode(NativeBroker.java:2381)
>        at org.exist.storage.DBBroker.storeNode(DBBroker.java:503)
>
> Is this an indication that I need to increase the cacheSize setting (which I
> still have at 48 MB)? Can increasing it too much lead to out of memory
> errors? Since eXist is forcing a full GC short of 200 MB before reaching the
> 1 GB max heap size, should I consider increasing cacheSize by close to 200
> MB (say 150 MB, to be on the safe side)?
>
> Ideally, I'd like eXist to automagically use as much memory as possible for
> the cache, up to the max heap size, and free some of the memory used by the
> cache on full garbage collection (sort of what weak references are promised
> to offer). But maybe this isn't realistic?
>
> Alex
>
> -----
> Orbeon Forms - Web 2.0 Forms, open-source, for the Enterprise
> Orbeon's Blog: http://www.orbeon.com/blog/
> Personal Blog: http://avernet.blogspot.com/
> Twitter - http://twitter.com/avernet
> --
> View this message in context: http://www.nabble.com/eXist-forcing-full-GC-before-heap-full--tp24276360p24276360.html
> Sent from the exist-open mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Exist-open mailing list
> Exist-open@...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open

[Wolfgang Meier-2]

> I am importing a very large XML document, and have set the VM to -Xms and
> -Xmx to 1 GB. Running the VM with -verbosegc, I notice that VM is doing
> repetitive garbage collections when the heap reaches 800 MB, 200 MB short of
> the 1 GB I set.

I thought I removed the call to System.gc, but obviously I did not. Or
did I reintroduce it? I forgot, sorry. Normally you would expect Java to
trigger a major gc before it runs out of memory, so the call should not
be required. But you never know.

Well, I would suggest to comment out the call to System.gc and see if
the system runs stable or not. If yes, we should consider removing it in
trunk as well.

The checkAvailableMemory is called because eXist will use the entire
free memory to buffer index entries before it flushes them to the actual
index. This is not affected by the cacheSize setting in conf.xml (or at
least only indirectly). The cacheSize setting controls how much
*permanent* cache space will be used, while the indexing process may use
additional memory if available.

> Is this an indication that I need to increase the cacheSize setting (which I
> still have at 48 MB)?

No, though 48M cacheSize is really small if you need to store larger
documents. I would recommend to carefully increase cacheSize to 128M or
so (but always less than 1/3 of the main memory).

Wolfgang

------------------------------------------------------------------------------
_______________________________________________
Exist-open mailing list
Exist-open@...
https://lists.sourceforge.net/lists/listinfo/exist-open