Re: Memory leak

The only thing I can think of is that it might be related to the transaction manager - for a long running transaction, the TM is probably retaining log entries to accommodate a rollback.  For such a long running transaction, that could be a lot of overhead.  What TM implementation are you using?

I myself just finished an application which required ridiculous amounts of data to be loaded on a daily basis in batch mode (gigabytes, millions of records across many tables).  I basically ensured that the insert operations happened in 1000 count 'mini' batches for 2 reasons:

1) to prevent a TM timeout (default was 5 minutes)

2) to prevent the TM overhead from getting too large (memory)

Here's some code I used to do that (Java + Spring TransactionTemplate).  Now I don't know if you are experiencing the same thing I did, but you might find it useful.

- note that the Iterator passed in represented a custom implementation that was reading records from a file stream - the file was over 10 Gigs, and the Iterator represented each 'chunked' record in the stream.  

- The 'flushCount' was a config variable from a .properties file - in our case, it was equal to 1000).

- the flush() method just called a DAO which internally called hibernateTemplate.flush() and then immediately hibernateTemplate.clear();

- you could try to extend the transaction timeout beyond the default (e.g. 5 minutes) by configuring the transaction manager, but

  odds are high you'd get into a TM housekeeping memory problem

protected int importBatch(final Iterator iterator, final int flushCount) {

        //Batch inserts need to be done as small chunks within their own transaction to avoid

        //transaction timeouts (and rollbacks) due to large data sets, so

        //specify PROPAGATION_REQUIRES_NEW.

        //

        //If we did NOT do this, and tried to import hundreds of thousands or millions of records,

        //odds are high that the transaction would time out

        //(e.g. after 5 minutes) and everything would be rolled back.  This is not desired.

        TransactionDefinition txnDef =

                new DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW);

        TransactionTemplate txnTemplate = new TransactionTemplate(getTransactionManager(), txnDef);

        Integer importCount = (Integer) txnTemplate.execute(new TransactionCallback() {

            public Object doInTransaction(TransactionStatus status) {

                int batchRecordCount = 0;

                while (iterator.hasNext() ) {

                    Record record = (Record) iterator.next();

                    if (record != null) { //can be null if any records are skipped

                        importRecord(record);

                        batchRecordCount++;

                        notifyRecordImported(record);

                        if (batchRecordCount % flushCount == 0) {

                            flush();

                        }

                    }

                }

                if (batchRecordCount > 0) {

                    flush();

                }

                if (log.isInfoEnabled()) {

                    log.info("Batch imported " + batchRecordCount + " records in a single transaction.");

                }

                return batchRecordCount;

            }

        });

        return importCount != null ? importCount : 0;

    }

Note that if any of those 'mini' transactions fail, you have to do a manual rollback of all of the records that went in.  We accounted for this by storing an id in the table where these fields were inserted, and if any error occurred, performed a bulk delete ('delete from blah where manual_tx_id = foo').

HTH,

Les

On Thu, Sep 4, 2008 at 10:53 AM, Burt Beckwith <burt@...> wrote:

I'm building a large database for load testing, so I'm creating millions of
domain instances. I'm trying to use the standard bulk insert approach in
Hibernate, i.e. flush() and clear() the session periodically, with the whole
operation running in a transaction.

The problem is the instances are still around - I run out of memory after a
few hours with a heap set at 1G. I've turned off 2nd-level caching and looked
at the Session in a debugger and clear() definitely works - it's empty
afterwards. There are no mapped collections, and I'm not keeping any explicit
references to the instances.

But running in a profiler I can see the instance count steadily increase and
never decrease. Running gc() has no effect.

Any thoughts on what might be keeping a reference to these instances?

Burt

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email