« Return to Thread: Memory leak
Re: Memory leak
by Burt Beckwith
::
Rate this Message:
I spent quite a while on this today and think the issue has to do with Grails
and has nothing to do with transactions. Btw - it's not a memory leak in the
classic sense, but a GC issue - domain instances were somehow referenced and
couldn't be GC'd. The profile instance count just kept growing and growing.
I kept stripping down the code to try to isolate what was going on, and it
even happened when I just created a single, 1-field dummy domain class and
didn't call a single method on it. The code was something like
for (int i = 0; i < 100000; i++) {
new Thing(name: "thing_${i}")
Thread.sleep(50); // to allow time to watch things in the profiler
}
While the loop was running I'd run gc() periodically, and it had no effect on
the domain class instance count, but clearly it was working since other class
counts dropped and the overall heap size dropped. Once the method exited,
finally the gc() call dropped the instance count down to near zero.
Burt
On Thursday 04 September 2008 5:34:39 pm Les Hazlewood wrote:
> Oops, I didn't give you all the code - the method I posted earlier
> performed one and only one 'mini' batch insert. It was called from another
> method that coordinated calling that method as necessary:
> public void importProductBatched(Product product) {
> if (log.isInfoEnabled()) {
> log.info("Performing a batching import for product of type [" +
> product.getClass().getName() + "]");
> }
> preBatch();
> Iterator i = product.iterator();
> while (i.hasNext()) {
> int batchImported = importBatch(i, getBatchCount(),
> getFlushCount());
> product.setCommittedBytesRead(product.getBytesRead());
> product.setCommittedRecordsRead(((RecordIterator)
> i).getIndex());
> log.info("Batched imported Count" + batchImported + " bytes
> read =" + product.getBytesRead() + "|" + ((RecordIterator) i).getIndex());
> }
> if (log.isInfoEnabled()) {
> log.info("Successfully imported batch for product of type [" +
> product.getClass().getName() + "]");
> }
> postBatch();
> }
>
> And then the importBatch method:
>
> protected int importBatch(final Iterator iterator, final int maxRecords,
> final int flushCount) {
> //Batch inserts need to be done as small chunks within their own
> transaction to avoid
> //transaction timeouts (and rollbacks) due to large data sets, so
> //specify PROPAGATION_REQUIRES_NEW.
> //
> //If we did NOT do this, and imported in one transaction, and the
> Product contained
> //thousands of records (as does happen during batch mode), odds are
> high that the transaction would time out
> //(e.g. after 5 minutes) and everything would be rolled back. This
> is not desired.
> TransactionDefinition txnDef =
> new
> DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW
>);
>
> TransactionTemplate txnTemplate = new
> TransactionTemplate(getTransactionManager(), txnDef);
> Integer importCount = (Integer) txnTemplate.execute(new
> TransactionCallback() {
> public Object doInTransaction(TransactionStatus status) {
> int batchRecordCount = 0;
>
> while (iterator.hasNext() && batchRecordCount < maxRecords)
> {
> Record record = (Record) iterator.next();
> if (record != null) { //can be null if any records are
> skipped
> importRecord(record);
> batchRecordCount++;
> notifyRecordImported(record);
> if (batchRecordCount % flushCount == 0) {
> flush();
> }
> }
> }
> if (batchRecordCount > 0) {
> flush();
> }
>
> if (log.isInfoEnabled()) {
> log.info("Batch imported " + batchRecordCount + "
> records in a single transaction.");
> }
> return batchRecordCount;
> }
> });
> return importCount != null ? importCount : 0;
> }
>
> Regards,
>
> Les
>
> On Thu, Sep 4, 2008 at 5:30 PM, Les Hazlewood <les@...> wrote:
> > The only thing I can think of is that it might be related to the
> > transaction manager - for a long running transaction, the TM is probably
> > retaining log entries to accommodate a rollback. For such a long running
> > transaction, that could be a lot of overhead. What TM implementation are
> > you using?
> > I myself just finished an application which required ridiculous amounts
> > of data to be loaded on a daily basis in batch mode (gigabytes, millions
> > of records across many tables). I basically ensured that the insert
> > operations happened in 1000 count 'mini' batches for 2 reasons:
> >
> > 1) to prevent a TM timeout (default was 5 minutes)
> > 2) to prevent the TM overhead from getting too large (memory)
> >
> > Here's some code I used to do that (Java + Spring TransactionTemplate).
> > Now I don't know if you are experiencing the same thing I did, but you
> > might find it useful.
> >
> > - note that the Iterator passed in represented a custom implementation
> > that was reading records from a file stream - the file was over 10 Gigs,
> > and the Iterator represented each 'chunked' record in the stream.
> > - The 'flushCount' was a config variable from a .properties file - in our
> > case, it was equal to 1000).
> > - the flush() method just called a DAO which internally called
> > hibernateTemplate.flush() and then immediately hibernateTemplate.clear();
> > - you could try to extend the transaction timeout beyond the default
> > (e.g. 5 minutes) by configuring the transaction manager, but
> > odds are high you'd get into a TM housekeeping memory problem
> >
> > protected int importBatch(final Iterator iterator, final int flushCount)
> > { //Batch inserts need to be done as small chunks within their own
> > transaction to avoid
> > //transaction timeouts (and rollbacks) due to large data sets, so
> > //specify PROPAGATION_REQUIRES_NEW.
> > //
> > //If we did NOT do this, and tried to import hundreds of
> > thousands or millions of records,
> > //odds are high that the transaction would time out
> > //(e.g. after 5 minutes) and everything would be rolled back.
> > This is not desired.
> > TransactionDefinition txnDef =
> > new
> > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_N
> >EW);
> >
> > TransactionTemplate txnTemplate = new
> > TransactionTemplate(getTransactionManager(), txnDef);
> > Integer importCount = (Integer) txnTemplate.execute(new
> > TransactionCallback() {
> > public Object doInTransaction(TransactionStatus status) {
> > int batchRecordCount = 0;
> >
> > while (iterator.hasNext() ) {
> > Record record = (Record) iterator.next();
> > if (record != null) { //can be null if any records
> > are skipped
> > importRecord(record);
> > batchRecordCount++;
> > notifyRecordImported(record);
> > if (batchRecordCount % flushCount == 0) {
> > flush();
> > }
> > }
> > }
> > if (batchRecordCount > 0) {
> > flush();
> > }
> >
> > if (log.isInfoEnabled()) {
> > log.info("Batch imported " + batchRecordCount + "
> > records in a single transaction.");
> > }
> > return batchRecordCount;
> > }
> > });
> > return importCount != null ? importCount : 0;
> > }
> >
> > Note that if any of those 'mini' transactions fail, you have to do a
> > manual rollback of all of the records that went in. We accounted for
> > this by storing an id in the table where these fields were inserted, and
> > if any error occurred, performed a bulk delete ('delete from blah where
> > manual_tx_id = foo').
> >
> > HTH,
> >
> > Les
> >
> > On Thu, Sep 4, 2008 at 10:53 AM, Burt Beckwith<burt@...>wrote:
> >> I'm building a large database for load testing, so I'm creating millions
> >> of
> >> domain instances. I'm trying to use the standard bulk insert approach in
> >> Hibernate, i.e. flush() and clear() the session periodically, with the
> >> whole
> >> operation running in a transaction.
> >>
> >> The problem is the instances are still around - I run out of memory
> >> after a
> >> few hours with a heap set at 1G. I've turned off 2nd-level caching and
> >> looked
> >> at the Session in a debugger and clear() definitely works - it's empty
> >> afterwards. There are no mapped collections, and I'm not keeping any
> >> explicit
> >> references to the instances.
> >>
> >> But running in a profiler I can see the instance count steadily increase
> >> and
> >> never decrease. Running gc() has no effect.
> >>
> >> Any thoughts on what might be keeping a reference to these instances?
> >>
> >> Burt
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe from this list, please visit:
> >>
> >> http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
and has nothing to do with transactions. Btw - it's not a memory leak in the
classic sense, but a GC issue - domain instances were somehow referenced and
couldn't be GC'd. The profile instance count just kept growing and growing.
I kept stripping down the code to try to isolate what was going on, and it
even happened when I just created a single, 1-field dummy domain class and
didn't call a single method on it. The code was something like
for (int i = 0; i < 100000; i++) {
new Thing(name: "thing_${i}")
Thread.sleep(50); // to allow time to watch things in the profiler
}
While the loop was running I'd run gc() periodically, and it had no effect on
the domain class instance count, but clearly it was working since other class
counts dropped and the overall heap size dropped. Once the method exited,
finally the gc() call dropped the instance count down to near zero.
Burt
On Thursday 04 September 2008 5:34:39 pm Les Hazlewood wrote:
> Oops, I didn't give you all the code - the method I posted earlier
> performed one and only one 'mini' batch insert. It was called from another
> method that coordinated calling that method as necessary:
> public void importProductBatched(Product product) {
> if (log.isInfoEnabled()) {
> log.info("Performing a batching import for product of type [" +
> product.getClass().getName() + "]");
> }
> preBatch();
> Iterator i = product.iterator();
> while (i.hasNext()) {
> int batchImported = importBatch(i, getBatchCount(),
> getFlushCount());
> product.setCommittedBytesRead(product.getBytesRead());
> product.setCommittedRecordsRead(((RecordIterator)
> i).getIndex());
> log.info("Batched imported Count" + batchImported + " bytes
> read =" + product.getBytesRead() + "|" + ((RecordIterator) i).getIndex());
> }
> if (log.isInfoEnabled()) {
> log.info("Successfully imported batch for product of type [" +
> product.getClass().getName() + "]");
> }
> postBatch();
> }
>
> And then the importBatch method:
>
> protected int importBatch(final Iterator iterator, final int maxRecords,
> final int flushCount) {
> //Batch inserts need to be done as small chunks within their own
> transaction to avoid
> //transaction timeouts (and rollbacks) due to large data sets, so
> //specify PROPAGATION_REQUIRES_NEW.
> //
> //If we did NOT do this, and imported in one transaction, and the
> Product contained
> //thousands of records (as does happen during batch mode), odds are
> high that the transaction would time out
> //(e.g. after 5 minutes) and everything would be rolled back. This
> is not desired.
> TransactionDefinition txnDef =
> new
> DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW
>);
>
> TransactionTemplate txnTemplate = new
> TransactionTemplate(getTransactionManager(), txnDef);
> Integer importCount = (Integer) txnTemplate.execute(new
> TransactionCallback() {
> public Object doInTransaction(TransactionStatus status) {
> int batchRecordCount = 0;
>
> while (iterator.hasNext() && batchRecordCount < maxRecords)
> {
> Record record = (Record) iterator.next();
> if (record != null) { //can be null if any records are
> skipped
> importRecord(record);
> batchRecordCount++;
> notifyRecordImported(record);
> if (batchRecordCount % flushCount == 0) {
> flush();
> }
> }
> }
> if (batchRecordCount > 0) {
> flush();
> }
>
> if (log.isInfoEnabled()) {
> log.info("Batch imported " + batchRecordCount + "
> records in a single transaction.");
> }
> return batchRecordCount;
> }
> });
> return importCount != null ? importCount : 0;
> }
>
> Regards,
>
> Les
>
> On Thu, Sep 4, 2008 at 5:30 PM, Les Hazlewood <les@...> wrote:
> > The only thing I can think of is that it might be related to the
> > transaction manager - for a long running transaction, the TM is probably
> > retaining log entries to accommodate a rollback. For such a long running
> > transaction, that could be a lot of overhead. What TM implementation are
> > you using?
> > I myself just finished an application which required ridiculous amounts
> > of data to be loaded on a daily basis in batch mode (gigabytes, millions
> > of records across many tables). I basically ensured that the insert
> > operations happened in 1000 count 'mini' batches for 2 reasons:
> >
> > 1) to prevent a TM timeout (default was 5 minutes)
> > 2) to prevent the TM overhead from getting too large (memory)
> >
> > Here's some code I used to do that (Java + Spring TransactionTemplate).
> > Now I don't know if you are experiencing the same thing I did, but you
> > might find it useful.
> >
> > - note that the Iterator passed in represented a custom implementation
> > that was reading records from a file stream - the file was over 10 Gigs,
> > and the Iterator represented each 'chunked' record in the stream.
> > - The 'flushCount' was a config variable from a .properties file - in our
> > case, it was equal to 1000).
> > - the flush() method just called a DAO which internally called
> > hibernateTemplate.flush() and then immediately hibernateTemplate.clear();
> > - you could try to extend the transaction timeout beyond the default
> > (e.g. 5 minutes) by configuring the transaction manager, but
> > odds are high you'd get into a TM housekeeping memory problem
> >
> > protected int importBatch(final Iterator iterator, final int flushCount)
> > { //Batch inserts need to be done as small chunks within their own
> > transaction to avoid
> > //transaction timeouts (and rollbacks) due to large data sets, so
> > //specify PROPAGATION_REQUIRES_NEW.
> > //
> > //If we did NOT do this, and tried to import hundreds of
> > thousands or millions of records,
> > //odds are high that the transaction would time out
> > //(e.g. after 5 minutes) and everything would be rolled back.
> > This is not desired.
> > TransactionDefinition txnDef =
> > new
> > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_N
> >EW);
> >
> > TransactionTemplate txnTemplate = new
> > TransactionTemplate(getTransactionManager(), txnDef);
> > Integer importCount = (Integer) txnTemplate.execute(new
> > TransactionCallback() {
> > public Object doInTransaction(TransactionStatus status) {
> > int batchRecordCount = 0;
> >
> > while (iterator.hasNext() ) {
> > Record record = (Record) iterator.next();
> > if (record != null) { //can be null if any records
> > are skipped
> > importRecord(record);
> > batchRecordCount++;
> > notifyRecordImported(record);
> > if (batchRecordCount % flushCount == 0) {
> > flush();
> > }
> > }
> > }
> > if (batchRecordCount > 0) {
> > flush();
> > }
> >
> > if (log.isInfoEnabled()) {
> > log.info("Batch imported " + batchRecordCount + "
> > records in a single transaction.");
> > }
> > return batchRecordCount;
> > }
> > });
> > return importCount != null ? importCount : 0;
> > }
> >
> > Note that if any of those 'mini' transactions fail, you have to do a
> > manual rollback of all of the records that went in. We accounted for
> > this by storing an id in the table where these fields were inserted, and
> > if any error occurred, performed a bulk delete ('delete from blah where
> > manual_tx_id = foo').
> >
> > HTH,
> >
> > Les
> >
> > On Thu, Sep 4, 2008 at 10:53 AM, Burt Beckwith
> >> I'm building a large database for load testing, so I'm creating millions
> >> of
> >> domain instances. I'm trying to use the standard bulk insert approach in
> >> Hibernate, i.e. flush() and clear() the session periodically, with the
> >> whole
> >> operation running in a transaction.
> >>
> >> The problem is the instances are still around - I run out of memory
> >> after a
> >> few hours with a heap set at 1G. I've turned off 2nd-level caching and
> >> looked
> >> at the Session in a debugger and clear() definitely works - it's empty
> >> afterwards. There are no mapped collections, and I'm not keeping any
> >> explicit
> >> references to the instances.
> >>
> >> But running in a profiler I can see the instance count steadily increase
> >> and
> >> never decrease. Running gc() has no effect.
> >>
> >> Any thoughts on what might be keeping a reference to these instances?
> >>
> >> Burt
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe from this list, please visit:
> >>
> >> http://xircles.codehaus.org/manage_email
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
« Return to Thread: Memory leak
| Free embeddable forum powered by Nabble | Forum Help |
