« Return to Thread: Memory leak
Re: Memory leak
by Graeme Rocher-2
::
Rate this Message:
I believe the issue is that we override the default constructor to
perform data binding. The binding errors object is bound to either the
current thread or the current request
Since these errors are scoped to the request they won't be GC'ed until
the request completes. You could regard this as a bug. One work around
may be do do:
for (int i = 0; i < 100000; i++) {
def t = new Thing(name: "thing_${i}")
t.errors = null
Thread.sleep(50); // to allow time to watch things in the profiler
}
I'm not sure what a better solution is for now. Maybe we could use a
ReferenceQueue or something to check if its a candidate for GC'ing and
then automatically clear the errors... hmmm needs some thought
Cheers
On Fri, Sep 5, 2008 at 3:24 AM, Les Hazlewood <les@...> wrote:
> Hrm - that sounds kinda serious - and may even be a Groovy thing and not a
> Grails thing. Anyone from G2One wanna comment?
>
> On Thu, Sep 4, 2008 at 8:15 PM, Burt Beckwith <burt@...> wrote:
>>
>> I spent quite a while on this today and think the issue has to do with
>> Grails
>> and has nothing to do with transactions. Btw - it's not a memory leak in
>> the
>> classic sense, but a GC issue - domain instances were somehow referenced
>> and
>> couldn't be GC'd. The profile instance count just kept growing and
>> growing.
>>
>> I kept stripping down the code to try to isolate what was going on, and it
>> even happened when I just created a single, 1-field dummy domain class and
>> didn't call a single method on it. The code was something like
>>
>> for (int i = 0; i < 100000; i++) {
>> new Thing(name: "thing_${i}")
>> Thread.sleep(50); // to allow time to watch things in the profiler
>> }
>>
>> While the loop was running I'd run gc() periodically, and it had no effect
>> on
>> the domain class instance count, but clearly it was working since other
>> class
>> counts dropped and the overall heap size dropped. Once the method exited,
>> finally the gc() call dropped the instance count down to near zero.
>>
>> Burt
>>
>> On Thursday 04 September 2008 5:34:39 pm Les Hazlewood wrote:
>> > Oops, I didn't give you all the code - the method I posted earlier
>> > performed one and only one 'mini' batch insert. It was called from
>> > another
>> > method that coordinated calling that method as necessary:
>> > public void importProductBatched(Product product) {
>> > if (log.isInfoEnabled()) {
>> > log.info("Performing a batching import for product of type
>> > [" +
>> > product.getClass().getName() + "]");
>> > }
>> > preBatch();
>> > Iterator i = product.iterator();
>> > while (i.hasNext()) {
>> > int batchImported = importBatch(i, getBatchCount(),
>> > getFlushCount());
>> > product.setCommittedBytesRead(product.getBytesRead());
>> > product.setCommittedRecordsRead(((RecordIterator)
>> > i).getIndex());
>> > log.info("Batched imported Count" + batchImported + " bytes
>> > read =" + product.getBytesRead() + "|" + ((RecordIterator)
>> > i).getIndex());
>> > }
>> > if (log.isInfoEnabled()) {
>> > log.info("Successfully imported batch for product of type ["
>> > +
>> > product.getClass().getName() + "]");
>> > }
>> > postBatch();
>> > }
>> >
>> > And then the importBatch method:
>> >
>> > protected int importBatch(final Iterator iterator, final int maxRecords,
>> > final int flushCount) {
>> > //Batch inserts need to be done as small chunks within their own
>> > transaction to avoid
>> > //transaction timeouts (and rollbacks) due to large data sets,
>> > so
>> > //specify PROPAGATION_REQUIRES_NEW.
>> > //
>> > //If we did NOT do this, and imported in one transaction, and
>> > the
>> > Product contained
>> > //thousands of records (as does happen during batch mode), odds
>> > are
>> > high that the transaction would time out
>> > //(e.g. after 5 minutes) and everything would be rolled back.
>> > This
>> > is not desired.
>> > TransactionDefinition txnDef =
>> > new
>> >
>> > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW
>> >);
>> >
>> > TransactionTemplate txnTemplate = new
>> > TransactionTemplate(getTransactionManager(), txnDef);
>> > Integer importCount = (Integer) txnTemplate.execute(new
>> > TransactionCallback() {
>> > public Object doInTransaction(TransactionStatus status) {
>> > int batchRecordCount = 0;
>> >
>> > while (iterator.hasNext() && batchRecordCount <
>> > maxRecords)
>> > {
>> > Record record = (Record) iterator.next();
>> > if (record != null) { //can be null if any records
>> > are
>> > skipped
>> > importRecord(record);
>> > batchRecordCount++;
>> > notifyRecordImported(record);
>> > if (batchRecordCount % flushCount == 0) {
>> > flush();
>> > }
>> > }
>> > }
>> > if (batchRecordCount > 0) {
>> > flush();
>> > }
>> >
>> > if (log.isInfoEnabled()) {
>> > log.info("Batch imported " + batchRecordCount + "
>> > records in a single transaction.");
>> > }
>> > return batchRecordCount;
>> > }
>> > });
>> > return importCount != null ? importCount : 0;
>> > }
>> >
>> > Regards,
>> >
>> > Les
>> >
>> > On Thu, Sep 4, 2008 at 5:30 PM, Les Hazlewood <les@...> wrote:
>> > > The only thing I can think of is that it might be related to the
>> > > transaction manager - for a long running transaction, the TM is
>> > > probably
>> > > retaining log entries to accommodate a rollback. For such a long
>> > > running
>> > > transaction, that could be a lot of overhead. What TM implementation
>> > > are
>> > > you using?
>> > > I myself just finished an application which required ridiculous
>> > > amounts
>> > > of data to be loaded on a daily basis in batch mode (gigabytes,
>> > > millions
>> > > of records across many tables). I basically ensured that the insert
>> > > operations happened in 1000 count 'mini' batches for 2 reasons:
>> > >
>> > > 1) to prevent a TM timeout (default was 5 minutes)
>> > > 2) to prevent the TM overhead from getting too large (memory)
>> > >
>> > > Here's some code I used to do that (Java + Spring
>> > > TransactionTemplate).
>> > > Now I don't know if you are experiencing the same thing I did, but
>> > > you
>> > > might find it useful.
>> > >
>> > > - note that the Iterator passed in represented a custom implementation
>> > > that was reading records from a file stream - the file was over 10
>> > > Gigs,
>> > > and the Iterator represented each 'chunked' record in the stream.
>> > > - The 'flushCount' was a config variable from a .properties file - in
>> > > our
>> > > case, it was equal to 1000).
>> > > - the flush() method just called a DAO which internally called
>> > > hibernateTemplate.flush() and then immediately
>> > > hibernateTemplate.clear();
>> > > - you could try to extend the transaction timeout beyond the default
>> > > (e.g. 5 minutes) by configuring the transaction manager, but
>> > > odds are high you'd get into a TM housekeeping memory problem
>> > >
>> > > protected int importBatch(final Iterator iterator, final int
>> > > flushCount)
>> > > { //Batch inserts need to be done as small chunks within their own
>> > > transaction to avoid
>> > > //transaction timeouts (and rollbacks) due to large data sets,
>> > > so
>> > > //specify PROPAGATION_REQUIRES_NEW.
>> > > //
>> > > //If we did NOT do this, and tried to import hundreds of
>> > > thousands or millions of records,
>> > > //odds are high that the transaction would time out
>> > > //(e.g. after 5 minutes) and everything would be rolled back.
>> > > This is not desired.
>> > > TransactionDefinition txnDef =
>> > > new
>> > >
>> > > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_N
>> > >EW);
>> > >
>> > > TransactionTemplate txnTemplate = new
>> > > TransactionTemplate(getTransactionManager(), txnDef);
>> > > Integer importCount = (Integer) txnTemplate.execute(new
>> > > TransactionCallback() {
>> > > public Object doInTransaction(TransactionStatus status) {
>> > > int batchRecordCount = 0;
>> > >
>> > > while (iterator.hasNext() ) {
>> > > Record record = (Record) iterator.next();
>> > > if (record != null) { //can be null if any records
>> > > are skipped
>> > > importRecord(record);
>> > > batchRecordCount++;
>> > > notifyRecordImported(record);
>> > > if (batchRecordCount % flushCount == 0) {
>> > > flush();
>> > > }
>> > > }
>> > > }
>> > > if (batchRecordCount > 0) {
>> > > flush();
>> > > }
>> > >
>> > > if (log.isInfoEnabled()) {
>> > > log.info("Batch imported " + batchRecordCount + "
>> > > records in a single transaction.");
>> > > }
>> > > return batchRecordCount;
>> > > }
>> > > });
>> > > return importCount != null ? importCount : 0;
>> > > }
>> > >
>> > > Note that if any of those 'mini' transactions fail, you have to do a
>> > > manual rollback of all of the records that went in. We accounted for
>> > > this by storing an id in the table where these fields were inserted,
>> > > and
>> > > if any error occurred, performed a bulk delete ('delete from blah
>> > > where
>> > > manual_tx_id = foo').
>> > >
>> > > HTH,
>> > >
>> > > Les
>> > >
>> > > On Thu, Sep 4, 2008 at 10:53 AM, Burt Beckwith
>> <burt@...>wrote:
>> > >> I'm building a large database for load testing, so I'm creating
>> > >> millions
>> > >> of
>> > >> domain instances. I'm trying to use the standard bulk insert approach
>> > >> in
>> > >> Hibernate, i.e. flush() and clear() the session periodically, with
>> > >> the
>> > >> whole
>> > >> operation running in a transaction.
>> > >>
>> > >> The problem is the instances are still around - I run out of memory
>> > >> after a
>> > >> few hours with a heap set at 1G. I've turned off 2nd-level caching
>> > >> and
>> > >> looked
>> > >> at the Session in a debugger and clear() definitely works - it's
>> > >> empty
>> > >> afterwards. There are no mapped collections, and I'm not keeping any
>> > >> explicit
>> > >> references to the instances.
>> > >>
>> > >> But running in a profiler I can see the instance count steadily
>> > >> increase
>> > >> and
>> > >> never decrease. Running gc() has no effect.
>> > >>
>> > >> Any thoughts on what might be keeping a reference to these instances?
>> > >>
>> > >> Burt
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe from this list, please visit:
>> > >>
>> > >> http://xircles.codehaus.org/manage_email
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>> http://xircles.codehaus.org/manage_email
>>
>>
>
>
--
Graeme Rocher
Grails Project Lead
G2One, Inc. Chief Technology Officer
http://www.g2one.com
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
perform data binding. The binding errors object is bound to either the
current thread or the current request
Since these errors are scoped to the request they won't be GC'ed until
the request completes. You could regard this as a bug. One work around
may be do do:
for (int i = 0; i < 100000; i++) {
def t = new Thing(name: "thing_${i}")
t.errors = null
Thread.sleep(50); // to allow time to watch things in the profiler
}
I'm not sure what a better solution is for now. Maybe we could use a
ReferenceQueue or something to check if its a candidate for GC'ing and
then automatically clear the errors... hmmm needs some thought
Cheers
On Fri, Sep 5, 2008 at 3:24 AM, Les Hazlewood <les@...> wrote:
> Hrm - that sounds kinda serious - and may even be a Groovy thing and not a
> Grails thing. Anyone from G2One wanna comment?
>
> On Thu, Sep 4, 2008 at 8:15 PM, Burt Beckwith <burt@...> wrote:
>>
>> I spent quite a while on this today and think the issue has to do with
>> Grails
>> and has nothing to do with transactions. Btw - it's not a memory leak in
>> the
>> classic sense, but a GC issue - domain instances were somehow referenced
>> and
>> couldn't be GC'd. The profile instance count just kept growing and
>> growing.
>>
>> I kept stripping down the code to try to isolate what was going on, and it
>> even happened when I just created a single, 1-field dummy domain class and
>> didn't call a single method on it. The code was something like
>>
>> for (int i = 0; i < 100000; i++) {
>> new Thing(name: "thing_${i}")
>> Thread.sleep(50); // to allow time to watch things in the profiler
>> }
>>
>> While the loop was running I'd run gc() periodically, and it had no effect
>> on
>> the domain class instance count, but clearly it was working since other
>> class
>> counts dropped and the overall heap size dropped. Once the method exited,
>> finally the gc() call dropped the instance count down to near zero.
>>
>> Burt
>>
>> On Thursday 04 September 2008 5:34:39 pm Les Hazlewood wrote:
>> > Oops, I didn't give you all the code - the method I posted earlier
>> > performed one and only one 'mini' batch insert. It was called from
>> > another
>> > method that coordinated calling that method as necessary:
>> > public void importProductBatched(Product product) {
>> > if (log.isInfoEnabled()) {
>> > log.info("Performing a batching import for product of type
>> > [" +
>> > product.getClass().getName() + "]");
>> > }
>> > preBatch();
>> > Iterator i = product.iterator();
>> > while (i.hasNext()) {
>> > int batchImported = importBatch(i, getBatchCount(),
>> > getFlushCount());
>> > product.setCommittedBytesRead(product.getBytesRead());
>> > product.setCommittedRecordsRead(((RecordIterator)
>> > i).getIndex());
>> > log.info("Batched imported Count" + batchImported + " bytes
>> > read =" + product.getBytesRead() + "|" + ((RecordIterator)
>> > i).getIndex());
>> > }
>> > if (log.isInfoEnabled()) {
>> > log.info("Successfully imported batch for product of type ["
>> > +
>> > product.getClass().getName() + "]");
>> > }
>> > postBatch();
>> > }
>> >
>> > And then the importBatch method:
>> >
>> > protected int importBatch(final Iterator iterator, final int maxRecords,
>> > final int flushCount) {
>> > //Batch inserts need to be done as small chunks within their own
>> > transaction to avoid
>> > //transaction timeouts (and rollbacks) due to large data sets,
>> > so
>> > //specify PROPAGATION_REQUIRES_NEW.
>> > //
>> > //If we did NOT do this, and imported in one transaction, and
>> > the
>> > Product contained
>> > //thousands of records (as does happen during batch mode), odds
>> > are
>> > high that the transaction would time out
>> > //(e.g. after 5 minutes) and everything would be rolled back.
>> > This
>> > is not desired.
>> > TransactionDefinition txnDef =
>> > new
>> >
>> > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW
>> >);
>> >
>> > TransactionTemplate txnTemplate = new
>> > TransactionTemplate(getTransactionManager(), txnDef);
>> > Integer importCount = (Integer) txnTemplate.execute(new
>> > TransactionCallback() {
>> > public Object doInTransaction(TransactionStatus status) {
>> > int batchRecordCount = 0;
>> >
>> > while (iterator.hasNext() && batchRecordCount <
>> > maxRecords)
>> > {
>> > Record record = (Record) iterator.next();
>> > if (record != null) { //can be null if any records
>> > are
>> > skipped
>> > importRecord(record);
>> > batchRecordCount++;
>> > notifyRecordImported(record);
>> > if (batchRecordCount % flushCount == 0) {
>> > flush();
>> > }
>> > }
>> > }
>> > if (batchRecordCount > 0) {
>> > flush();
>> > }
>> >
>> > if (log.isInfoEnabled()) {
>> > log.info("Batch imported " + batchRecordCount + "
>> > records in a single transaction.");
>> > }
>> > return batchRecordCount;
>> > }
>> > });
>> > return importCount != null ? importCount : 0;
>> > }
>> >
>> > Regards,
>> >
>> > Les
>> >
>> > On Thu, Sep 4, 2008 at 5:30 PM, Les Hazlewood <les@...> wrote:
>> > > The only thing I can think of is that it might be related to the
>> > > transaction manager - for a long running transaction, the TM is
>> > > probably
>> > > retaining log entries to accommodate a rollback. For such a long
>> > > running
>> > > transaction, that could be a lot of overhead. What TM implementation
>> > > are
>> > > you using?
>> > > I myself just finished an application which required ridiculous
>> > > amounts
>> > > of data to be loaded on a daily basis in batch mode (gigabytes,
>> > > millions
>> > > of records across many tables). I basically ensured that the insert
>> > > operations happened in 1000 count 'mini' batches for 2 reasons:
>> > >
>> > > 1) to prevent a TM timeout (default was 5 minutes)
>> > > 2) to prevent the TM overhead from getting too large (memory)
>> > >
>> > > Here's some code I used to do that (Java + Spring
>> > > TransactionTemplate).
>> > > Now I don't know if you are experiencing the same thing I did, but
>> > > you
>> > > might find it useful.
>> > >
>> > > - note that the Iterator passed in represented a custom implementation
>> > > that was reading records from a file stream - the file was over 10
>> > > Gigs,
>> > > and the Iterator represented each 'chunked' record in the stream.
>> > > - The 'flushCount' was a config variable from a .properties file - in
>> > > our
>> > > case, it was equal to 1000).
>> > > - the flush() method just called a DAO which internally called
>> > > hibernateTemplate.flush() and then immediately
>> > > hibernateTemplate.clear();
>> > > - you could try to extend the transaction timeout beyond the default
>> > > (e.g. 5 minutes) by configuring the transaction manager, but
>> > > odds are high you'd get into a TM housekeeping memory problem
>> > >
>> > > protected int importBatch(final Iterator iterator, final int
>> > > flushCount)
>> > > { //Batch inserts need to be done as small chunks within their own
>> > > transaction to avoid
>> > > //transaction timeouts (and rollbacks) due to large data sets,
>> > > so
>> > > //specify PROPAGATION_REQUIRES_NEW.
>> > > //
>> > > //If we did NOT do this, and tried to import hundreds of
>> > > thousands or millions of records,
>> > > //odds are high that the transaction would time out
>> > > //(e.g. after 5 minutes) and everything would be rolled back.
>> > > This is not desired.
>> > > TransactionDefinition txnDef =
>> > > new
>> > >
>> > > DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_N
>> > >EW);
>> > >
>> > > TransactionTemplate txnTemplate = new
>> > > TransactionTemplate(getTransactionManager(), txnDef);
>> > > Integer importCount = (Integer) txnTemplate.execute(new
>> > > TransactionCallback() {
>> > > public Object doInTransaction(TransactionStatus status) {
>> > > int batchRecordCount = 0;
>> > >
>> > > while (iterator.hasNext() ) {
>> > > Record record = (Record) iterator.next();
>> > > if (record != null) { //can be null if any records
>> > > are skipped
>> > > importRecord(record);
>> > > batchRecordCount++;
>> > > notifyRecordImported(record);
>> > > if (batchRecordCount % flushCount == 0) {
>> > > flush();
>> > > }
>> > > }
>> > > }
>> > > if (batchRecordCount > 0) {
>> > > flush();
>> > > }
>> > >
>> > > if (log.isInfoEnabled()) {
>> > > log.info("Batch imported " + batchRecordCount + "
>> > > records in a single transaction.");
>> > > }
>> > > return batchRecordCount;
>> > > }
>> > > });
>> > > return importCount != null ? importCount : 0;
>> > > }
>> > >
>> > > Note that if any of those 'mini' transactions fail, you have to do a
>> > > manual rollback of all of the records that went in. We accounted for
>> > > this by storing an id in the table where these fields were inserted,
>> > > and
>> > > if any error occurred, performed a bulk delete ('delete from blah
>> > > where
>> > > manual_tx_id = foo').
>> > >
>> > > HTH,
>> > >
>> > > Les
>> > >
>> > > On Thu, Sep 4, 2008 at 10:53 AM, Burt Beckwith
>> <burt@...>wrote:
>> > >> I'm building a large database for load testing, so I'm creating
>> > >> millions
>> > >> of
>> > >> domain instances. I'm trying to use the standard bulk insert approach
>> > >> in
>> > >> Hibernate, i.e. flush() and clear() the session periodically, with
>> > >> the
>> > >> whole
>> > >> operation running in a transaction.
>> > >>
>> > >> The problem is the instances are still around - I run out of memory
>> > >> after a
>> > >> few hours with a heap set at 1G. I've turned off 2nd-level caching
>> > >> and
>> > >> looked
>> > >> at the Session in a debugger and clear() definitely works - it's
>> > >> empty
>> > >> afterwards. There are no mapped collections, and I'm not keeping any
>> > >> explicit
>> > >> references to the instances.
>> > >>
>> > >> But running in a profiler I can see the instance count steadily
>> > >> increase
>> > >> and
>> > >> never decrease. Running gc() has no effect.
>> > >>
>> > >> Any thoughts on what might be keeping a reference to these instances?
>> > >>
>> > >> Burt
>> > >>
>> > >> ---------------------------------------------------------------------
>> > >> To unsubscribe from this list, please visit:
>> > >>
>> > >> http://xircles.codehaus.org/manage_email
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>> http://xircles.codehaus.org/manage_email
>>
>>
>
>
--
Graeme Rocher
Grails Project Lead
G2One, Inc. Chief Technology Officer
http://www.g2one.com
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
« Return to Thread: Memory leak
| Free embeddable forum powered by Nabble | Forum Help |
