[jira] Created: (ODE-639) Memory leak in InMem DAO

Memory leak in InMem DAO
------------------------

                 Key: ODE-639
                 URL: https://issues.apache.org/jira/browse/ODE-639
             Project: ODE
          Issue Type: Bug
          Components: BPEL Runtime
    Affects Versions: 1.3.2
         Environment: any
            Reporter: Rafal Rusin
            Assignee: Rafal Rusin


In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.

I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.

A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rafal Rusin resolved ODE-639.
-----------------------------

    Resolution: Fixed

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734596#action_12734596 ]

Ciaran Jessup commented on ODE-639:
-----------------------------------

Excellent news, if this is the case you've just saved me a couple of days work I've had to schedule to look at why we're (still) receiving OutOfMemory errors in our ODE workflows (even after all the memory leak purges I went through a month or so ago.)  I look forward to testing this patch!

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rafal Rusin updated ODE-639:
----------------------------

    Attachment: memdao-dumper.patch

Attaches is a memdao to xml dumper using xstream.
It does a dump each 30 secs.
I used it for diagnosing leak.

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>         Attachments: memdao-dumper.patch
>
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774401#action_12774401 ]

Karthick Sankarachary commented on ODE-639:
-------------------------------------------

I believe we already cleanup process instances on a process-by-process basis. As you can tell from the declaration below, the _lastRemoval field is local to the process DAO (at least in ODE 1.X):

    private volatile long _lastRemoval = 0;

Right now, every time a new instance is started, the process checks to see if it can clean up any instances that have been running for more than 10 minutes. I think a cleaner approach would be to try and discard long-running instances periodically (say every minute), as opposed to just when a new instance starts.

The rationale behind the TTL constraint is that in-memory processes are typically short-lived and should not have to run for longer than 10 minutes.  Having said that, if there is a good reason for creating in-memory processes that need to run for more than 10 minutes, then we may want to make that TTL threshold configurable (i.e., initialize it based on a user-defined configuration parameter).

Finally, I see that you marked a number of DAO fields as transient, which seems redundant, considering that the in-memory DAO objects are never serialized.

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>         Attachments: memdao-dumper.patch
>
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774406#action_12774406 ]

Rafal Rusin commented on ODE-639:
---------------------------------

What's up with this comment?
memdao-dumper was just for diagnosing leak. For actual commits, please check subversion commits.
BTW. This issue is fixed. If you have some problems with it, please reopen.

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>         Attachments: memdao-dumper.patch
>
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774427#action_12774427 ]

Karthick Sankarachary commented on ODE-639:
-------------------------------------------

Rafal, Can you please add the subversion revision number of your actual commit here? Thanks, Karthick

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>         Attachments: memdao-dumper.patch
>
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/ODE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774431#action_12774431 ]

Rafal Rusin commented on ODE-639:
---------------------------------

#797066, #797065

> Memory leak in InMem DAO
> ------------------------
>
>                 Key: ODE-639
>                 URL: https://issues.apache.org/jira/browse/ODE-639
>             Project: ODE
>          Issue Type: Bug
>          Components: BPEL Runtime
>    Affects Versions: 1.3.2
>         Environment: any
>            Reporter: Rafal Rusin
>            Assignee: Rafal Rusin
>         Attachments: memdao-dumper.patch
>
>
> In Memory DAO, there is a cleanup strategy, which assigns TTL 10 min. for in-memory instances.
> I works for 1 in-memory process, but for 2 or more, there are cases when it doesn't do cleanup.
> I figured out that cleaning strategy in bpel-runtime/src/main/java/org/apache/ode/bpel/memdao/ProcessDaoImpl.java holds a static _lastRemoval field, which is updated globally. But if there are >1 in-mem processes, it may cause only one to do cleanup.
> A simple patch, which removes static declaration solves problem, because then, cleanup is done separately for each process.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.