[jira] Created: (DIRSERVER-644) Memory Leak in Persistent search ?

Memory Leak in Persistent search ?
----------------------------------

         Key: DIRSERVER-644
         URL: http://issues.apache.org/jira/browse/DIRSERVER-644
     Project: Directory ApacheDS
        Type: Bug

  Components: core  
    Versions: 1.0-RC3    
    Reporter: Emmanuel Lecharny
 Assigned to: Alex Karasulu
    Priority: Blocker
     Fix For: 1.0-RC4


After having profiled memory, it seems we have a memory leak in SessionRegistry.

A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
SessionRegistry.addOutstandingRequest
and
SessionRegistry.removeOutstandingRequest

Session Released
addOutstandingRequest 2
addOutstandingRequest 3
addOutstandingRequest 4
... ( 100 requests)
addOutstandingRequest 99
addOutstandingRequest 100
addOutstandingRequest 101
remove session

The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).

It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.


--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

     [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=all ]

Emmanuel Lecharny updated DIRSERVER-644:
----------------------------------------

    Attachment: SearchTest.java

The test used.

> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

    [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=comments#action_12416093 ]

Alex Karasulu commented on DIRSERVER-644:
-----------------------------------------

This actually has nothing to do with psearch but obviously psearch benefits the most from it since they never terminate ;).  Basically we store outstanding long running requests in the session and remove all this stuff when MINA tell's us the connection has dropped.

Perhaps the mechanism we have setup is not working correctly and the requests are collecting.  We need to get Trustin's input on this.



> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

    [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=comments#action_12416096 ]

Emmanuel Lecharny commented on DIRSERVER-644:
---------------------------------------------

That's good news !

However, requests are not supposed to be long here : they are simple search with a DN as key. The request must be discarded when the result has been sent to the client, not when the session dies.

Now, is it necesary to store this request ? What is this good for? If we have a very long request, like a search through all the base, then we will have three cases :
1) there is a time limit : we returns the data we found at this point.
2) there is a size limit : we returns the data we found at this point.
3) there are no limit : we wait for the server to build the full result, and we send it back to the client.

I any case, we don't need the Request to be stored, am I totally wrong ?

> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

    [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=comments#action_12416101 ]

Alex Karasulu commented on DIRSERVER-644:
-----------------------------------------

Man I was totally wrong about this.  We do not remove outstanding requests at the end of a search.  What we need to do is make the SearchResponseIterator remove the outstanding request from the session when it sends a response done result.



> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

     [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=all ]
     
Alex Karasulu resolved DIRSERVER-644:
-------------------------------------

    Resolution: Fixed

Fix committed revision 414011:


http://svn.apache.org/viewvc?view=rev&revision=414011

> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

     [ http://issues.apache.org/jira/browse/DIRSERVER-644?page=all ]
     
Emmanuel Lecharny closed DIRSERVER-644:
---------------------------------------


Closed after the fix has been applied

> Memory Leak in Persistent search ?
> ----------------------------------
>
>          Key: DIRSERVER-644
>          URL: http://issues.apache.org/jira/browse/DIRSERVER-644
>      Project: Directory ApacheDS
>         Type: Bug

>   Components: core
>     Versions: 1.0-RC3
>     Reporter: Emmanuel Lecharny
>     Assignee: Alex Karasulu
>     Priority: Blocker
>      Fix For: 1.0-RC4
>  Attachments: SearchTest.java
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

     [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny reopened DIRSERVER-644:
-----------------------------------------

      Assignee:     (was: Alex Karasulu)

Reopen the issue as the patch seems to have been applied in a branch but not have been merged in trunk.

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.0-RC4
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny updated DIRSERVER-644:
----------------------------------------

    Affects Version/s: 1.5.4
        Fix Version/s:     (was: 1.0-RC4)
                       1.5.5

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.5.4, 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.5.5
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722602#action_12722602 ]

Simon Temple commented on DIRSERVER-644:
----------------------------------------

I think the original patch has made it through although it's now in different classes due to refactoring.

However, I think there is a leak if a search is valid but unsuccessful.  I.e.  If the search returns an empty enumeration the server side registry continues to hold a reference to the search session.

Here's an example client snippet I used to re-create the problem:

<snip>
        Attributes matchAttrs = new BasicAttributes( "objectClass", "typeinfo", true );
        NamingEnumeration<SearchResult> en = null;
        en = ctx.search( name, matchAttrs, new String[] { "networkid" } );
        en.close(  );
<snip>

My proposed fix to the 1.5.2 code is in DefaultSearchHandler,  in searchMessageReceived() Line# 373:

            if ( list.hasMore() )
            {
                <snip>
            }
            else
            {
                <snip>

                // SPT - empty result lists should not accumulate registered sessions
                getSessionRegistry().removeOutstandingRequest( session, req.getMessageId() );
            }

I see there is a persistent search implementation that may also suffer the same problem.

Comments appreciated.

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.5.4, 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.5.5
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny reassigned DIRSERVER-644:
-------------------------------------------

    Assignee: Emmanuel Lecharny

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.5.4, 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Assignee: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.5.5
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733855#action_12733855 ]

Emmanuel Lecharny commented on DIRSERVER-644:
---------------------------------------------

The initial issue has been fixed when we moved to cursor implementation.

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.5.4, 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Assignee: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.5.5
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/DIRSERVER-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny closed DIRSERVER-644.
---------------------------------------

    Resolution: Fixed

Fixed a while ago.

> Memory Leak in Persistent search ?
> ----------------------------------
>
>                 Key: DIRSERVER-644
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-644
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.5.4, 1.0-RC3
>            Reporter: Emmanuel Lecharny
>            Assignee: Emmanuel Lecharny
>            Priority: Blocker
>             Fix For: 1.5.5
>
>         Attachments: SearchTest.java
>
>
> After having profiled memory, it seems we have a memory leak in SessionRegistry.
> A little test (attached) does a search N times for N threads, and for each search, a OutstandingRequest is attached to the session. After a few thousands of search we fall in OOM. I've put some trace in those methods :
> SessionRegistry.addOutstandingRequest
> and
> SessionRegistry.removeOutstandingRequest
> Session Released
> addOutstandingRequest 2
> addOutstandingRequest 3
> addOutstandingRequest 4
> ... ( 100 requests)
> addOutstandingRequest 99
> addOutstandingRequest 100
> addOutstandingRequest 101
> remove session
> The SessionRegistry.removeOutstandingRequest is never called, except if an exception is raised (NamingException).
> It may be on purpose ( persistent search), but we can't assume the server will be able to hold as many OutstandingRequest as we have search requests - or entries -.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.