possible memory leak issue in 1.2 dev (as of SVN:821662)

[Ferenc Pápai]

Hi

I think I discovered a memory leak while an XPath query is executed using Xindice 1.2 dev svn revision number 821662.

I only had to run a query that generated a sufficiently large number of results.

After some debugging I discovered that in class XPathQueryResolver.ResultSet
the XPathResolverContext member variable is cached now as opposed to as it was in its previous version.
This is probably for performance reasons.

This class XPathResolverContext has internally a DTMManager implementation.
As ResultSet.prepareNextNode method is called many times (while one query is executed),  the DTMManager instance will grow and grow in size until JVM runs out of memory.

Calling XPathResolverContext .reset() in ResultSet.prepareNextNode would solve this problem but it would be against the intent of optimizing the code.


I am not familiar yet with Xalan's DTM API to provide a proper solution. So i am just speculating here:
Internally DTMManager caches DTM objects generated while it executed the query on a given context. These DTMs consume the memory.
I think these are cached to be reused by the two-way iterators provided by DTMManager.
However in Xindice, the Collection API provides the NodeSet as forward only iterator, thus no DTM caching is need for Xindice. ?


regards,
Ferenc

[Natalia Shilenkova]

Ferenc,

Thank you for reporting the problem. Could you please let me know what
OS and JVM version you use to run Xindice (please include all the
information "java -version" reports)?

I'll investigate the problem.

Regards,
Natalia

2009/10/21 Ferenc Pápai <fpapai@...>:

>
>
> Hi
>
> I think I discovered a memory leak while an XPath query is executed using
> Xindice 1.2 dev svn revision number 821662.
>
> I only had to run a query that generated a sufficiently large number of
> results.
>
> After some debugging I discovered that in class XPathQueryResolver.ResultSet
> the XPathResolverContext member variable is cached now as opposed to as it
> was in its previous version.
> This is probably for performance reasons.
>
> This class XPathResolverContext has internally a DTMManager implementation.
> As ResultSet.prepareNextNode method is called many times (while one query is
> executed),  the DTMManager instance will grow and grow in size until JVM
> runs out of memory.
>
> Calling XPathResolverContext .reset() in ResultSet.prepareNextNode would
> solve this problem but it would be against the intent of optimizing the
> code.
>
>
> I am not familiar yet with Xalan's DTM API to provide a proper solution. So
> i am just speculating here:
> Internally DTMManager caches DTM objects generated while it executed the
> query on a given context. These DTMs consume the memory.
> I think these are cached to be reused by the two-way iterators provided by
> DTMManager.
> However in Xindice, the Collection API provides the NodeSet as forward only
> iterator, thus no DTM caching is need for Xindice. ?
>
>
> regards,
> Ferenc
>
>
>
>
>
>
>
>
>

[Ferenc Pápai]

Hi Natalia,

I had the issue on 2 Vistas and 1 XP, all running Java 1.6.0_16

Output on on Vista:

C:\Users\feri>java -version
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing)

let me know how can I help more
Ferenc


On Fri, Oct 23, 2009 at 13:50, Natalia Shilenkova <nshilenkova@...> wrote:

Ferenc,

Thank you for reporting the problem. Could you please let me know what
OS and JVM version you use to run Xindice (please include all the
information "java -version" reports)?

I'll investigate the problem.

Regards,
Natalia

2009/10/21 Ferenc Pápai <fpapai@...>:

>
>
> Hi
>
> I think I discovered a memory leak while an XPath query is executed using
> Xindice 1.2 dev svn revision number 821662.
>
> I only had to run a query that generated a sufficiently large number of
> results.
>
> After some debugging I discovered that in class XPathQueryResolver.ResultSet
> the XPathResolverContext member variable is cached now as opposed to as it
> was in its previous version.
> This is probably for performance reasons.
>
> This class XPathResolverContext has internally a DTMManager implementation.
> As ResultSet.prepareNextNode method is called many times (while one query is
> executed),  the DTMManager instance will grow and grow in size until JVM
> runs out of memory.
>
> Calling XPathResolverContext .reset() in ResultSet.prepareNextNode would
> solve this problem but it would be against the intent of optimizing the
> code.
>
>
> I am not familiar yet with Xalan's DTM API to provide a proper solution. So
> i am just speculating here:
> Internally DTMManager caches DTM objects generated while it executed the
> query on a given context. These DTMs consume the memory.
> I think these are cached to be reused by the two-way iterators provided by
> DTMManager.
> However in Xindice, the Collection API provides the NodeSet as forward only
> iterator, thus no DTM caching is need for Xindice. ?
>
>
> regards,
> Ferenc
>
>
>
>
>
>
>
>
>

[Natalia Shilenkova]

Ferenc,

The latest trunk shouldn't have memory problems with XPath queries. Unfortunately, it is slower now, too, as there were not many options for fixing the problem short of reimplementing DTMManager, so the patch is the solution you proposed.

Again, thank you for reporting the bug and providing fix for it.

Natalia

On Oct 23, 2009, at 5:04 PM, Ferenc Pápai wrote:

Hi Natalia,

I had the issue on 2 Vistas and 1 XP, all running Java 1.6.0_16

Output on on Vista:

C:\Users\feri>java -version
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing)

let me know how can I help more
Ferenc


On Fri, Oct 23, 2009 at 13:50, Natalia Shilenkova <nshilenkova@...> wrote:

Ferenc,

Thank you for reporting the problem. Could you please let me know what
OS and JVM version you use to run Xindice (please include all the
information "java -version" reports)?

I'll investigate the problem.

Regards,
Natalia

2009/10/21 Ferenc Pápai <fpapai@...>:

>
>
> Hi
>
> I think I discovered a memory leak while an XPath query is executed using
> Xindice 1.2 dev svn revision number 821662.
>
> I only had to run a query that generated a sufficiently large number of
> results.
>
> After some debugging I discovered that in class XPathQueryResolver.ResultSet
> the XPathResolverContext member variable is cached now as opposed to as it
> was in its previous version.
> This is probably for performance reasons.
>
> This class XPathResolverContext has internally a DTMManager implementation.
> As ResultSet.prepareNextNode method is called many times (while one query is
> executed),  the DTMManager instance will grow and grow in size until JVM
> runs out of memory.
>
> Calling XPathResolverContext .reset() in ResultSet.prepareNextNode would
> solve this problem but it would be against the intent of optimizing the
> code.
>
>
> I am not familiar yet with Xalan's DTM API to provide a proper solution. So
> i am just speculating here:
> Internally DTMManager caches DTM objects generated while it executed the
> query on a given context. These DTMs consume the memory.
> I think these are cached to be reused by the two-way iterators provided by
> DTMManager.
> However in Xindice, the Collection API provides the NodeSet as forward only
> iterator, thus no DTM caching is need for Xindice. ?
>
>
> regards,
> Ferenc
>
>
>
>
>
>
>
>
>

[Ferenc Pápai]

hello Natalia,

thanks for your time.
I think the current version is still faster than as in Xindice 1.1 where the whole context object was recreated in each iteration.

One possible improvement might be to reset the context at every "n" iteration only.
However this might work only if we could get a hint of the size of the context somehow.
Iterating over large documents would produce more DTM objects than small documents.

I'll try to research this when I'll have time.

regards,
Ferenc

On Fri, Oct 30, 2009 at 02:30, Natalia Shilenkova <nshilenkova@...> wrote:

Ferenc,

The latest trunk shouldn't have memory problems with XPath queries. Unfortunately, it is slower now, too, as there were not many options for fixing the problem short of reimplementing DTMManager, so the patch is the solution you proposed.

Again, thank you for reporting the bug and providing fix for it.

Natalia

On Oct 23, 2009, at 5:04 PM, Ferenc Pápai wrote:

Hi Natalia,

I had the issue on 2 Vistas and 1 XP, all running Java 1.6.0_16

Output on on Vista:

C:\Users\feri>java -version
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing)

let me know how can I help more
Ferenc


On Fri, Oct 23, 2009 at 13:50, Natalia Shilenkova <nshilenkova@...> wrote:

Ferenc,

Thank you for reporting the problem. Could you please let me know what
OS and JVM version you use to run Xindice (please include all the
information "java -version" reports)?

I'll investigate the problem.

Regards,
Natalia

2009/10/21 Ferenc Pápai <fpapai@...>:

>
>
> Hi
>
> I think I discovered a memory leak while an XPath query is executed using
> Xindice 1.2 dev svn revision number 821662.
>
> I only had to run a query that generated a sufficiently large number of
> results.
>
> After some debugging I discovered that in class XPathQueryResolver.ResultSet
> the XPathResolverContext member variable is cached now as opposed to as it
> was in its previous version.
> This is probably for performance reasons.
>
> This class XPathResolverContext has internally a DTMManager implementation.
> As ResultSet.prepareNextNode method is called many times (while one query is
> executed),  the DTMManager instance will grow and grow in size until JVM
> runs out of memory.
>
> Calling XPathResolverContext .reset() in ResultSet.prepareNextNode would
> solve this problem but it would be against the intent of optimizing the
> code.
>
>
> I am not familiar yet with Xalan's DTM API to provide a proper solution. So
> i am just speculating here:
> Internally DTMManager caches DTM objects generated while it executed the
> query on a given context. These DTMs consume the memory.
> I think these are cached to be reused by the two-way iterators provided by
> DTMManager.
> However in Xindice, the Collection API provides the NodeSet as forward only
> iterator, thus no DTM caching is need for Xindice. ?
>
>
> regards,
> Ferenc
>
>
>
>
>
>
>
>
>