[jira] Created: (LUCENE-504) FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount

FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
--------------------------------------------------------------------------------------------------------------------------------------------------

         Key: LUCENE-504
         URL: http://issues.apache.org/jira/browse/LUCENE-504
     Project: Lucene - Java
        Type: Bug
  Components: Search  
    Versions: 1.9    
    Reporter: Joerg Henss
    Priority: Minor


PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ http://issues.apache.org/jira/browse/LUCENE-504?page=all ]

Joerg Henss updated LUCENE-504:
-------------------------------

    Attachment: TestFuzzyQueryError.java

Simple test showing the error

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12414438 ]

Paul Borgermans commented on LUCENE-504:
----------------------------------------

Here the same problem, used a MultiFieldQueryParser in combination with fuzzy search which triggered the exception

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12415800 ]

Doron Cohen commented on LUCENE-504:
------------------------------------

LuceneFAQ item    "Why am I getting a TooManyClauses exception?"  
suggests:    "use BooleanQuery.setMaxClauseCount(Integer.MAX_VALUE)".

This would cause PriorityQueue to create an array of size maxint:
  - SUN JRE throws an out-of-memory error.
  - IBM JRE throws a "too large allocation" runtime error.

Seems that at least two fixes are required:
  + BooleanQuery - hard limit on value of maxClauseCount
  + PriorityQueue - hard limit on size of queue, since it is stored in memory.

The attached would set 1,000,000 hard limit - defined in PriorityQueu  - I think the most intensive use of
priority queue is for docs/hits during search, and 1M docs seems sufficient, unless there are uses that I am not aware of.


> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ http://issues.apache.org/jira/browse/LUCENE-504?page=all ]

Doron Cohen updated LUCENE-504:
-------------------------------

    Attachment: BooleanQuery.java.diff
                PriorityQueue.java.diff

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12415807 ]

Otis Gospodnetic commented on LUCENE-504:
-----------------------------------------

I imagine the limit will depend on how big of a heap you allow, no?  What happens if you increase the heap size drammatically with -XmxNNNm?


> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12415825 ]

Doron Cohen commented on LUCENE-504:
------------------------------------

Yes this is correct - e.g. on a win32 machine with 2GB RAM, SUN 1.5 JRE would accept up to Xmx1470m and in that case you could set the limit on the queue  size to 355,638,512 - 17% of maxint, before getting an out of mem error.  

For allowing the caller maximal flexibility (and responsibility), BooleanQuery could interpret the maxint as a hint saying "maximal possible value" and then silently modify it to maxint-1, thereby avoiding the negative array size issue in PriorityQueue (and possibly fail later with out of memory).

Is this what you have in mind?

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12416168 ]

Nadav Har'El commented on LUCENE-504:
-------------------------------------

Hi Doron and Otis,

My view is that this bug is a problem in FuzzyQuery, not in PriorityQueue or BooleanQuery. It is the caller's duty to create a priority queue with a sensible size, and it's not BooleanQuery's fault that other classes are using its getMaxClauseCount() wrongly. Moreover, changing PriorityQueue or BooleanQuery in the way Doron did, might potentially have side-effects because they are used in many places in Lucene. How do we know that nowhere in Lucene will we ever need a priority queue with a million elements?

Therefore I suggest a different patch, changing only FuzzyQuery.

he idea in the patch I'm attaching is that while FuzzyQuery *is allowed to* create BooleanQuery.getMaxClauseCount() clauses, it doesn't have to do so, and doesn't need to create a priority queue of that size when this number is huge. I added to FuzzyQuery its own maxClauseCount (with per-instance getter/setter methods)  and FuzzyQuery will never try to create more than this number of clauses, even if Boolean.getMaxClauseCount() is huge.

I set FuzzyQuery's default maxClauseCount to 1024, for backward compatibility (this is what happens today when you use BooleanQuery's defaults). However, it is likely that a user will want to set this number much lower than that. In fact, the whole idea of the priority queue is that it may be enough to take only a small number the most similar terms, so setting a FuzzyQuery's maxClauseCount to 100 or even 10 makes sense to me. This new feature is an added benefit of my patch.

My patch also includes Javadoc changes describing the new feature, and a new test case that failed before the fix, and succeeds after this patch.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ http://issues.apache.org/jira/browse/LUCENE-504?page=all ]

Nadav Har'El updated LUCENE-504:
--------------------------------

    Attachment: fuzzyquery.patch

This is my proposed patch described above.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java, fuzzyquery.patch
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12418446 ]

Nadav Har'El commented on LUCENE-504:
-------------------------------------

Hi Otis, you did not comment on my patch (fuzzyquery.patch), which I think solves your objections to Doron's previous patch. Do you also see problems in this patchl, or would prefer a different approach to solving this issue?
If not, perhaps you can commit this patch?
Thanks in advance, Nadav.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java, fuzzyquery.patch
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ http://issues.apache.org/jira/browse/LUCENE-504?page=comments#action_12419624 ]

Doron Cohen commented on LUCENE-504:
------------------------------------

I think it makes sense to separate here between efficiency and correctness.

The proposed fix above deals with efficiency, and maybe it should become a new separate future issue, perhaps also considering a more lazy allocation of space by PriorityQueue.

This issue is about correctness - setting max-clause "by-the-book" throws an exception - let's fix just that..

Modifying PriorityQueue to be tolerant to maxint capacity is one possible solution, though perhaps an overkill.  

A simpler way is for PriorityQueue to silently decrement its maxsize by 1 in case maxsize is requested to be maxint.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: LUCENE-504
>          URL: http://issues.apache.org/jira/browse/LUCENE-504
>      Project: Lucene - Java
>         Type: Bug

>   Components: Search
>     Versions: 1.9
>     Reporter: Joerg Henss
>     Priority: Minor
>  Attachments: BooleanQuery.java.diff, PriorityQueue.java.diff, TestFuzzyQueryError.java, fuzzyquery.patch
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624345#action_12624345 ]

Mark Miller commented on LUCENE-504:
------------------------------------

This really should be fixed. I agree with Nadav, the problem is with the Queries that use MaxClauses to build the priority queue - its just not good use of the queue.

I also agree with Doron that its really two distinct issues though:

1. FuzzyQuery et al should be using lazy allocation of some kind. It makes no sense to create an array that large for every multi term query, especially when we direct users to possibly use Integer.MAX_VALUE.

2.  The FAQ should not say to use Integer.MAX_VALUE if its going to kak with FuzzyQuery. At a minimum, FuzzyQuery should not init the priority queue larger than Integer.MAX_VALUE-1.

I suppose, that almost combines the 2 issues though - we are telling users to set the max clauses to a number that will force ridiculously wasteful memory allocation. Almost seems like both issues really need to be addressed together. Or maybe just the FAQ changed :)

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657602#action_12657602 ]

George Papas commented on LUCENE-504:
-------------------------------------

Hi,

This is still an issue in 2.4.0.  I know this is low priority, but has there been any more thinking about how to address this?

Thanks
George.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731744#action_12731744 ]

Florian Waltersdorfer commented on LUCENE-504:
----------------------------------------------

Hi,

something along the line of
ScoreDoc[] results = searcher.search(query, MAX_NUMBER_OF_HITS).scoreDocs;
causes a viable crash. I would consider this a "natural" start when trying to write an own searcher (MAX_NUMBER_OF_HITS beeing Integer.MAX_VALUE) and without the sources attached & the (eclipse) debugger I would have never found the problem I'd guess.
So please do something about this pitfall, if only adding a warning in the javadoc of i.e. Searcher.search(..., int n, ...)

Thanks
Florian

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772479#action_12772479 ]

Ivan Rozhnov commented on LUCENE-504:
-------------------------------------

Hi guys,

It seems to me that problem is still opened. Can it be fixed with dynamic size of storage in PriorityQueue and couple of similar classes and using MaxClauseCount as top limit for size of such storage. It seems to be very weird to have preinitialized array of Max size in collection constructor.

Thanks,
Ivan

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773001#action_12773001 ]

Michael McCandless commented on LUCENE-504:
-------------------------------------------

Since 3.0 is now on Java 1.5, can't we switch to Java's PriorityQueue?  Anyone want to cough up a patch?

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773098#action_12773098 ]

Uwe Schindler commented on LUCENE-504:
--------------------------------------

We had a discussion about that in another issue. In general PriorityQueue of Java 1.5 does not have the features we need for Java 1.5 (it dynamically grows, but the grow process is not controllable, making it unuseable for collecting TopDocs and so on). But I think for this special case, we could use Java 5's PQ.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773098#action_12773098 ]

Uwe Schindler edited comment on LUCENE-504 at 11/3/09 6:27 PM:
---------------------------------------------------------------

We had a discussion about that in another issue. In general PriorityQueue of Java 1.5 does not have the features we need for Lucene (it dynamically grows, but the grow process is not controllable, making it unuseable for collecting TopDocs and so on). But I think for this special case, we could use Java 5's PQ.

      was (Author: thetaphi):
    We had a discussion about that in another issue. In general PriorityQueue of Java 1.5 does not have the features we need for Java 1.5 (it dynamically grows, but the grow process is not controllable, making it unuseable for collecting TopDocs and so on). But I think for this special case, we could use Java 5's PQ.
 

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773402#action_12773402 ]

Nadav Har'El commented on LUCENE-504:
-------------------------------------

Hi Uwe, I think that even though PriorityQueue doesn't have a size limit, it is easy to implement a size limit: after an add(), if size() becomes greater than the bound, you simply poll() to remove the lowest element (this poll() returns the old object which insertWithOverflow() is to return).

However, I think it's a good idea to compare the performance of Java's PriorityQueue (used as in the paragraph above) . I'm especially worried about the slowdown by the fact that adding a small element (below the current heap's head) in our code just does one comparison and returns, but in the usage I described above it actually modifies the heap twice (adds the element to the heap and then removes it).

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773421#action_12773421 ]

Uwe Schindler commented on LUCENE-504:
--------------------------------------

Nadav:
I suggest to keep Lucene's PriorityQueue, because it is a very central and highly optimized part of Liucene. In Lucene 3.0 it is already generified, so it also fits perfectly into Java's Collection API. The only problem is that the name is now identical to one internal Java class, but we cannot change it without BW breaks.

For this special issue, we should fix *only* FuzzyQuery to use Java5's PQ, which dynamically grows when new elements are added. And we do not need the upper limit here, like you propsed.

I will prepare a patch tomorrow in the ApacheCon hacking session.

> FuzzyQuery produces a "java.lang.NegativeArraySizeException" in PriorityQueue.initialize if I use Integer.MAX_VALUE as BooleanQuery.MaxClauseCount
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-504
>                 URL: https://issues.apache.org/jira/browse/LUCENE-504
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 1.9
>            Reporter: Joerg Henss
>            Priority: Minor
>         Attachments: BooleanQuery.java.diff, fuzzyquery.patch, PriorityQueue.java.diff, TestFuzzyQueryError.java
>
>
> PriorityQueue creates an "java.lang.NegativeArraySizeException" when initialized with Integer.MAX_VALUE, because Integer overflows. I think this could be a general problem with PriorityQueue. The Error occured when I set BooleanQuery.MaxClauseCount to Integer.MAX_VALUE and user a FuzzyQuery for searching.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...