[jira] Created: (LUCENE-1844) Speed up junit tests

Speed up junit tests
--------------------

                 Key: LUCENE-1844
                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
             Project: Lucene - Java
          Issue Type: Improvement
            Reporter: Mark Miller


As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.

There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.

Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.

I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1844:
--------------------------------

    Attachment: hi_junit_test_runtimes.png

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated LUCENE-1844:
--------------------------------

    Attachment: FastCnstScoreQTest.patch

TestCustomScoreQuery appears to be the longest test - its about a minute or worse.

The test can be set to log output, where it may display explain info - on many, many loops this explain info is calculated, only to be *not* logged because logging is off by default.

1. Excessive explain calculations turned off if logging is off.

In a logging call, QueryUtils.check(query, searcher) is called (overall ends up in a tight loop) - but it keeps getting called with the same query objects and the same searcher.

2. QueryUtils.check(query, searcher) moved out of loop and only tested once for each query.


Test from approx 60 seconds to approx 5 seconds.

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746605#action_12746605 ]

Mark Miller commented on LUCENE-1844:
-------------------------------------

Should also be able to speed up TestBooleanMinShouldMatch somehow. Its nearly a minute as well.

In a loop of 1000 random queries, this is called each time:

        QueryUtils.check(q1,s);
        QueryUtils.check(q2,s);

Take it out and the test is like 2-5 seconds. Must be some way to optimize this down without losing coverage.

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746605#action_12746605 ]

Mark Miller edited comment on LUCENE-1844 at 8/23/09 7:23 AM:
--------------------------------------------------------------

Should also be able to speed up TestBooleanMinShouldMatch somehow. Its nearly a minute as well (30s in attached list, but nearly a min on other hardware I have).

In a loop of 1000 random queries, this is called each time:

        QueryUtils.check(q1,s);
        QueryUtils.check(q2,s);

Take it out and the test is like 2-5 seconds. Must be some way to optimize this down without losing coverage.

      was (Author: markrmiller@...):
    Should also be able to speed up TestBooleanMinShouldMatch somehow. Its nearly a minute as well.

In a loop of 1000 random queries, this is called each time:

        QueryUtils.check(q1,s);
        QueryUtils.check(q2,s);

Take it out and the test is like 2-5 seconds. Must be some way to optimize this down without losing coverage.
 

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson updated LUCENE-1844:
-----------------------------------

    Attachment: LUCENE-1844.patch

moved/skipped calls to CheckUtils.check


In TestCustomScoreQuery, there was no purpose served that I could see by testing the same query over and over and over.

In TestBooleanMinShouldMatch, arbitrarily stopped calling check after 100 queries on the theory that we'd reached diminishing returns.

Drops test time by 3-4 minutes total...

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson updated LUCENE-1844:
-----------------------------------

    Attachment:     (was: LUCENE-1844.patch)

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson updated LUCENE-1844:
-----------------------------------

    Attachment: LUCENE-1844.patch

This supersedes the first patch I submitted. Apply after LUCENE-2037.

Render judgment on whether TestBooleanMinShouldMatch it's really OK to cut off checking the queries after 100.
 

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson updated LUCENE-1844:
-----------------------------------

    Attachment:     (was: LUCENE-1844.patch)

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

     [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson updated LUCENE-1844:
-----------------------------------

    Attachment: LUCENE-1844.patch

Saves 3-4 minutes overall. Arbitrarily limited the TestBooleanMinShouldMatch to stop checking queries after 100.

I don't see much point in checking the *same* queries again and again in TestCustomScoreQuery, so I just moved the check outside the loop.

Apply this patch *after* LUCENE-2037 since TestCustomScoreQuery happens to be common to both patches.

Sorry about the noise with the license grant...

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497 ]

Michael McCandless commented on LUCENE-1844:
--------------------------------------------

Is this ready to go in?  I'd really love to see unit tests run faster :)

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Erick Erickson]

They're ready to go, but at Uwe's suggestion, I've been waiting for 3.0 to get settled before prompting someone to apply this patch. I was going to generate a new patch for this and for 2037 (junit4 tests) just to make sure they were easy to apply. But if you're willing, the patches are already attached to the JIRA issues. Do note that the decision in MinBooleanShouldMatch to stop checking the query after 100 rather than checking all 1,000 is included in the patch....

Do you want to apply the patches or should I regenerate? It's no big deal to regenerate them and I'll have a better feel for reconciling any conflicts. I don't know whether there even *are* any conflicts, but just in case....

For my info, though, if I have a more recent patch that *replaces* an earlier patch, especially one that hasn't yet been applied, is it preferred to delete the earlier patch when providing a new one?

I'm not pleased with the Junit4 documentation, most of what I've been able to glean has come from brave souls blogging. Does anyone have a gold mine or is it as hit-or-miss as I think? There are hints of parallelization capabilities in Junit4, but I'm having a hard time finding anything in much depth..... The Junit website is pathetic, I can't even find 4.7 javadocs, it keeps giving me 4.5, as evidenced by no @Rule docs or @Intercept. And no version information in the javadocs..... Or I'm completely missing the boat.... I was thinking about getting the entire project over the weekend and generating my own if I have the time

Erick

On Wed, Nov 25, 2009 at 11:49 AM, Michael McCandless (JIRA) <jira@...> wrote:

   [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497 ]

Michael McCandless commented on LUCENE-1844:
--------------------------------------------

Is this ready to go in?  I'd really love to see unit tests run faster :)

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller

>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch

>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[markrmiller]

junit 4 parallelization is still in its infancy. I think the docs for it
are just in the changes file that it was first released with. That
version had severe bugs that made it almost unusable - I think thats
mostly fixed in a newer release. There is also a much better impl of one
of the key classes (I think they call it computer) written by someone
else that will eventually go into the code base I think (written by the
guy(s) that I think found/fixed the initial buggy-ness) - essentially, I
think its still unbaked.

Here are some docs from the release notes of 4.6:
http://sourceforge.net/project/shownotes.php?release_id=675664&group_id=15278

Thats also an issue - it arrived only in 4.6 - so this would need to be
optional unless we bumped up our req from 4 - and it really requires at
least 4.7 for the fixes (if everything is even fixed).

You also have to setup which tests run in parallel by hand essentially.
No ant task to help with this last I looked. So it will probably end up
being an alternate way to run the tests initially (at best).

- Mark

Erick Erickson wrote:

> They're ready to go, but at Uwe's suggestion, I've been waiting for
> 3.0 to get settled before prompting someone to apply this patch. I was
> going to generate a new patch for this and for 2037 (junit4 tests)
> just to make sure they were easy to apply. But if you're willing, the
> patches are already attached to the JIRA issues. Do note that the
> decision in MinBooleanShouldMatch to stop checking the query after 100
> rather than checking all 1,000 is included in the patch....
>
> Do you want to apply the patches or should I regenerate? It's no big
> deal to regenerate them and I'll have a better feel for reconciling
> any conflicts. I don't know whether there even *are* any conflicts,
> but just in case....
>
> For my info, though, if I have a more recent patch that *replaces* an
> earlier patch, especially one that hasn't yet been applied, is it
> preferred to delete the earlier patch when providing a new one?
>
> I'm not pleased with the Junit4 documentation, most of what I've been
> able to glean has come from brave souls blogging. Does anyone have a
> gold mine or is it as hit-or-miss as I think? There are hints of
> parallelization capabilities in Junit4, but I'm having a hard time
> finding anything in much depth..... The Junit website is pathetic, I
> can't even find 4.7 javadocs, it keeps giving me 4.5, as evidenced by
> no @Rule docs or @Intercept. And no version information in the
> javadocs..... Or I'm completely missing the boat.... I was thinking
> about getting the entire project over the weekend and generating my
> own if I have the time
>
> Erick
>
> On Wed, Nov 25, 2009 at 11:49 AM, Michael McCandless (JIRA)
> <jira@... <mailto:jira@...>> wrote:
>
>
>        [
>     https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497
>     <https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497>
>     ]
>
>     Michael McCandless commented on LUCENE-1844:
>     --------------------------------------------
>
>     Is this ready to go in?  I'd really love to see unit tests run
>     faster :)
>
>     > Speed up junit tests
>     > --------------------
>     >
>     >                 Key: LUCENE-1844
>     >                 URL:
>     https://issues.apache.org/jira/browse/LUCENE-1844
>     >             Project: Lucene - Java
>     >          Issue Type: Improvement
>     >            Reporter: Mark Miller
>     >         Attachments: FastCnstScoreQTest.patch,
>     hi_junit_test_runtimes.png, LUCENE-1844.patch
>     >
>     >
>     > As Lucene grows, so does the number of JUnit tests. This is
>     obviously a good thing, but it comes with longer and longer test
>     times. Now that we also run back compat tests in a standard test
>     run, this problem is essentially doubled.
>     > There are some ways this may get better, including running
>     parallel tests. You will need the hardware to fully take
>     advantage, but it should be a nice gain. There is already an issue
>     for this, and Junit 4.6, 4.7 have the beginnings of something we
>     might be able to count on soon. 4.6 was buggy, and 4.7 still
>     doesn't come with nice ant integration. Parallel tests will come
>     though.
>     > Beyond parallel testing, I think we also need to concentrate on
>     keeping our tests lean. We don't want to sacrifice coverage or
>     quality, but I'm sure there is plenty of fat to skim.
>     > I've started making a list of some of the longer tests - I think
>     with some work we can make our tests much faster - and then with
>     parallelization, I think we could see some really great gains.
>
>     --
>     This message is automatically generated by JIRA.
>     -
>     You can reply to this email to add a comment to the issue online.
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: java-dev-unsubscribe@...
>     <mailto:java-dev-unsubscribe@...>
>     For additional commands, e-mail: java-dev-help@...
>     <mailto:java-dev-help@...>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Erick Erickson]

Hmmm, the patches that I supplied for Junit4 *require* 4.7 anyway, which

I included in the patch... Is this a problem? Or just a document problem?

Erick

On Wed, Nov 25, 2009 at 1:14 PM, Mark Miller <markrmiller@...> wrote:

junit 4 parallelization is still in its infancy. I think the docs for it
are just in the changes file that it was first released with. That
version had severe bugs that made it almost unusable - I think thats
mostly fixed in a newer release. There is also a much better impl of one
of the key classes (I think they call it computer) written by someone
else that will eventually go into the code base I think (written by the
guy(s) that I think found/fixed the initial buggy-ness) - essentially, I
think its still unbaked.

Here are some docs from the release notes of 4.6:
http://sourceforge.net/project/shownotes.php?release_id=675664&group_id=15278

Thats also an issue - it arrived only in 4.6 - so this would need to be
optional unless we bumped up our req from 4 - and it really requires at
least 4.7 for the fixes (if everything is even fixed).

You also have to setup which tests run in parallel by hand essentially.
No ant task to help with this last I looked. So it will probably end up
being an alternate way to run the tests initially (at best).

- Mark

Erick Erickson wrote:
> They're ready to go, but at Uwe's suggestion, I've been waiting for
> 3.0 to get settled before prompting someone to apply this patch. I was
> going to generate a new patch for this and for 2037 (junit4 tests)
> just to make sure they were easy to apply. But if you're willing, the
> patches are already attached to the JIRA issues. Do note that the
> decision in MinBooleanShouldMatch to stop checking the query after 100
> rather than checking all 1,000 is included in the patch....
>
> Do you want to apply the patches or should I regenerate? It's no big
> deal to regenerate them and I'll have a better feel for reconciling
> any conflicts. I don't know whether there even *are* any conflicts,
> but just in case....
>
> For my info, though, if I have a more recent patch that *replaces* an
> earlier patch, especially one that hasn't yet been applied, is it
> preferred to delete the earlier patch when providing a new one?
>
> I'm not pleased with the Junit4 documentation, most of what I've been
> able to glean has come from brave souls blogging. Does anyone have a
> gold mine or is it as hit-or-miss as I think? There are hints of
> parallelization capabilities in Junit4, but I'm having a hard time
> finding anything in much depth..... The Junit website is pathetic, I
> can't even find 4.7 javadocs, it keeps giving me 4.5, as evidenced by
> no @Rule docs or @Intercept. And no version information in the
> javadocs..... Or I'm completely missing the boat.... I was thinking
> about getting the entire project over the weekend and generating my
> own if I have the time
>
> Erick
>
> On Wed, Nov 25, 2009 at 11:49 AM, Michael McCandless (JIRA)

> <jira@... <mailto:jira@...>> wrote:
>
>
>        [
>     https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497
>     <https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782497#action_12782497>
>     ]
>
>     Michael McCandless commented on LUCENE-1844:
>     --------------------------------------------
>
>     Is this ready to go in?  I'd really love to see unit tests run
>     faster :)
>
>     > Speed up junit tests
>     > --------------------
>     >
>     >                 Key: LUCENE-1844
>     >                 URL:
>     https://issues.apache.org/jira/browse/LUCENE-1844
>     >             Project: Lucene - Java
>     >          Issue Type: Improvement
>     >            Reporter: Mark Miller
>     >         Attachments: FastCnstScoreQTest.patch,
>     hi_junit_test_runtimes.png, LUCENE-1844.patch
>     >
>     >
>     > As Lucene grows, so does the number of JUnit tests. This is
>     obviously a good thing, but it comes with longer and longer test
>     times. Now that we also run back compat tests in a standard test
>     run, this problem is essentially doubled.
>     > There are some ways this may get better, including running
>     parallel tests. You will need the hardware to fully take
>     advantage, but it should be a nice gain. There is already an issue
>     for this, and Junit 4.6, 4.7 have the beginnings of something we
>     might be able to count on soon. 4.6 was buggy, and 4.7 still
>     doesn't come with nice ant integration. Parallel tests will come
>     though.
>     > Beyond parallel testing, I think we also need to concentrate on
>     keeping our tests lean. We don't want to sacrifice coverage or
>     quality, but I'm sure there is plenty of fat to skim.
>     > I've started making a list of some of the longer tests - I think
>     with some work we can make our tests much faster - and then with
>     parallelization, I think we could see some really great gains.
>
>     --
>     This message is automatically generated by JIRA.
>     -
>     You can reply to this email to add a comment to the issue online.
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: java-dev-unsubscribe@...

>     <mailto:java-dev-unsubscribe@...>

>     For additional commands, e-mail: java-dev-help@...

>     <mailto:java-dev-help@...>

>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Michael McCandless-2]

On Wed, Nov 25, 2009 at 1:01 PM, Erick Erickson <erickerickson@...> wrote:
> They're ready to go, but at Uwe's suggestion, I've been waiting for 3.0 to
> get settled before prompting someone to apply this patch. I was going to
> generate a new patch for this and for 2037 (junit4 tests) just to make sure
> they were easy to apply. But if you're willing, the patches are already
> attached to the JIRA issues. Do note that the decision in
> MinBooleanShouldMatch to stop checking the query after 100 rather than
> checking all 1,000 is included in the patch....

Uwe do you still think we should wait?  3.0.0 looks like it's "out"?

> Do you want to apply the patches or should I regenerate? It's no big deal to
> regenerate them and I'll have a better feel for reconciling any conflicts. I
> don't know whether there even *are* any conflicts, but just in case....

Let's hold up until we hear from Uwe...

> For my info, though, if I have a more recent patch that *replaces* an
> earlier patch, especially one that hasn't yet been applied, is it preferred
> to delete the earlier patch when providing a new one?

Actually in general I prefer people not delete the old patches...
This way if there's a question/confusion we can always go back &
compare old to new.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Uwe Schindler]

It's out and already on most of the mirrors...

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@...

> -----Original Message-----
> From: Michael McCandless [mailto:lucene@...]
> Sent: Wednesday, November 25, 2009 8:31 PM
> To: java-dev@...
> Subject: Re: [jira] Commented: (LUCENE-1844) Speed up junit tests
>
> On Wed, Nov 25, 2009 at 1:01 PM, Erick Erickson <erickerickson@...>
> wrote:
> > They're ready to go, but at Uwe's suggestion, I've been waiting for 3.0
> to
> > get settled before prompting someone to apply this patch. I was going to
> > generate a new patch for this and for 2037 (junit4 tests) just to make
> sure
> > they were easy to apply. But if you're willing, the patches are already
> > attached to the JIRA issues. Do note that the decision in
> > MinBooleanShouldMatch to stop checking the query after 100 rather than
> > checking all 1,000 is included in the patch....
>
> Uwe do you still think we should wait?  3.0.0 looks like it's "out"?
>
> > Do you want to apply the patches or should I regenerate? It's no big
> deal to
> > regenerate them and I'll have a better feel for reconciling any
> conflicts. I
> > don't know whether there even *are* any conflicts, but just in case....
>
> Let's hold up until we hear from Uwe...
>
> > For my info, though, if I have a more recent patch that *replaces* an
> > earlier patch, especially one that hasn't yet been applied, is it
> preferred
> > to delete the earlier patch when providing a new one?
>
> Actually in general I prefer people not delete the old patches...
> This way if there's a question/confusion we can always go back &
> compare old to new.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@...
> For additional commands, e-mail: java-dev-help@...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Uwe Schindler]

> junit 4 parallelization is still in its infancy. I think the docs for it
> are just in the changes file that it was first released with. That
> version had severe bugs that made it almost unusable - I think thats
> mostly fixed in a newer release. There is also a much better impl of one
> of the key classes (I think they call it computer) written by someone
> else that will eventually go into the code base I think (written by the
> guy(s) that I think found/fixed the initial buggy-ness) - essentially, I
> think its still unbaked.

There is another problem. Parallelization would only work with tests, that
do not change gloabl defaults. E.g. LocalizedTestCase changes the default
locale. If another test would run in Paralale, it would break.

Son only isolated tests can run in parallel. This LocalizedTestCase cannot
solved in another way. The same would have been in 2.9 with the
TokenStream.useOnlyNewAPI switch, but this is now longer the case for 3.1.

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

    [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716 ]

Michael McCandless commented on LUCENE-1844:
--------------------------------------------

Will we also speed up back-compat tests?

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[Erick Erickson]

I posted a rather long diatribe outlining why I think speed-ups
are a false goal for Lucene. Briefly, I'm convinced that as long
as the tests are run when Hudson builds Lucene, 99% of the
value of unit tests is realized. I suppose this implies that the
hard-core committers agree that as long as failed tests
are caught/corrected within a day things are fine.

Although coming from a background where unit
tests are not always required, my viewpoint may be
suspect <G>.

Erick@.......

On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA) <jira@...> wrote:

   [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716 ]

Michael McCandless commented on LUCENE-1844:
--------------------------------------------

Will we also speed up back-compat tests?

> Speed up junit tests
> --------------------
>
>                 Key: LUCENE-1844
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1844
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Mark Miller
>         Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>
>
> As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
> There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
> Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
> I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------

To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...

[markrmiller]

On Nov 25, 2009, at 9:32 PM, Erick Erickson <erickerickson@...>  
wrote:
>  I suppose this implies that the
> hard-core committers agree that as long as failed tests
> are caught/corrected within a day things are fine.

Personally, I don't agree with that. Others are checking out,  
updating, and using the tests to help verify and figure out their  
changes. Breaks in the build are detrimental to this processes. If it  
happens on the rare occassion fine, but you should run the the tests  
before committing - it's being polite to others working with the  
build. Don't break the build is a rule I'm strongly behind - and it  
requires running the tests outside of Hudson.

-mark

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@...
For additional commands, e-mail: java-dev-help@...