[jira] Created: (STDCXX-536) allow thread safety tests to time out without failing

allow thread safety tests to time out without failing
-----------------------------------------------------

                 Key: STDCXX-536
                 URL: https://issues.apache.org/jira/browse/STDCXX-536
             Project: C++ Standard Library
          Issue Type: Improvement
          Components: Tests
    Affects Versions: trunk
            Reporter: Martin Sebor


The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Sebor updated STDCXX-536:
--------------------------------

    Fix Version/s: 4.2

Tentatively scheduled for 4.2.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Sebor reassigned STDCXX-536:
-----------------------------------

    Assignee: Travis Vitek

Travis, can you take a look at this to see if we could implement it for 4.2 and avoid the test timeouts?

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Vitek updated STDCXX-536:
--------------------------------

    Attachment: stdcxx-536.patch

2007-09-10  Travis Vitek  <vitek@...>

        STDCXX-536
        rw_timeout.h: New file declares timeout functions used by
        tests.
        opt_timeout.h: New file declares timeout functions used by
        the test suite driver.
        opt_timeout.cpp: New file defines timeout functions used by
        the test suite driver.
        driver.cpp (rw_vtest): setup new command line options,
        enable and disable alarm for timeout.
        22.locale.messages.mt.cpp (thread_func): use timeout to
        let test threads exit early.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526315 ]

Travis Vitek commented on STDCXX-536:
-------------------------------------


I'm not convinced that this is actually a smart thing to do. Tests that would noramlly time out will now exit successfully. This is probably okay if there is only one part to the entire test, but that is not the case with any of the multithreaded tests that I've seen. If a test has three parts, and the first part times out, parts two and three won't execute, but it will appear that the test ran just fine.

That, and the new code doesn't provide anything for a test that doesn't poll the rw_timeout_expired() function. So this code is only really useful for the multithreaded tests. If that is the case then maybe the functionality thould be isolated to just those tests.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Vitek updated STDCXX-536:
--------------------------------

    Attachment:     (was: stdcxx-536.patch)

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Vitek updated STDCXX-536:
--------------------------------

    Attachment: stdcxx-536.patch

fixed patch to not use rw_note when timer expires because that function is not mt-safe.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Martin Sebor]

Hi Travis,

Can you describe how this all works? I.e., what happens to a test
invoked without the --timeout option when it exceeds the default
295ms timeout? What happens to one that is invoked with the option?

 From your comment below it sounds like the solution you implemented
is different from what I described in the issue, i.e., "set an alarm
in response to this command line option and in handler for the alarm
set a flag that each thread would check at each iteration of its
loop to see if it should break."

https://issues.apache.org/jira/browse/STDCXX-536#action_12526315

Thanks
Martin

Travis Vitek (JIRA) wrote:

>      [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Travis Vitek updated STDCXX-536:
> --------------------------------
>
>     Attachment: stdcxx-536.patch
>
> fixed patch to not use rw_note when timer expires because that function is not mt-safe.
>
>> allow thread safety tests to time out without failing
>> -----------------------------------------------------
>>
>>                 Key: STDCXX-536
>>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>>             Project: C++ Standard Library
>>          Issue Type: Improvement
>>          Components: Tests
>>    Affects Versions: trunk
>>            Reporter: Martin Sebor
>>            Assignee: Travis Vitek
>>             Fix For: 4.2
>>
>>         Attachments: stdcxx-536.patch
>>
>>
>> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.
>

[Travis Vitek-5]

Ah, maybe there was some miscommunication in our phone conversation on
Friday. When we were talking about this, I mentioned adding a little bit
of code to main for the command line argument stuff, and a little code
to the thread_func to check the flag in each multithreaded test. You
responded that as much of the common stuff should be pushed into the
driver as possible and that the only thing that should be changed in the
tests is the polling of the 'timeout' flag.

That is essentially what I've done here.

Whatever timeout period you specify to --timeout-period=<N> will be the
number of seconds [remember rw_alarm uses seconds not milliseconds] that
will elapse before the _rw_timeout_expired flag will be set to a
non-zero value. After all of the command line arguments have been
processed, the driver calls _rw_runopts_timeout() which will setup the
alarm. After N seconds have elapsed the flag will be set and the polling
threads should see that and exit early.

After stepping back and thinking about how it probably should work, I've
got some thoughts. I think that the changes to the driver should be
eliminated. If the user wants this timeout stuff in their multithreading
test, then they have to add a hook in the command line arg processing
stuff in main [just like they do for the --nthreads arg]. There would be
one command line argument --timeout that will set the timeout period.
From there I see a choice. Either the test can be expected to explicitly
start/reset the alarm timer [via rw_timeout_start/stop() maybe], or it
could be done implicitly within the rw_thread_pool() function. Either
way, before the threads are started the _rw_timeout_expired flag would
need to be cleared and the alarm would need to be set. After all threads
had exited, the alarm would need to be cancelled.

This way each section of the test would use the provided timeout. The
timeout could still have a default value of 30 seconds if we wanted the
tests to use this without needing to supply additional command line
parameters to them from the build system.

Travis

>Martin Sebor wrote:
>
>Hi Travis,
>
>Can you describe how this all works? I.e., what happens to a test
>invoked without the --timeout option when it exceeds the default
>295ms timeout? What happens to one that is invoked with the option?
>
> From your comment below it sounds like the solution you implemented
>is different from what I described in the issue, i.e., "set an alarm
>in response to this command line option and in handler for the alarm
>set a flag that each thread would check at each iteration of its
>loop to see if it should break."
>
>https://issues.apache.org/jira/browse/STDCXX-536#action_12526315
>
>Thanks
>Martin
>
>

[Martin Sebor]

Travis Vitek wrote:
> Ah, maybe there was some miscommunication in our phone conversation on
> Friday. When we were talking about this, I mentioned adding a little bit
> of code to main for the command line argument stuff, and a little code
> to the thread_func to check the flag in each multithreaded test. You
> responded that as much of the common stuff should be pushed into the
> driver as possible and that the only thing that should be changed in the
> tests is the polling of the 'timeout' flag.
>
> That is essentially what I've done here.

Okay, this makes sense. I thought the patch would set a default
timeout of 295ms for every test and once reached, force it to
exit with a zero exit status. That would not be good.

>
> Whatever timeout period you specify to --timeout-period=<N> will be the
> number of seconds [remember rw_alarm uses seconds not milliseconds] that
> will elapse before the _rw_timeout_expired flag will be set to a
> non-zero value. After all of the command line arguments have been
> processed, the driver calls _rw_runopts_timeout() which will setup the
> alarm. After N seconds have elapsed the flag will be set and the polling
> threads should see that and exit early.

Right. But tests whose threads that don't check the flag will
continue running and, presumably, be killed by the infrastructure.

>
> After stepping back and thinking about how it probably should work, I've
> got some thoughts. I think that the changes to the driver should be
> eliminated. If the user wants this timeout stuff in their multithreading
> test, then they have to add a hook in the command line arg processing
> stuff in main [just like they do for the --nthreads arg]. There would be
> one command line argument --timeout that will set the timeout period.

How about implementing something more along the lines of what we
do with rw_opt_setlocales()? That way the whole command line option
handler along with the logic to set the alarm is in the driver,
and the tests that want to make use of it only need to add
&rw_opt_settimeout to the rw_test() call in main()?

> From there I see a choice. Either the test can be expected to explicitly
> start/reset the alarm timer [via rw_timeout_start/stop() maybe], or it
> could be done implicitly within the rw_thread_pool() function.

I don't follow this. Why would the test want to reset the alarm?
Or do you mean set it in response to the command line option?

> Either
> way, before the threads are started the _rw_timeout_expired flag would
> need to be cleared and the alarm would need to be set. After all threads
> had exited, the alarm would need to be cancelled.
>
> This way each section of the test would use the provided timeout. The
> timeout could still have a default value of 30 seconds if we wanted the
> tests to use this without needing to supply additional command line
> parameters to them from the build system.

I see where you're going with this. You want the test to be able
to timeout, shut down all running threads when it does, and then
if there's more work to be done (i.e., another group of threads
to create), to reset the alarm and restart the whole process.
Right?

I guess I hadn't thought of the fact that some of these tests
create multiple thread pools. Hmm. I don't think the test should
be allowed to reset its own timeout but I'm not sure I see a good
solution. It certainly shouldn't exit successfully without doing
the rest of the work. Should it fail? Or should it just continue
with the next pool and allow itself to be killed?

Martin

>
> Travis
>
>
>> Martin Sebor wrote:
>>
>> Hi Travis,
>>
>> Can you describe how this all works? I.e., what happens to a test
>> invoked without the --timeout option when it exceeds the default
>> 295ms timeout? What happens to one that is invoked with the option?
>>
>> From your comment below it sounds like the solution you implemented
>> is different from what I described in the issue, i.e., "set an alarm
>> in response to this command line option and in handler for the alarm
>> set a flag that each thread would check at each iteration of its
>> loop to see if it should break."
>>
>> https://issues.apache.org/jira/browse/STDCXX-536#action_12526315
>>
>> Thanks
>> Martin
>>
>>

[Travis Vitek-5]

 

>Martin Sebor wrote
>
>Travis Vitek wrote:
>>
>>
>> Whatever timeout period you specify to --timeout-period=<N>
>> will be the number of seconds that will elapse before the
>> _rw_timeout_expired flag will be set to a non-zero value.
>> [...]
>
>Right. But tests whose threads that don't check the flag will
>continue running and, presumably, be killed by the infrastructure.
>

Exactly.

>
>> If the user wants this timeout stuff in their multithreading
>> test, then they have to add a hook in the command line arg
>> processing stuff in main [just like they do for the --nthreads
>> arg]. [...]
>
>How about implementing something more along the lines of what we
>do with rw_opt_setlocales()?
>

Sorry, I gave a poor example. That is exactly what I mean.

>
>> From there I see a choice. Either the test can be expected
>> to explicitly start/reset the alarm timer or it could be
>> done implicitly within the rw_thread_pool() function.
>
>I don't follow this. Why would the test want to reset the alarm?
>Or do you mean set it in response to the command line option?
>

Many of the multithreaded tests have several 'sections'. They create a
pool of threads for each section.

Take, for example, the test 22.locale.globals.mt.cpp. It has two
sections. The first creates a pool of threads to call the function
test_has_facet, and the other calls test_use_facet. Imagine that the
first section would run for 30 seconds if given the time, and there was
a 5 second timeout enabled. When the test is run, the first section
would run for 5 seconds and the second section of the test would not run
at all. That is no good. Even worse is the fact that the test would exit
with a successful return code, so there would be no indication that the
second part of the test has been skipped. I think that is a bad thing.

>> Either
>> way, before the threads are started the _rw_timeout_expired
>flag would
>> need to be cleared and the alarm would need to be set. After
>all threads
>> had exited, the alarm would need to be cancelled.
>>
>> This way each section of the test would use the provided timeout. The
>> timeout could still have a default value of 30 seconds if we
>wanted the
>> tests to use this without needing to supply additional command line
>> parameters to them from the build system.
>
>I see where you're going with this. You want the test to be able
>to timeout, shut down all running threads when it does, and then
>if there's more work to be done (i.e., another group of threads
>to create), to reset the alarm and restart the whole process.
>Right?

Essentially. I don't mean restart the whole process in the literal
sense. I mean it would go on to the next section of the test so at least
that section would be exercised before the process exited.

>
>I guess I hadn't thought of the fact that some of these tests
>create multiple thread pools. Hmm. I don't think the test should
>be allowed to reset its own timeout but I'm not sure I see a good
>solution.

Yeah, ugh.

>It certainly shouldn't exit successfully without doing
>the rest of the work.

Agreed.

>Should it fail?

Well, if we make the test fail aren't we essentially setting another
hard limit?

>Or should it just continue with the next pool and allow
>itself to be killed?
>

If we do that, then this timeout idea is done.

If the goal is to get the tests to run to completion sometime before the
hard limit there are a few obvious changes we could make. In some cases
we might be able to just reduce the default number of iterations. In
other cases, we could combine 'sections' of the test. Some tests have a
section to test a char specialization, then a wchar_t specialization
section, and finally a combined section that tests both. We could
potentially remove the first two sections, or at least disable them by
default. Another option would be to continue testing these sections, but
separate them out into their own tests.

Travis

[Martin Sebor]

Travis Vitek wrote:
[...]

>> I guess I hadn't thought of the fact that some of these tests
>> create multiple thread pools. Hmm. I don't think the test should
>> be allowed to reset its own timeout but I'm not sure I see a good
>> solution.
>
> Yeah, ugh.
>
>> It certainly shouldn't exit successfully without doing
>> the rest of the work.
>
> Agreed.
>
>> Should it fail?
>
> Well, if we make the test fail aren't we essentially setting another
> hard limit?

You're right, we would be for all the tests that create multiple
thread pools.

>
>> Or should it just continue with the next pool and allow
>> itself to be killed?
>>
>
> If we do that, then this timeout idea is done.
>
> If the goal is to get the tests to run to completion sometime before the
> hard limit there are a few obvious changes we could make. In some cases
> we might be able to just reduce the default number of iterations.

The problem is that we don't know apriori how many iterations
each test will be able to complete in the specified amount of
[wall clock] time because the number depends on the hardware,
and on the system load at the time the test runs, in addition
to the test itself.

> In
> other cases, we could combine 'sections' of the test. Some tests have a
> section to test a char specialization, then a wchar_t specialization
> section, and finally a combined section that tests both. We could
> potentially remove the first two sections, or at least disable them by
> default.

Yes, that's a possibility.

> Another option would be to continue testing these sections, but
> separate them out into their own tests.

Right. That's the solution I keep ending up with as the most robust
one. The main advantage I see is minimizing the scope of the tested
functionality. The biggest disadvantage is the significant overhead
of building and running twice as many (thread safety) tests as we
have now. We could eliminate some of the overhead due to building
these tests by building just one binary and rigging our test driver
to invoke it two or more times with different options (I like this
feature in and of itself a lot) but I fear there is little we can
do to deal with the overhead of running the test(s) multiple times.

Over to you.

Martin

[Travis Vitek-5]

Martin Sebor wrote:
>
>> Another option would be to continue testing these sections, but
>> separate them out into their own tests.
>
>Right. That's the solution I keep ending up with as the most robust
>one. The main advantage I see is minimizing the scope of the tested
>functionality. The biggest disadvantage is the significant overhead
>of building and running twice as many (thread safety) tests as we
>have now.

If we had one test to verify that method foo() was mt-safe, and another
test to verify bar(), we still need a test that foo() and bar() are
mt-safe when used together. So that's another test to have. And it
brings us back to just testing all of the tests into one again.
Duplicate code or file dependencies could be a problem also.

>We could eliminate some of the overhead due to building
>these tests by building just one binary and rigging our test driver
>to invoke it two or more times with different options (I like this
>feature in and of itself a lot) but I fear there is little we can
>do to deal with the overhead of running the test(s) multiple times.
>

Are you talking about modifying the nightly build infrastructure so that
it will invoke a test several times with different command line
arguments to run the test differently, or are you thinking about having
the test invoke itself with different arguments or something? I don't
think any option other than the first will work.

Travis

[Martin Sebor]

Travis Vitek wrote:
> Martin Sebor wrote:
[...]

>> We could eliminate some of the overhead due to building
>> these tests by building just one binary and rigging our test driver
>> to invoke it two or more times with different options (I like this
>> feature in and of itself a lot) but I fear there is little we can
>> do to deal with the overhead of running the test(s) multiple times.
>>
>
> Are you talking about modifying the nightly build infrastructure so that
> it will invoke a test several times with different command line
> arguments to run the test differently, or are you thinking about having
> the test invoke itself with different arguments or something? I don't
> think any option other than the first will work.

The former. I'm envisioning changing the exec utility (which runs
the tests) to look for some "command" file corresponding to each
test from which it would figure out how many times to run it and
with what options. If the file didn't exist exec would run the
test the same way it does now.

This approach could even be extended to compiling and linking
certain tests with special options, or to implement negative
testing, i.e., to exercise the ability of the library to reject
invalid programs. It would even let us deal with the increasing
disk space problem by compiling and linking each test on demand
just before running it and then immediately deleting it to free
up disk space.

What do you think about this?

Martin

    [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527564 ]

Travis Vitek commented on STDCXX-536:
-------------------------------------

Discussion on this issue can be found here
http://www.nabble.com/-jira--Created%3A-%28STDCXX-536%29-allow-thread-safety-tests-to-time-out-without-failing-tf4386314.html

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Vitek updated STDCXX-536:
--------------------------------

    Fix Version/s:     (was: 4.2)
                   4.2.1

Rescheduled for 4.2.1

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: trunk
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2.1
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Sebor updated STDCXX-536:
--------------------------------

      Severity: Usability
    Patch Info: [Patch Available]

Set Severity to Usability and checked Patch Available.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2.1
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555652#action_12555652 ]

Martin Sebor commented on STDCXX-536:
-------------------------------------

I'm not sure where we left this issue. The discussion kind of petered out... Do we want the patch or not?

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2.1
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/STDCXX-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555660#action_12555660 ]

Travis Vitek commented on STDCXX-536:
-------------------------------------

The patch is not ready for submission. I think we need to pick up discussion on this issue again.

> allow thread safety tests to time out without failing
> -----------------------------------------------------
>
>                 Key: STDCXX-536
>                 URL: https://issues.apache.org/jira/browse/STDCXX-536
>             Project: C++ Standard Library
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Travis Vitek
>             Fix For: 4.2.1
>
>         Attachments: stdcxx-536.patch
>
>
> The newly added thread safety tests (and possibly some of the existing ones) tend to run for a long time, consuming a lot of CPU cycles, and sometimes even failing due to a timeout (currently 300 seconds in nightly builds). It would be useful to provide a mechanism such as a command line option whereby the tests' runtime could be limited without necessarily causing them to fail when the amount of time is exceeded. One way to do it would be for each test to set an alarm in response to this command line option and in handler for the alarm set a flag that each thread would check at each iteration of its loop to see if it should break.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Travis Vitek]

Martin Sebor wrote:

Travis Vitek wrote:
>
> Are you talking about modifying the nightly build infrastructure so that
> it will invoke a test several times with different command line
> arguments to run the test differently, or are you thinking about having
> the test invoke itself with different arguments or something? I don't
> think any option other than the first will work.

The former. I'm envisioning changing the exec utility (which runs
the tests) to look for some "command" file corresponding to each
test from which it would figure out how many times to run it and
with what options. If the file didn't exist exec would run the
test the same way it does now.

This seems quite a bit more complicated than what was originally proposed. It involves changes to the test infrastructure in addition to modifications to the tests themselves, and probably the build result reporting code.

Martin Sebor wrote:

This approach could even be extended to compiling and linking
certain tests with special options, or to implement negative
testing, i.e., to exercise the ability of the library to reject
invalid programs.

I think that being able these types of testing might be useful. Unfortunately I see this as being a very complicated way to do what we claim we want.

Martin Sebor wrote:

It would even let us deal with the increasing
disk space problem by compiling and linking each test on demand
just before running it and then immediately deleting it to free
up disk space.

It seems that we could already do this with the existing makefile. The run_all rule would just have to be modified to build the executable before it is run and clean it up afterward. That doesn't seem like it would be terribly difficult.

I've gone back to read through the previous messages in this thread and I don't understand why we cannot just use a single timeout per rw_thread_pool() call. If a test has 3 sections (as most of them do), the timeout would apply to each section independently. We could set the default soft limit to 30 seconds or so, and then such a test would finish in approximately 90 seconds.

I think that is what you originally intended, but I got off track at some point and made everything all confusing.

Travis