Submitting a run in multiple steps

[Jeff Squyres (jsquyres)]

We're looking at using perfbase for a variety of long-running test
suites, each of which are comprised of hundreds of little steps
(specifically: hundreds of separate executables), each of which have
individual outputs.  Most of them are MPI correctness tests (e.g., the
Intel MPI test suite), and they conveniently allow us to treat all the
outputs identically (i.e., they passed or they failed).

Running the entire Intel test suite, depending on your configuration,
can take many hours.  Additionally, our testing engine may actually
interlace multiple runs of the Intel test suite (e.g., run test A in
configurations 1, 2, 3, and 4, then proceed to test B in configurations
in 1, 2, 3, and 4, ...).  So the final results may not be available for
(N * many_hours).

We would greatly prefer to not have to wait for all the tests to finish
before we can see any of the results.  Specifically, we'd like to be
able to see partial results -- perhaps not in real-time, but at some
frequency that is considerably faster than "(N * many_hours)".

One way to do this would be to define a perfbase experiment to be the
result of a single test from the Intel test suite (for example).  We
could then submit to perfbase after each individual test executable (or
be a bit more efficient and submit "a bunch" of them at once).  But this
will lead to a table explosion in the back-end database -- in the worst
case where we submit one result at a time, each result will be a
perfbase run, and therefore one table for will be created for each
result.  This will likely be so inefficient as to be prohibitive.

What we really want to be able to do is have a lot of test results
submitted incrementally over time but not lead to a table explosion.
Perhaps something like this scenario:

me: "hey perfbase, please create an empty run for me and give me back an
identifier for it"
perfbase: "sure thing -- you now have run X"
<time passes>
me: "hey perfbase, here's a bunch of results, please put them in run X"
<more time passes>
me: "hey perfbase, here's a bunch of results, please put them in run X"

...and so on.

Is something like this possible?  Or, if not, do you have any
suggestions about how we can submit partial results to perfbase without
being horribly inefficient in the back-end database?

Thanks!

--
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...