open issues for ns-3 testing

We last left off the testing discussion [1] prior to ns-3.5 release,
when we discussed the relative merits of running the tests from within
or outside of waf.  However, there are still several issues that need to
be addressed before we could move towards a merge of a new testing
framework.

I tried to collect and summarize below what I understand to be many of
the remaining issues, and maybe Craig and others can respond or add to
the below list, and propose their requirements or goals.

Here is some prototype code that Craig has been putting together (it is
in separate repos):
http://code.nsnam.org/craigdo/ns-3-test-patches/file/e6a504a6568e/test.patch
http://code.nsnam.org/craigdo/ns-3-valver/

To recap, we already have a unit test framework, and a regression
testing framework for trace-based comparison of our examples.  We don't
have a good place yet for validation and verification tests, we don't
have good examples to point to for how to write more complicated tests
(including stochastic tests), and further, we would like to not lean too
heavily on simply comparing pcap trace files for regression or
validation purposes, so we need a better way for generating, saving, and
comparing input and output test vectors that are more specific than
packet trace dumps.

Here are some lingering issues.

1) what is the overall plan for where this test code lives, and how does
it relate to existing test.h and test.cc?  Should existing src/test be
moved to src/core?

2) Why not just borrow one of the existing GPLed testing frameworks and
reuse it here?  Which ones have we looked at for comparison?

3) How are non-trivial output test vectors (they could be trace files,
long sequences of numbers or timestamps, etc.) going to be stored?

4) What is the story for waf integration, if test.py is retained?  In
particular, what kind of front end could waf provide for test.py script?
  In a previous post, I suggested that basic modes of test.py could
still be invoked by ./waf  --check, ./waf --test, ./waf --regression
(etc.). Are the issues of accessing the configuration cache and of
running programs through waf (or not) settled?

5) Related to this, what is the expected API and granularity for running
these tests?  Do we have different levels of testing that have simple
APIs (such as the above sequence of waf commands)?

6) Can we review some really simple examples (e.g. in samples/
directory) showing different examples of what different types of tests
would look like?  What about a TCP model verification example?

7) Where do more complicated test and validation programs (i.e. the
actual tests) reside?  src/test?

8) What about Mathieu's comment recently about having a mode that dumps
a lot of trace locally (without downloading saved regression traces) for
more extensive regression testing before a checkin?

9) What output do people want to see, typically (Gustavo had made a
comment about minimal output for successful tests and verbose outputs
for failed tests)?

10) Can we avoid baking in a GSL dependency in case we want to rewrite
histogram or chi-squared routines ourselves in the future?

11) Mathieu remarked early on that it would be nice to factor out the
chi-squared testing tools from within the random number tests, so they
could be used elsewhere.

12) How would lcov, nightly builds (buildbots, others) etc. be plumbed in?

- Tom

[1] http://mailman.isi.edu/pipermail/ns-developers/2009-May/005884.html