« Return to Thread: open issues for ns-3 testing
Re: open issues for ns-3 testing
by Pavel Boyko
::
Rate this Message:
Hi,
On Thursday 09 July 2009 09:15:06 am Tom Henderson wrote:
> I tried to collect and summarize below what I understand to be many of
> the remaining issues, and maybe Craig and others can respond or add to
> the below list, and propose their requirements or goals.
Let me add few words.
First of all, I believe that the model validation is a research activity
which can't be automated or restricted to use some build-in "Validation
toolkit". IMHO the best thing we can do is to encourage a companion "The NS-3
Foo Model Validation" paper(s) for every foo module. The recent 802.11b PHY
model is an excellent example of this, see
http://www.nsnam.org/~pei/80211b.pdf M.b. we should create and maintain a wiki
page listing all available models together with references to their validation
reports (see e.g. http://www.scalable-networks.com/pdf/QualNet_Library.pdf for
a nice table of all available models). When model is validated, usual
regression tests (both unit and functional) can be used to assert that it is
not broken. To summarize -- I propose do not create any kind of validation
toolkit, rather than encourage and collect references to the validation
efforts.
> we need a better way for generating, saving, and
> comparing input and output test vectors that are more specific than
> packet trace dumps.
Agree.
> 2) Why not just borrow one of the existing GPLed testing frameworks and
> reuse it here? Which ones have we looked at for comparison?
Existing test.h + waf --check unit test facility just fits my needs and is
not worse than boost::test + cmake/ctest .
> 7) Where do more complicated test and validation programs (i.e. the
> actual tests) reside? src/test?
Why not?
> 9) What output do people want to see, typically (Gustavo had made a
> comment about minimal output for successful tests and verbose outputs
> for failed tests)?
Personally I'd like to see (finally) usual green-red html table like
http://www.cdash.org/CDash/index.php?project=Slicer3#Coverage or
http://buildbot.net/trac/wiki/ScreenShots with optional cpu time / memory
usage / stdout / stderr available by click.
> 10) Can we avoid baking in a GSL dependency in case we want to rewrite
> histogram or chi-squared routines ourselves in the future?
Why not interface with dedicated statistical systems like R
(http://www.rproject.org/) for data analysis? I don't believe that we really
need to create one more homegrown histogram implementation, to say nothing
about hypothesis testing.
Regards,
Pavel Boyko, IITP
On Thursday 09 July 2009 09:15:06 am Tom Henderson wrote:
> I tried to collect and summarize below what I understand to be many of
> the remaining issues, and maybe Craig and others can respond or add to
> the below list, and propose their requirements or goals.
Let me add few words.
First of all, I believe that the model validation is a research activity
which can't be automated or restricted to use some build-in "Validation
toolkit". IMHO the best thing we can do is to encourage a companion "The NS-3
Foo Model Validation" paper(s) for every foo module. The recent 802.11b PHY
model is an excellent example of this, see
http://www.nsnam.org/~pei/80211b.pdf M.b. we should create and maintain a wiki
page listing all available models together with references to their validation
reports (see e.g. http://www.scalable-networks.com/pdf/QualNet_Library.pdf for
a nice table of all available models). When model is validated, usual
regression tests (both unit and functional) can be used to assert that it is
not broken. To summarize -- I propose do not create any kind of validation
toolkit, rather than encourage and collect references to the validation
efforts.
> we need a better way for generating, saving, and
> comparing input and output test vectors that are more specific than
> packet trace dumps.
Agree.
> 2) Why not just borrow one of the existing GPLed testing frameworks and
> reuse it here? Which ones have we looked at for comparison?
Existing test.h + waf --check unit test facility just fits my needs and is
not worse than boost::test + cmake/ctest .
> 7) Where do more complicated test and validation programs (i.e. the
> actual tests) reside? src/test?
Why not?
> 9) What output do people want to see, typically (Gustavo had made a
> comment about minimal output for successful tests and verbose outputs
> for failed tests)?
Personally I'd like to see (finally) usual green-red html table like
http://www.cdash.org/CDash/index.php?project=Slicer3#Coverage or
http://buildbot.net/trac/wiki/ScreenShots with optional cpu time / memory
usage / stdout / stderr available by click.
> 10) Can we avoid baking in a GSL dependency in case we want to rewrite
> histogram or chi-squared routines ourselves in the future?
Why not interface with dedicated statistical systems like R
(http://www.rproject.org/) for data analysis? I don't believe that we really
need to create one more homegrown histogram implementation, to say nothing
about hypothesis testing.
Regards,
Pavel Boyko, IITP
« Return to Thread: open issues for ns-3 testing
| Free embeddable forum powered by Nabble | Forum Help |
