Automated Unit tests for the Future

Hi everyone ,

I'm a Marble developer and have just started to think seriously about unit
tests and test driven development, possibly a bit late in the game but better
late then never! I was surprised to see that we already had some unit tests in
Marble but what was worse was that the core developers ( including myself )  
were "Surprised to see" that more than 50% of the tests failed. This is why
I've come to you lot, hopefully to spark some discussion and get some things
done ;)

The shocking thing about ^^^ is that the core developers were completely
unaware that the unit tests were failing, which i find a bit odd when there are
some pretty good Dashboard programs out there that can display and notify
about the results of unit tests. I would really like to see one of these
programs being used in a big way in KDE sooner rather than later so thats why
I spend most of yesterday chatting with someone that knows much more about
this stuff than I do. Sorry in advance if this turns out to be a very long
email.

Over the course of yesterday's discussions we have identified two major
"styles" of unit test dashboard system. 1) a distributed model using something
like CTest and CDash to display the results or 2) a centralised model where
the build system is "part" of the *dedicated* unit test running system and the
dashboard, like buildbot and CruiseControl . Each have their merits and
hopefully we can discuss which one would be "more right" for KDE.

For the centralised model we have a few organisational problems, mainly that
we would need a (dedicated ) server to be provided so that we could run the
build and display the results. This may not be so difficulty because I know that
"Dirk" has a system like this setup already where the build results are
displayed here: http://ktown.kde.org/~dirk/dashboard/ but this currently only
works for displaying the actual result of the build and does not include unit
tests. Using this model to incorporate unit tests shouldn't be too hard but it
might cause an organisational nightmare for the sysadmins ( who have a hard
enough time already ). On the other side of that, if we used a buildbot system
and some of the new cool buzzwords in computer science like "distributed
virtualised cloud computing" we could make a really cool system that would be
able to check the build of KDE on Linux, Windows and Mac. This would be pretty
cool but like i said an organisational nightmare.

The other possibility is that we could use the distributed unit test reporting
model that is currently incorporated by the CTest and CDash system. This is
favorable for a number of reasons:
1) we are currently using CMake so adding CTest support is *easy*!
2) we don't have to have a centralised build system because any time the unit
tests are run on any computer the results are submitted to the CDash dashboard
3) to set up the CDash system we would only need to add a few CMake variables
to our CMakeList.txt files and we will be submitting results to a database in
no time
4) from what I hear the kind people at http://www.cdash.org/CDash/index.php 
have already offered to host the CDash installation so our sysadmins would be
able to take it easy.
There are a lot of good points for using the CDash system but there is one
pretty big problem with it that may render our test results somewhat useless.
The fact that we are now starting to use the QtTest system makes things very
easy for us but it means that each QtTest executable will be regarded as a
single test by the CTest system. This is conceptually wrong because each
QtTest executable contains many sub tests that can fail or pass independently
of the single QtTest executable. Currently the CTest system only creates a
test result on the test executable level which means that the results may not
give any direct information as to why the test failed. Some may say that this
is only a small detail and results on the executable level are "good enough"
but if we are building a KDE-wide build system we should at least try and get
it as close to perfect as we possibly can, or at least move in that direction.
This problem could possibly be fixed with a patch to the CTest system but that
would require some effort by someone far smarter than myself ;)

Sorry again for the really long email, but i think that this really needs to
start getting discussed. I'm CCing this email to kde-devel and kde-quality so
that we can get as many people into the discussion as possible. I personally
believe that this discussion should be on the kde-buildsystem mailing list so
lets try and keep the discussions there ( feel free to correct me on this one
).

Happy coding!
-Andrew

--
Andrew Manson
Working for Marble Desktop Globe http://edu.kde.org/marble/
Implementing an OSM Annotation layer http://tinyurl.com/OSMMarble
Blog http://realate.blogspot.com
_______________________________________________
kde-quality mailing list
kde-quality@...
https://mail.kde.org/mailman/listinfo/kde-quality

Andrew Manson wrote:
> 4) from what I hear the kind people at
> http://www.cdash.org/CDash/index.php have already offered to host the

AFAIK they already did (http://www.kdedevelopers.org/node/3531) but last
time I checked it wasn't getting used. Now it seems to be gone - at least I
can't find it any longer.

Regards,
Michael

_______________________________________________
kde-quality mailing list
kde-quality@...
https://mail.kde.org/mailman/listinfo/kde-quality

[Alexander Neundorf]

Hi,

On Thursday 06 August 2009, Andrew Manson wrote:
> Hi everyone ,
>
> I'm a Marble developer and have just started to think seriously about unit
> tests and test driven development, possibly a bit late in the game but
> better late then never! I was surprised to see that we already had some
> unit tests in Marble but what was worse was that the core developers (
> including myself ) were "Surprised to see" that more than 50% of the tests
> failed. This is why I've come to you lot, hopefully to spark some
> discussion and get some things done ;)

Nice, I'd also like to get things going in this regard.

> The shocking thing about ^^^ is that the core developers were completely
> unaware that the unit tests were failing, which i find a bit odd when there
> are some pretty good Dashboard programs out there that can display and
> notify about the results of unit tests. I would really like to see one of
> these programs being used in a big way in KDE sooner rather than later so
> thats why I spend most of yesterday chatting with someone that knows much
> more about this stuff than I do. Sorry in advance if this turns out to be a
> very long email.
>
> Over the course of yesterday's discussions we have identified two major
> "styles" of unit test dashboard system. 1) a distributed model using
> something like CTest and CDash to display the results or 2) a centralised
> model where the build system is "part" of the *dedicated* unit test running
> system and the dashboard, like buildbot and CruiseControl . Each have their
> merits and hopefully we can discuss which one would be "more right" for
> KDE.
>
> For the centralised model we have a few organisational problems, mainly
> that we would need a (dedicated ) server to be provided so that we could
> run the build and display the results. This may not be so difficulty
> because I know that "Dirk" has a system like this setup already where the
> build results are displayed here: http://ktown.kde.org/~dirk/dashboard/ but
> this currently only works for displaying the actual result of the build and
> does not include unit tests. Using this model to incorporate unit tests
> shouldn't be too hard but it might cause an organisational nightmare for
> the sysadmins ( who have a hard enough time already ). On the other side of
> that, if we used a buildbot system and some of the new cool buzzwords in
> computer science like "distributed virtualised cloud computing" we could
> make a really cool system that would be able to check the build of KDE on
> Linux, Windows and Mac. This would be pretty cool but like i said an
> organisational nightmare.
>
> The other possibility is that we could use the distributed unit test
> reporting model that is currently incorporated by the CTest and CDash
> system. This is favorable for a number of reasons:
> 1) we are currently using CMake so adding CTest support is *easy*!
> 2) we don't have to have a centralised build system because any time the
> unit tests are run on any computer the results are submitted to the CDash
> dashboard

Nevertheless having some big central machine ideally running a bunch of
virtual machines, i.e. one with Windows, with OSX, FreeBSD and Solaris would
be nice :-)
I guess this would mean an Apple server...

> 3) to set up the CDash system we would only need to add a few
> CMake variables to our CMakeList.txt files and we will be submitting
> results to a database in no time
> 4) from what I hear the kind people at http://www.cdash.org/CDash/index.php
> have already offered to host the CDash installation so our sysadmins would
> be able to take it easy.

Yes, Kitware offers free hosting for free projects at my.cdash.org.
I have already set up dashboards for kdelibs, kdepimlibs, kdesupport and
automoc. I still need to document this somewhere, and I need help with
running the builds :-)

> There are a lot of good points for using the CDash system but there is one
> pretty big problem with it that may render our test results somewhat
> useless. The fact that we are now starting to use the QtTest system makes
> things very easy for us but it means that each QtTest executable will be
> regarded as a single test by the CTest system. This is conceptually wrong
> because each QtTest executable contains many sub tests that can fail or
> pass independently of the single QtTest executable. Currently the CTest
> system only creates a test result on the test executable level which means
> that the results may not give any direct information as to why the test
> failed. Some may say that this is only a small detail and results on the
> executable level are "good enough" but if we are building a KDE-wide build
> system we should at least try and get it as close to perfect as we possibly
> can, or at least move in that direction. This problem could possibly be
> fixed with a patch to the CTest system but that would require some effort
> by someone far smarter than myself ;)

I don't think it would be too hard.
And you are not the first one with this issue. Not using ctest/cdash because
of this one issue would be stupid, instead we (the people interested in
testing) should contribute and write a patch :-)
I can commit to the cmake (ctest) cvs, so this wouldn't be a big problem.
Not sure whether it would also require modifications in cdash. I don't have
commit access there, but Julien, the main developer, is usually very
responsive.

Alex

P.S. I'm offline now until Tuesday
_______________________________________________
kde-quality mailing list
kde-quality@...
https://mail.kde.org/mailman/listinfo/kde-quality

[Alexander Neundorf]

On Thursday 06 August 2009, Michael Leupold wrote:
> Andrew Manson wrote:
> > 4) from what I hear the kind people at
> > http://www.cdash.org/CDash/index.php have already offered to host the
>
> AFAIK they already did (http://www.kdedevelopers.org/node/3531) but last
> time I checked it wasn't getting used. Now it seems to be gone - at least I
> can't find it any longer.

It's at http://my.cdash.org now:
http://my.cdash.org/index.php?project=kdelibs

Alex
_______________________________________________
kde-quality mailing list
kde-quality@...
https://mail.kde.org/mailman/listinfo/kde-quality

On Thursday 06 August 2009 10:34:19 Andrew Manson wrote:
> Hi everyone ,
<snip>

Sounds great Andrew, looking forward to it reaching fruition. I think it would
definitely be good to have a system when people can sign up to a commit-digest
equivalent for monitoring build or test failures and if people who haven't
signed up can be emailed regardless to hopefully help people learn when they
break stuff for other platforms.

Nice work!
--
Cheers,
Mike Arthur
http://mikearthur.co.uk/
_______________________________________________
kde-quality mailing list
kde-quality@...
https://mail.kde.org/mailman/listinfo/kde-quality