JUnit 4 XML schematized?

[dfabulich]

There's a pretty standard format for JUnit XML output.  You've
probably seen it: there's a <testsuite> root element containing
zero-or-more <testcase> elements, each of which may contain a
<failure> or <error> element.  (There's also a <properties> element
with zero-or-more <property> elements.)  A lot of tools know how to
read this format and report on it, including at least Ant, Maven,
Cruise Control, Hudson, Bamboo, Eclipse and IntelliJ IDEA.

Is this XML standardized anywhere in a DTD or XML Schema or something?

If there isn't a standard, could we go about making a standard and
blessing it?  (Perhaps JUnit 4.5 could include an XMLReporter that
could be a reference implementation.)

In particular, I'm curious to know how one would represent that a test
has been ignored in "standard" JUnit XML.

-Dan

[Alexandru Popescu ☀]

On Nov 26, 2007 3:05 AM, dfabulich <dan@...> wrote:

>
> There's a pretty standard format for JUnit XML output. You've
>  probably seen it: there's a <testsuite> root element containing
>  zero-or-more <testcase> elements, each of which may contain a
>  <failure> or <error> element. (There's also a <properties> element
>  with zero-or-more <property> elements.) A lot of tools know how to
>  read this format and report on it, including at least Ant, Maven,
>  Cruise Control, Hudson, Bamboo, Eclipse and IntelliJ IDEA.
>
>  Is this XML standardized anywhere in a DTD or XML Schema or something?
>
>  If there isn't a standard, could we go about making a standard and
>  blessing it? (Perhaps JUnit 4.5 could include an XMLReporter that
>  could be a reference implementation.)
>
>  In particular, I'm curious to know how one would represent that a test
>  has been ignored in "standard" JUnit XML.
>

I never found such a "standard". And imo it is a good idea as you may
enhance the report by using your own information and still have the
tools working. But if you go with a DTD/schema and validation then
this will stop working.

./alex
--
.w( the_mindstorm )p.
 TestNG co-founder
EclipseTestNG Creator

>  -Dan
>
>  

[j_cumps]

--- In junit@..., "dfabulich" <dan@...> wrote:
>
> There's a pretty standard format for JUnit XML output.  You've
> probably seen it: there's a <testsuite> root element containing
> zero-or-more <testcase> elements, each of which may contain a
> <failure> or <error> element.  (There's also a <properties> element
> with zero-or-more <property> elements.)  A lot of tools know how to
> read this format and report on it, including at least Ant, Maven,
> Cruise Control, Hudson, Bamboo, Eclipse and IntelliJ IDEA.
>

I think the XML result format is owned (-in the open source meaning of
the word-) by Ant.
They started generating the XML result files in the JUnit task, and the
consolidated XML result file in the JUnitReport task.

>
> -Dan
>

Regards, Jan

[dfabulich]

"Alexandru Popescu ☀" <the.mindstorm.mailinglist@...> wrote:
> I never found such a "standard". And imo it is a good idea as you may
> enhance the report by using your own information and still have the
> tools working. But if you go with a DTD/schema and validation then
> this will stop working.

Well, I don't really want to make a strict validator.  All I want to
do is define semantics.

For example, when tests are marked with @Ignore, how should that be
represented in XML?  TestNG puts a <skipped /> element... is that what
everybody should do?  Where should the reason go?  What if it's a
thrown SkipException?

-Dan

[Alexandru Popescu ☀]

On Nov 27, 2007 1:28 AM, dfabulich <dan@...> wrote:

>
> "Alexandru Popescu ☀" <the.mindstorm.mailinglist@...> wrote:
>  > I never found such a "standard". And imo it is a good idea as you may
>  > enhance the report by using your own information and still have the
>  > tools working. But if you go with a DTD/schema and validation then
>  > this will stop working.
>
>  Well, I don't really want to make a strict validator. All I want to
>  do is define semantics.
>
>  For example, when tests are marked with @Ignore, how should that be
>  represented in XML? TestNG puts a <skipped /> element... is that what
>  everybody should do? Where should the reason go? What if it's a
>  thrown SkipException?
>

I think @Ignore supports as attribute a string. Now considering that
Java has native support for UTF-8 that "explanation/description"
string will not always make it to an XML attribute, so I am suggesting
a CDATA. This will also accomodate for the more advanced TestNG
SkipException.

For the rest it is just a matter on how to call the element: ignore,
skipped, etc.
I think I'm oke with either of the names, or if you have better ones
please speak up.

./alex
--
.w( the_mindstorm )p.
 TestNG co-founder
EclipseTestNG Creator

>  -Dan
>
>  

[James Abley]

On 27/11/2007, Alexandru Popescu ☀ <the.mindstorm.mailinglist@...> wrote:

>
> On Nov 27, 2007 1:28 AM, dfabulich <dan@...> wrote:
>  >
>  > "Alexandru Popescu ☀" <the.mindstorm.mailinglist@...> wrote:
>  >  > I never found such a "standard". And imo it is a good idea as you may
>  >  > enhance the report by using your own information and still have the
>  >  > tools working. But if you go with a DTD/schema and validation then
>  >  > this will stop working.
>  >
>  >  Well, I don't really want to make a strict validator. All I want to
>  >  do is define semantics.
>  >
>  >  For example, when tests are marked with @Ignore, how should that be
>  >  represented in XML? TestNG puts a <skipped /> element... is that what
>  >  everybody should do? Where should the reason go? What if it's a
>  >  thrown SkipException?
>  >
>
>  I think @Ignore supports as attribute a string. Now considering that
>  Java has native support for UTF-8 that "explanation/description"
>  string will not always make it to an XML attribute, so I am suggesting
>  a CDATA. This will also accomodate for the more advanced TestNG
>  SkipException.
>
>  For the rest it is just a matter on how to call the element: ignore,
>  skipped, etc.
>  I think I'm oke with either of the names, or if you have better ones
>  please speak up.
>
>  ./alex
>  --
>  .w( the_mindstorm )p.
>   TestNG co-founder
>  EclipseTestNG Creator
>
>  >  -Dan
>  >
>  >

Has anyone discussed this with the ANT devs? I think Stefan Bodewig
originally did the JUnitResultFormatter work, but IIRC Steve Loughran
may have been doing some work in there recently. Might be worth
pooling ideas.

Cheers,

James

[Alexandru Popescu ☀]

On Nov 28, 2007 12:53 PM, James Abley <james.abley@...> wrote:

>
> On 27/11/2007, Alexandru Popescu ☀ <the.mindstorm.mailinglist@...>
> wrote:
>  >
>  > On Nov 27, 2007 1:28 AM, dfabulich <dan@...> wrote:
>  > >
>  > > "Alexandru Popescu ☀" <the.mindstorm.mailinglist@...> wrote:
>  > > > I never found such a "standard". And imo it is a good idea as you may
>  > > > enhance the report by using your own information and still have the
>  > > > tools working. But if you go with a DTD/schema and validation then
>  > > > this will stop working.
>  > >
>  > > Well, I don't really want to make a strict validator. All I want to
>  > > do is define semantics.
>  > >
>  > > For example, when tests are marked with @Ignore, how should that be
>  > > represented in XML? TestNG puts a <skipped /> element... is that what
>  > > everybody should do? Where should the reason go? What if it's a
>  > > thrown SkipException?
>  > >
>  >
>  > I think @Ignore supports as attribute a string. Now considering that
>  > Java has native support for UTF-8 that "explanation/description"
>  > string will not always make it to an XML attribute, so I am suggesting
>  > a CDATA. This will also accomodate for the more advanced TestNG
>  > SkipException.
>  >
>  > For the rest it is just a matter on how to call the element: ignore,
>  > skipped, etc.
>  > I think I'm oke with either of the names, or if you have better ones
>  > please speak up.
>  >
>  > ./alex
>  > --
>  > .w( the_mindstorm )p.
>  > TestNG co-founder
>  > EclipseTestNG Creator
>  >
>  > > -Dan
>  > >
>  > >
>
>  Has anyone discussed this with the ANT devs? I think Stefan Bodewig
>  originally did the JUnitResultFormatter work, but IIRC Steve Loughran
>  may have been doing some work in there recently. Might be worth
>  pooling ideas.
>

Yes James, we are currently discussing this with Steve on the TestNG
mailing list. And as far as I heard the same topic is discussed also
on the Ant dev list.

./alex
--
.w( the_mindstorm )p.
 TestNG co-founder
EclipseTestNG Creator

>  Cheers,
>
>  James
>  

[dsaff]

On Nov 27, 2007 1:18 AM, Alexandru Popescu ☀
<the.mindstorm.mailinglist@...> wrote:
> I think @Ignore supports as attribute a string. Now considering that
> Java has native support for UTF-8 that "explanation/description"
> string will not always make it to an XML attribute, so I am suggesting
> a CDATA. This will also accomodate for the more advanced TestNG
> SkipException.

We're considering something for JUnit 4.x that may be similar to the
SkipException, although it's based on failed assumptions.  In early
returns, I'm seeing that I might want a quick summary of the
assumption that failed, with a more detailed description also
available.  However, this is still experimental.

That said, I am neither producer nor consumer of the XML syntax, so my
only interest is in whether anything we're currently doing makes the
XML generation harder than it feels like it needs to be (on which
point, the earlier discussion about Ignore reasons is appreciated).

   David Saff

[Stefan Bodewig-2]

On Wed, 28 Nov 2007, James Abley <james.abley@...> wrote:

> Has anyone discussed this with the ANT devs? I think Stefan Bodewig
> originally did the JUnitResultFormatter work,

True, more than seven years ago.  I must be getting old 8-)

> but IIRC Steve Loughran may have been doing some work in there
> recently.

We realized that the format has some problems, you can't stream it
easily because the number of failed tests is an attribute of the test
element, for example.  AntUnit uses a slightly modified version.

Steve has started a Wiki page collecting requirements of a modified
format[1], one of our goals is to use the same format for different
testing tools (unit and functional testing).

Stefan

Footnotes:
[1]  http://wiki.apache.org/ant/Proposals/EnhancedTestReports

[dfabulich]

David Saff wrote:
> That said, I am neither producer nor consumer of the XML syntax, so my
> only interest is in whether anything we're currently doing makes the
> XML generation harder than it feels like it needs to be (on which
> point, the earlier discussion about Ignore reasons is appreciated).

There's another similar issue I noticed just now, namely that it's
impossible to extract the parameters passed to a Parameterized test.

Instead, the index of the parameter array is packed into the test name
in brackets, like this: "testFoo[0]" "testFoo[1]" "testFoo[2]".  This
is true even if the parameters are strings.  There is no way to find
out what the strings are/were.

It would be better if the runner could add metadata (like parameters)
to the Description; the metadata could then be handled/interpreted by
RunListener reporters.

-Dan

[dsaff]

The Description should contain only those things about a test that are
available before the test is run.  I think we're closing in on a
request that additional information discovered during the test run can
be reported through the listener--this is already possible for failed
tests; it looks like we need ignore reasons for ignored tests, and
possibly parameters for parameterized tests.

   David Saff

On Nov 28, 2007 2:17 PM, dfabulich <dan@...> wrote:

> David Saff wrote:
> > That said, I am neither producer nor consumer of the XML syntax, so my
> > only interest is in whether anything we're currently doing makes the
> > XML generation harder than it feels like it needs to be (on which
> > point, the earlier discussion about Ignore reasons is appreciated).
>
> There's another similar issue I noticed just now, namely that it's
> impossible to extract the parameters passed to a Parameterized test.
>
> Instead, the index of the parameter array is packed into the test name
> in brackets, like this: "testFoo[0]" "testFoo[1]" "testFoo[2]".  This
> is true even if the parameters are strings.  There is no way to find
> out what the strings are/were.
>
> It would be better if the runner could add metadata (like parameters)
> to the Description; the metadata could then be handled/interpreted by
> RunListener reporters.
>
> -Dan
>
>
>
>
> Yahoo! Groups Links
>
>
>
>

[dfabulich]

"David Saff" wrote:
> The Description should contain only those things about a test that are
> available before the test is run.  I think we're closing in on a
> request that additional information discovered during the test run can
> be reported through the listener--this is already possible for failed
> tests; it looks like we need ignore reasons for ignored tests, and
> possibly parameters for parameterized tests.

Yes, that sounds basically right.  I'd only add that I think I'd want
runners to be able to specify arbitrary reportable facts, depending on
what they think is useful.

-Dan