Inspect/implementation defined order fn-trace-*

[Frans Englich-2]

Hi,

Many of the fn:trace tests are of 'Inspect' type, and I don't get why. Here's
an example fn-trace21:

(: Name: fn-trace-3 :)
(: Description: Simple call of "fn:trace" function used with an addition
operation. :)

for $var in (1,2,3,4,5)
return fn:trace($var + 1,"The Value of $var + 1 is: ")

Baseline(inspect):
2 3 4 5 6

This suggests that what fn:trace returns, is in implementation dependent
order.  That is: fn:trace((1, 2), "msg") can return either (1, 2) or (2, 1).
I have a hard time backing this up from the spec. It says:

<quote>
The destination of the trace output is ·implementation-defined·. The format of
the trace output is ·implementation dependent·. The ordering of output from
invocations of the fn:trace() function is ·implementation dependent·.
</quote>

So, if the return value is supposed to be in implementation defined order,
that must be spec'd by "The ordering of output from invocations of the
fn:trace() function is ·implementation dependent".

Then look at the example:

<quote>
Writing fn:trace($v, 'the value of $v is:') will put the strings "124.84" and
"the value of $v is:" in the trace data set in implementation dependent
order.
</quote>

Here it says the trace data will be in implementation dependent order, but the
first paragraph only says the format and the destination is implementation
dependent.

So, as I see it, there's two interpretations on what "implementation dependent
order" applies to: the example says it's the trace data, the XQTS WG's
interpretation says it's the return value.

I think F&O's choice of "output" is vague and that it could use editorial
clarification. Any thoughts on this? I'll submt a report on F&O unless
something unexpected pops up.


Regards,

                Frans

[Carmelo Montanez]

Frans:

Thanks of the comment on fn:trace tests.  I will ask the task force to
add this to our telecon this week.

Thanks,
Carmelo

On Monday 24 April 2006 17:34, Carmelo Montanez wrote:
> Frans:
>
> Thanks of the comment on fn:trace tests.  I will ask the task force to
> add this to our telecon this week.

In case you've missed it, I opened a report on it such that it wouldn't get
lost:

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3139


Cheers,

                Frans

[David Carlisle]

Frans,

I think (as a non WG member) that the reason these are Inspect is that
the point of trace() is that you need to _inspect_ the trace output
(which has gone to some system specific place).

The actual result returned by the query isn't system dependent, but
it's for you to look at the log generated by the trace() function and if
it's empty or if it just says "system error: couldn't write to log file"
or something else other than the expected trace, it's for you to decide
whether that constitutes a pass or fail.

Having said that, I think that this interpretation is really justified
by the Catalogue markup which does, as you say, imply that one should
manually compare the result output with the supplied result file.

The Catalogue format should probably be extended to allow the Catalogue
to specify both the Query result (checked by XML comparisopn) and the
trace log (checked by inspection).

David

________________________________________________________________________
This e-mail has been scanned for all viruses by Star. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk
________________________________________________________________________

On Tuesday 25 April 2006 10:11, David Carlisle wrote:
> Frans,

David,

> I think (as a non WG member) that the reason these are Inspect is that
> the point of trace() is that you need to _inspect_ the trace output
> (which has gone to some system specific place).
> The actual result returned by the query isn't system dependent, but
> it's for you to look at the log generated by the trace() function and if
> it's empty or if it just says "system error: couldn't write to log file"
> or something else other than the expected trace, it's for you to decide
> whether that constitutes a pass or fail.
>
> Having said that, I think that this interpretation is really justified
> by the Catalogue markup which does, as you say, imply that one should
> manually compare the result output with the supplied result file.
>
> The Catalogue format should probably be extended to allow the Catalogue
> to specify both the Query result (checked by XML comparisopn) and the
> trace log (checked by inspection).

I don't see what the XQTS can do in order to verify interoperability on the
trace. The destination is ·implementation-defined·, the format is
·implementation dependent·, the ordering between multiple invokations is
·implementation dependent·, and not the least that the data "may be directed
to a trace data set". I see nothing which an implementation is required to
do, not even to report anything. As I see it, there is nothing to verify
since an implementation can do anything it chooses to.

So, let's say the tests are of type 'Inspect' in order to somehow verify/test
the trace output, it currently happens at the cost of that the data output is
not fully verified(well, it depends on inspection).

So, the alternatives are to either extend the catalog as you describe, or to
skip anykind of validation of the trace output. I think the latter is the
right one, because trace is implementation defined.


Cheers,

                Frans

[David Carlisle]

Just to be clear, I said

me> Having said that, I think that this interpretation is really justified
me> by the Catalogue markup

But of course meant

   Having said that, I think that this interpretation is NOT really justified
                                                         ^^^
   by the Catalogue markup



> As I see it, there is nothing to verify
> since an implementation can do anything it chooses to.

I agree that from a conformance point of view trace() can be a no-op so
that the only thing that should be tested as part of a conformance or
interoperability test is the query result (which can be mechanically
tested with xml comparison). However officially the W3C doesn't go in
for conformance testing of particular systems and the purpose of the
suite as far as getting out of CR is concerned is to show that the
features are implementable, and as a general test of "have I implemented
all features" I don't mind having a few tests that make some calls on
trace() and I check by hand that they end up in the standard error
stream (or wherever I think they should go).


> So, let's say the tests are of type 'Inspect' in order to somehow verify/test
> the trace output, it currently happens at the cost of that the data output is
> not fully verified (well, it depends on inspection).

Yes I agree, it would be better to use XML comparison on the expected
output and have another element <expected-trace> or something which you
could additionally verify by hand, or something....


David

________________________________________________________________________
This e-mail has been scanned for all viruses by Star. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk
________________________________________________________________________