Nabble - w3.org - public-rdf-dawg-comments

Re: Comments on SPARQL 1.1

2009-12-15T02:15:55Z

Rob,

Thank you for your comments. Responses to your queries are inline below.

> Hi all
>
> Here are my comments/questions on the draft. Some of this is based
on
> personal opinion and some of this is based on starting to try and
implement
> some of this stuff in my SPARQL implementation in my latest
development
> builds of dotNetRDF (early Alpha release .Net RDF library).
>
> Aggregates
> - I definitely support the use of a separate HAVING keyword, I
agree with
> Leigh that it makes clear that the type of constraint is different
and helps
> SQL programmers make the move to SPARQL by making the syntax familiar

The WG has not yet made a decision on which syntax to use, but the
editors are currently considering the options.

> - The proposed grammar has a couple of things which bug me:
>
> 1. It allows for a Having clause without a Group By clause which
doesn't
> necessarily make sense depending on the constraint given. I assume
that if
> you have a Having clause without a Group By then it should act as a
Filter
> on the result set?
> The only reason I can see for allowing this is if the having
clause uses
> an aggregate then this makes sense in some circumstances e.g.
seeing whether
> there are a given number of Triples matching a pattern:
>
> SELECT * WHERE {?s a ?type} HAVING (COUNT(?s) > 10)

Currently the proposal is that by default a binding multiset will
consist of 1 groups containing all solutions. This is compatible with
the way SQL works. However this design is not final, and may well
change in the near future.

> 2. What is the intended meaning of grouping by an aggregate since
the
> grammar permits this? Should this be permitted at all?

Perhaps not. The grammar does not explicitly rule out all the
syntactically invalid expressions, some things are left to the
application, to reduce the complexity of the grammar.

> - On the subject of which aggregates to include I think the ones
currently
> proposed are all useful and I can't think of any obvious missing
ones. With
> regards to how MIN and MAX should operate would it be reasonable to
suggest
> that since SPARQL defines a partial ordering over values that MIN/
MAX should
> return the minimum/maximum based on that ordering. This is much
easier to
> implement than doing some form of type detection or having some
complex
> algorithm for how they operate over mixed datatypes. While it does
have the
> disadvantage or potentially returning different results depending
on how
> exactly the SPARQL engine orders values I would be happy with this
> behaviour. If people really need type specific minima/maxima then
they can
> use appropriate FILTERs in their queries or possibly extra
aggregates could
> be introduced eg. NMIN/NMAX (Numeric minimum/maximum)

This is the behavior that the group is proposing to go with.

> Subqueries
> - I would appreciate some clearer guidance on variable scoping as I
don't
> feel comfortable attempting to implement these until I have a
better idea of
> how this should work.

Certainly. The intention is that only variables projected from the
subquery be in scope in the outer expression.

> Projection Expressions
> - I like these very much and have implemented these already. I
personally
> don't like the idea of a LET keyword, it certainly restricts the
ability of
> the SPARQL processor to decide in what order it wishes to execute
the query.
> Plus to my mind it starts to make SPARQL into the equivalent of
Transact SQL
> (and other vendor specific SQL stored procedure languages) which
feels wrong
> to me.

The ordering concern is not necessarily a problem, depending on how
LET would be defined.

Your concern in the second point is noted.

The group has previously decided not to to address LET like structures
in this version of the recommendation, as yet that decision has not
been revisited.

Steve Harris,
on behalf of the SPARQL working group.

Re: Some comments on the SPARQL 1.1 draft documents

2009-12-07T13:26:22Z

Hi Ben,

First, thanks for your comments.

> The first, smaller point was a disappointment with the choice
> of RDF/XML as the one standard required to be supported by the
> new RDF graph HTTP management API. [...] Other easy to parse
> protocols have existed for years. Why the continued support
> of RDF/XML as the canonical standard?

The working group is operating in the W3C realm, and RDF/XML is the only normative format to exchange RDF at this point. While other formats are being used and supported, the standardisation of these formats is not in the scope of the SPARQL WG.

> The larger point was that the sizable syntax extensions make a
> protocol with limited adoption even more difficult to implement.
> Implementations do not exist for several popular web development
> languages, and enlarging the syntax only increases the range of
> features that are 'required' to be supported.

In case this is referring to the Update part of the protocol, let us emphasize that we do not require nor intend that a SPARQL endpoint has to support both SPARQL/Query and SPARQL/Update. In this sense, and in the sense that SPARQL1.1 shall be fully upwards compatible, any SPARQL endpoint implementing only query shall be fully conformant.

There are a large number of implementations of SPARQL 1.0 in a wide variety of language environments spanning both scripting languages and compiled languages. You can see a community-maintained list of implementations here: http://esw.w3.org/topic/SparqlImplementations

The Working Group selected the current set of deliverables based upon three months of deliberations that included feedback from both SPARQL implementors and users as to what features & extensions are most widely needed for improved utility and interoperability of SPARQL.

> I suggested that a better route would be to establish a more
> machine-friendly format as the standard, and define SPARQL in terms of
> how it compiles into that. That algebra, already published alongside
> the syntax, is readably expressible as S-expressions, and those should
> be the protocol, with the human-readable form as an addendum. The end
> result is that *every* language feature can easily be subject to
> extension, service discovery, and the incremental implementation that
> small-scale open source projects need, instead of only those features
> expressible by extension functions.

I am not sure what exactly you are referring to here, but the goal of the group is to develop widely requested and implemented extensions of SPARQL in an upwards compatible fashion. This does not foresee to change the format of the language radically, nor have concrete proposals for such new format been suggested when the group was chartered.

> The service discovery feature was an excellent step in the right
> direction, and some of us would love to see it apply to everything
> SPARQL can do, not just part of it.

Thank you for this comment. Indeed the service description feature is intended to be minimal and extensible by external vocabularies to describe e.g. datasets, additional endpoint features, etc. The group will very much welcome such proposals which might, if widely used, be adopted in further versions of the standard.

with best regards,
Axel Polleres, on behalf of the SPARQL WG

On 5 Nov 2009, at 21:00, Ben Lavender wrote:

> I posted some comments about the 1.1 draft on my blog, which Axel
> Polleres asked that I post here. I will not paste the entire
> contents, as I suffer from a clinical lack of conciseness. Instead, I
> will say my two main points and link.
>
> The first, smaller point was a disappointment with the choice of
> RDF/XML as the one standard required to be supported by the new RDF
> graph HTTP management API. The protocol has issues marked as
> 'postponed' for years. Other easy to parse protocols have existed for
> years. Why the continued support of RDF/XML as the canonical
> standard?
>
> The larger point was that the sizable syntax extensions make a
> protocol with limited adoption even more difficult to implement.
> Implementations do not exist for several popular web development
> languages, and enlarging the syntax only increases the range of
> features that are 'required' to be supported.
>
> I suggested that a better route would be to establish a more
> machine-friendly format as the standard, and define SPARQL in terms of
> how it compiles into that. That algebra, already published alongside
> the syntax, is readably expressible as S-expressions, and those should
> be the protocol, with the human-readable form as an addendum. The end
> result is that *every* language feature can easily be subject to
> extension, service discovery, and the incremental implementation that
> small-scale open source projects need, instead of only those features
> expressible by extension functions.
>
> Further, it's worth noting that SQL has spent the last 20 years being
> abstracted away by necessarily complicated libraries, and having the
> structure of a query language be well-defined in a machine-readable
> format seems to be more forward-looking than the incidental details of
> a human-readable version.
>
> The service discovery feature was an excellent step in the right
> direction, and some of us would love to see it apply to everything
> SPARQL can do, not just part of it.
>
> The long-winded version is at:
>
> http://bhuga.net/2009/11/w3c-going-wrong-direction-sparql-11
>
> Ben Lavender
>
>
>

Re: SPARQL 1.1 Aggregates

2009-11-30T02:54:15Z

Hi Toby,

As for MEAN/MEDIAN, this issue is being discussed, as for the others,
can you give us an indication which current system implements these?
the main rationale with the new features which we try to violate only in exceptional cases is
that we aim to standardise what is implemented by systems "out there". Others can be added by
means of the standard extensibility mechanisms.

As for LONGEST/SHORTEST, that seems to be doable with MIN/MAX in combination with
fn:string-length(), right?

best,
Axel

On 30 Nov 2009, at 09:46, Toby Inkster wrote:

> I see that the built-in set of aggregates for SPARQL 1.1 has not yet
> been decided.
>
> The current list is quite numerically oriented. Here are some I'd like
> to see:
>
> CONCAT - concatenates values, with an optional second
> parameter to provide a joiner character. Result
> is a plain literal with no language.
>
> XML_CONCAT - Concatenates values into an XMLLiteral
> using an SPARQL-Results-like structure.
>
> LONGEST/SHORTEST - returns the longest or shortest
> result (in terms of character count). Optional
> second parameter specifies a language.
>
> MODE/MEDIAN - while AVG returns the mean result, these
> two would return other kinds of average. With
> named graphs, the same triple can occur
> multiple times, so MODE makes sense. Optional
> second parameter specifies a language.
>
> In the case where I've indicated that the second parameter specifies a
> language, the aggregate function would work like this:
>
> 1. Do any values in the list match the specified language?
> (Using same definition of "match" as langMatches.)
> If so, then discard any results which don't match.
>
> 2. Run the aggregate as normal.
>
> So for example, on the following graph:
>
> <http://example.com/cat>
> rdfs:label "cat"@en, "chat"@fr, "feline"@en, "felis"@la.
>
> This SPARQL query:
>
> SELECT ?resource (SHORTEST(?label,"fr") AS ?mylabel)
> WHERE { ?resource rdfs:label ?label . }
>
> Would return:
>
> resource | mylabel
> -------------------------+-----------
> <http://example.com/cat> | "chat"@fr
>
> Because the non-French values would be discarded, with the shortest
> remaining label being selected. However, this:
>
> SELECT ?resource (SHORTEST(?label,"de") AS ?mylabel)
> WHERE { ?resource rdfs:label ?label . }
>
> Would return
>
> resource | mylabel
> -------------------------+-----------
> <http://example.com/cat> | "cat"@en
>
> There was no German label in the data, so the discarding step never
> happens - thus the shortest of any language is selected.
>
> I think in terms of presenting views of graph data, having these
> aggregate language preferences (and they're preferences, not filters, as
> the second example illustrates) would be very useful - especially for
> "label" and "description" kinds of fields.
>
> While I'm giving examples, I'll provide some for CONCAT and XML_CONCAT:
>
> SELECT
> ?resource
> (CONCAT(?label, ";") AS ?concat)
> (XML_CONCAT(?label) AS ?xmlconcat)
> WHERE { ?resource rdfs:label ?label . }
> ORDER BY ?label
>
> ?concat would be "cat;chat;feline;felis" (the ORDER BY clause having
> been used by the aggregate function). ?xmlconcat would be:
>
> """<literal xml:lang="en">cat</literal>
> <literal xml:lang="fr">chat</literal>
> <literal xml:lang="en">feline</literal>
> <literal xml:lang="la">felis</literal>"""^^rdf:XMLLiteral
>
> Perhaps the data type could be more specialised - instead of
> rdf:XMLLiteral, it could be, say, sparql:XMLResultsLiteral, which SPARQL
> libraries could recognise and automagically parse for you.
>
> --
> Toby A Inkster
> <mailto:mail@...>
> <http://tobyinkster.co.uk>
>
>
>

SPARQL 1.1 Aggregates

2009-11-30T01:46:24Z

I see that the built-in set of aggregates for SPARQL 1.1 has not yet
been decided.

The current list is quite numerically oriented. Here are some I'd like
to see:

CONCAT - concatenates values, with an optional second
parameter to provide a joiner character. Result
is a plain literal with no language.

XML_CONCAT - Concatenates values into an XMLLiteral
using an SPARQL-Results-like structure.

LONGEST/SHORTEST - returns the longest or shortest
result (in terms of character count). Optional
second parameter specifies a language.

MODE/MEDIAN - while AVG returns the mean result, these
two would return other kinds of average. With
named graphs, the same triple can occur
multiple times, so MODE makes sense. Optional
second parameter specifies a language.

In the case where I've indicated that the second parameter specifies a
language, the aggregate function would work like this:

1. Do any values in the list match the specified language?
(Using same definition of "match" as langMatches.)
If so, then discard any results which don't match.

2. Run the aggregate as normal.

So for example, on the following graph:

<http://example.com/cat>
rdfs:label "cat"@en, "chat"@fr, "feline"@en, "felis"@la.

This SPARQL query:

SELECT ?resource (SHORTEST(?label,"fr") AS ?mylabel)
WHERE { ?resource rdfs:label ?label . }

Would return:

resource | mylabel
-------------------------+-----------
<http://example.com/cat> | "chat"@fr

Because the non-French values would be discarded, with the shortest
remaining label being selected. However, this:

SELECT ?resource (SHORTEST(?label,"de") AS ?mylabel)
WHERE { ?resource rdfs:label ?label . }

Would return

resource | mylabel
-------------------------+-----------
<http://example.com/cat> | "cat"@en

There was no German label in the data, so the discarding step never
happens - thus the shortest of any language is selected.

I think in terms of presenting views of graph data, having these
aggregate language preferences (and they're preferences, not filters, as
the second example illustrates) would be very useful - especially for
"label" and "description" kinds of fields.

While I'm giving examples, I'll provide some for CONCAT and XML_CONCAT:

SELECT
?resource
(CONCAT(?label, ";") AS ?concat)
(XML_CONCAT(?label) AS ?xmlconcat)
WHERE { ?resource rdfs:label ?label . }
ORDER BY ?label

?concat would be "cat;chat;feline;felis" (the ORDER BY clause having
been used by the aggregate function). ?xmlconcat would be:

"""<literal xml:lang="en">cat</literal>
<literal xml:lang="fr">chat</literal>
<literal xml:lang="en">feline</literal>
<literal xml:lang="la">felis</literal>"""^^rdf:XMLLiteral

Perhaps the data type could be more specialised - instead of
rdf:XMLLiteral, it could be, say, sparql:XMLResultsLiteral, which SPARQL
libraries could recognise and automagically parse for you.

--
Toby A Inkster
<mailto:mail@...>
<http://tobyinkster.co.uk>

Re: Inferencing on graph patterns

2009-11-19T18:28:27Z

Ooops, sorry: I realise just now that you were talking about OWL2 *RL*.
I don't have a full picture of OWL2 RL, but: if OWL2 RL allows for
recursive rules, then again SPARQL can not encode it due to a data
complexity argument - linear time lower bound for recursive rules as
opposed to AC0 (sub-linear and sub-logspace) upper bound for SPARQL.
cheers
--e.

On 20 Nov 2009, at 02:20, Enrico Franconi wrote:

> On 14 Nov 2009, at 11:41, Simon Reinhardt wrote:
>
>>> Note, however, that you may get what you want with a different
>>> query.
>>> For example, in this case, the
>>> SELECT ?type WHERE { ex:C1 rdfs:subclass ?type. }
>>> will return all possible types.
>>
>> Ok. Querying for inferred types was just an example, the initial
>> use case that brought me to this was actually a bit more complex
>> (property restrictions). For this that kind of inferencing would
>> have made the query a lot simpler. But it should still be possible
>> to cover a lot of the inferencing "rules" with SPARQL, especially
>> with property paths. Maybe it's worth investigating at some point
>> how much of OWL 2 RL could be implemented with pure SPARQL - and
>> what extensions would be needed to add the missing bits. But that's
>> just something to keep in mind for the future. :-)
>
> If you fix the entailment regime to RDFS, then very little of OWL2
> can be encoded in SPARQL, since it is has been shown that the
> computational complexities diverge too much. There are also simple
> counter-examples showing that it does not make sense to have an OWL2
> entailment regime in SPARQL, since you would get unsound results
> (wrt OWL2 semantics) very easily. So, really, SPARQL can hardly go
> beyond RDFS.
>
> cheers
> --e.
>
> [1] Enrico Franconi. The logic of RDF and SPARQL: a tutorial.
> Invited talk at the 25th ACM Symposium on Principles of Database
> Systems (PODS-2006), in Chicago IL, on 26-28 June 2006. <http://www.inf.unibz.it/~franconi/papers/franconi-slides-pods-2006.pdf
> >
>
>

smime.p7s (3K) Download Attachment

Re: Inferencing on graph patterns

2009-11-19T18:20:35Z

On 14 Nov 2009, at 11:41, Simon Reinhardt wrote:

>> Note, however, that you may get what you want with a different query.
>> For example, in this case, the
>> SELECT ?type WHERE { ex:C1 rdfs:subclass ?type. }
>> will return all possible types.
>
> Ok. Querying for inferred types was just an example, the initial use
> case that brought me to this was actually a bit more complex
> (property restrictions). For this that kind of inferencing would
> have made the query a lot simpler. But it should still be possible
> to cover a lot of the inferencing "rules" with SPARQL, especially
> with property paths. Maybe it's worth investigating at some point
> how much of OWL 2 RL could be implemented with pure SPARQL - and
> what extensions would be needed to add the missing bits. But that's
> just something to keep in mind for the future. :-)

If you fix the entailment regime to RDFS, then very little of OWL2 can
be encoded in SPARQL, since it is has been shown that the
computational complexities diverge too much. There are also simple
counter-examples showing that it does not make sense to have an OWL2
entailment regime in SPARQL, since you would get unsound results (wrt
OWL2 semantics) very easily. So, really, SPARQL can hardly go beyond
RDFS.

cheers
--e.

[1] Enrico Franconi. The logic of RDF and SPARQL: a tutorial. Invited
talk at the 25th ACM Symposium on Principles of Database Systems
(PODS-2006), in Chicago IL, on 26-28 June 2006. <http://www.inf.unibz.it/~franconi/papers/franconi-slides-pods-2006.pdf
>

smime.p7s (3K) Download Attachment

Re: Inferencing on graph patterns

2009-11-19T18:09:01Z

On 14 Nov 2009, at 05:30, Ivan Herman wrote:

>> Consider the following data in a store:
>>
>> (1) ex:C1 rdfs:subClassOf ex:C2 .
>> (2) :myC1 a ex:C1 .
>>
>> Now when you do the following query at the endpoint for the store:
>>
>> SELECT ?type WHERE { ?x a ex:C1 , ?type }
>>
>> and it supports the RDFS entailment regime the result would be
>> ex:C1 and
>> ex:C2 because the store would infer :myC1 to be of type ex:C2, either
>> when the data got added or on-the-fly.
>> But what if the store didn't contain triple (2) and so no data
>> about any
>> instances of those two classes? Is there any way that it could make
>> inferences about the data in the WHERE clause and therefore infer
>> that
>> ?x would be of type ex:C2 as well? I guess not since this is
>> querying by
>> matching patterns rather than "query by example". But I can see
>> various
>> uses for this. So if this is not how it's currently defined to work,
>> have you considered this?
>
> The inference regime is defined on possible solution. That indeed
> means
> that no inference is made on variables which means, as you say, that
> without the triple (2) the required results will not be returned. And
> there is no mechanism envisioned in SPARQL to achieve that.

The real "logical" reason why the answer is empty is that the ?x
variable is understood to be existential, so you are actually asking
"all the types of an object ?x if such object ?x *exists* and it is of
type ex:C1". But without (2) it is not guaranteed that such an object
*necessarily* exists, therefore the answer is empty.

> Note, however, that you may get what you want with a different query.
> For example, in this case, the
>
> SELECT ?type WHERE { ex:C1 rdfs:subclass ?type. }
>
> will return all possible types.

Well, only the one that can be inferred by the transitive closure of
rdfs:subClassOf. Indeed rdfs will never return any other type which
could be inferred by other means -- look at some examples of
inferences that will never be drawn by RDFS in the attachment.

cheers
--e.

rdfs-incompleteness.png (71K) Download Attachment
smime.p7s (3K) Download Attachment

Re: Comments on SPARQL 1.1 WD 20091022

2009-11-19T05:28:58Z

On 24/10/2009 14:09, Leigh Dodds wrote:
Leigh,

Thank you for your comments.

This first published working draft contains the new areas for query that
the working group is progressing. The material will be integrated with
the the previous version of the query language to produce a single, new
document for SPARQL 1.1 Query.

Things may be a little rough in this first draft ...

Leigh Dodds wrote:
> Hi,
>
> Here are some personal comments/questions on the 22/10/2009 WD of
SPARQL 1.1.
>
> * Example in Section 2. I don't think the project expression conforms
> with grammar shown latter in the doc; missing brackets? I know this is
> still up for discussion, but thought I'd point it out

Yes - it's not consistent. The WG had not reached a conclusion on the
exact syntax at the time of publication.

> * Section 2, Syntax. re: the note about using FILTER instead of
> HAVING. Assuming I'm reading the comment correctly, I think for
> readability purposes its better to have a separate keyword (HAVING)
> rather than re-use the FILTER keyword. I think its clearer that there
> are different constraints (i.e. aggregates allowed or not) on the
> expression, and retains similarity with SQL.
>
> * Section 3. Text above example query is wrong, I think it should
> ready "from all the people that Alice knows", not "that know Alice".

Agreed.

>
> * Section 4. What is the rationale for including both a FILTER and a
> graph pattern operator for EXISTS/NOT EXISTS? If there are benefits in
> terms of expressivity or ease of implementation it would be good to be
> able to call these out in the specification.

If the working group decides to allow EXISTS/NOT EXISTS in FILTERs, then
there will be a example to illustrate the point.

> * Section 4. The NOT EXISTS and EXISTS graph pattern operators are
> described as "applying only to variables defined earlier in the
> pattern". What does "earlier" mean if a SPARQL processor can re-order
> the statements for optimization purposes? Does use of those operators
> have some impact on an implementations ability to do that?

It's loose wording to discuss the scoping of variables. "earlier" means
the pattern to the left or above of the NOT EXISTS.

>
> Cheers,
>
> L.

Andy
on behalf of the SPARQL Working Group

Re: Inferencing on graph patterns

2009-11-14T03:41:44Z

Ivan Herman wrote:

> Dear Simon,
>
> Thanks for your comments. And apologies for the somewhat late reply.
>
> You wrote:
> > In trying to understand SPARQL 1.1 Entailment Regimes [1] I was
> > wondering if inferencing will ever take place on the RDF data in graph
> > patterns themselves. Consider the following data in a store:
> >
> > (1) ex:C1 rdfs:subClassOf ex:C2 .
> > (2) :myC1 a ex:C1 .
> >
> > Now when you do the following query at the endpoint for the store:
> >
> > SELECT ?type WHERE { ?x a ex:C1 , ?type }
> >
> > and it supports the RDFS entailment regime the result would be ex:C1 and
> > ex:C2 because the store would infer :myC1 to be of type ex:C2, either
> > when the data got added or on-the-fly.
> >
> > But what if the store didn't contain triple (2) and so no data about any
> > instances of those two classes? Is there any way that it could make
> > inferences about the data in the WHERE clause and therefore infer that
> > ?x would be of type ex:C2 as well? I guess not since this is querying by
> > matching patterns rather than "query by example". But I can see various
> > uses for this. So if this is not how it's currently defined to work,
> > have you considered this?
>
> The inference regime is defined on possible solution. That indeed means
> that no inference is made on variables which means, as you say, that
> without the triple (2) the required results will not be returned. And
> there is no mechanism envisioned in SPARQL to achieve that.

Thanks for your reply, Ivan. Yes, that makes sense. And fair enough. :-)

> Note, however, that you may get what you want with a different query.
> For example, in this case, the
>
> SELECT ?type WHERE { ex:C1 rdfs:subclass ?type. }
>
> will return all possible types.

Ok. Querying for inferred types was just an example, the initial use case that brought me to this was actually a bit more complex (property restrictions). For this that kind of inferencing would have made the query a lot simpler. But it should still be possible to cover a lot of the inferencing "rules" with SPARQL, especially with property paths. Maybe it's worth investigating at some point how much of OWL 2 RL could be implemented with pure SPARQL - and what extensions would be needed to add the missing bits. But that's just something to keep in mind for the future. :-)

Regards,
Simon

Re: Inferencing on graph patterns

2009-11-13T21:30:40Z

Dear Simon,

Thanks for your comments. And apologies for the somewhat late reply.

You wrote:
> In trying to understand SPARQL 1.1 Entailment Regimes [1] I was
> wondering if inferencing will ever take place on the RDF data in graph
> patterns themselves. Consider the following data in a store:
>
> (1) ex:C1 rdfs:subClassOf ex:C2 .
> (2) :myC1 a ex:C1 .
>
> Now when you do the following query at the endpoint for the store:
>
> SELECT ?type WHERE { ?x a ex:C1 , ?type }
>
> and it supports the RDFS entailment regime the result would be ex:C1 and
> ex:C2 because the store would infer :myC1 to be of type ex:C2, either
> when the data got added or on-the-fly.
>
> But what if the store didn't contain triple (2) and so no data about any
> instances of those two classes? Is there any way that it could make
> inferences about the data in the WHERE clause and therefore infer that
> ?x would be of type ex:C2 as well? I guess not since this is querying by
> matching patterns rather than "query by example". But I can see various
> uses for this. So if this is not how it's currently defined to work,
> have you considered this?

The inference regime is defined on possible solution. That indeed means
that no inference is made on variables which means, as you say, that
without the triple (2) the required results will not be returned. And
there is no mechanism envisioned in SPARQL to achieve that.

Note, however, that you may get what you want with a different query.
For example, in this case, the

SELECT ?type WHERE { ex:C1 rdfs:subclass ?type. }

will return all possible types.

Sincerely

Ivan
On behalf of the SPARQL Working Group

--

Ivan Herman, W3C Semantic Web Activity Lead
Home: http://www.w3.org/People/Ivan/
mobile: +31-641044153
PGP Key: http://www.ivan-herman.net/pgpkey.html
FOAF: http://www.ivan-herman.net/foaf.rdf

smime.p7s (5K) Download Attachment

Re: Comments on SPARQL 1.1

2009-11-13T06:08:51Z

Rob,

Thanks for all of the feedback, it's very helpful. We're working through
all of these issues right now, so we'll get back to you with a better
response in due course.

Lee

Rob Vesse wrote:

> Hi all
>
> Here are my comments/questions on the draft. Some of this is based on
> personal opinion and some of this is based on starting to try and implement
> some of this stuff in my SPARQL implementation in my latest development
> builds of dotNetRDF (early Alpha release .Net RDF library).
>
> Aggregates
> - I definitely support the use of a separate HAVING keyword, I agree with
> Leigh that it makes clear that the type of constraint is different and helps
> SQL programmers make the move to SPARQL by making the syntax familiar
>
> - The proposed grammar has a couple of things which bug me:
>
> 1. It allows for a Having clause without a Group By clause which doesn't
> necessarily make sense depending on the constraint given. I assume that if
> you have a Having clause without a Group By then it should act as a Filter
> on the result set?
> The only reason I can see for allowing this is if the having clause uses
> an aggregate then this makes sense in some circumstances e.g. seeing whether
> there are a given number of Triples matching a pattern:
>
> SELECT * WHERE {?s a ?type} HAVING (COUNT(?s) > 10)
>
> 2. What is the intended meaning of grouping by an aggregate since the
> grammar permits this? Should this be permitted at all?
>
> - On the subject of which aggregates to include I think the ones currently
> proposed are all useful and I can't think of any obvious missing ones. With
> regards to how MIN and MAX should operate would it be reasonable to suggest
> that since SPARQL defines a partial ordering over values that MIN/MAX should
> return the minimum/maximum based on that ordering. This is much easier to
> implement than doing some form of type detection or having some complex
> algorithm for how they operate over mixed datatypes. While it does have the
> disadvantage or potentially returning different results depending on how
> exactly the SPARQL engine orders values I would be happy with this
> behaviour. If people really need type specific minima/maxima then they can
> use appropriate FILTERs in their queries or possibly extra aggregates could
> be introduced eg. NMIN/NMAX (Numeric minimum/maximum)
>
> Subqueries
> - I would appreciate some clearer guidance on variable scoping as I don't
> feel comfortable attempting to implement these until I have a better idea of
> how this should work.
>
> Projection Expressions
> - I like these very much and have implemented these already. I personally
> don't like the idea of a LET keyword, it certainly restricts the ability of
> the SPARQL processor to decide in what order it wishes to execute the query.
> Plus to my mind it starts to make SPARQL into the equivalent of Transact SQL
> (and other vendor specific SQL stored procedure languages) which feels wrong
> to me.
>
> Rob Vesse
> dotNetRDF Lead Developer
> ================================================================
> Developer Discussion & Feature Request -
> dotnetrdf-develop@...
> Bug Reports - dotnetrdf-bugs@...
> User Help & Support - dotnetrdf-support@...
>
> Website: http://www.dotnetrdf.org
> User Guide: http://www.dotnetrdf.org/content.asp?pageID=User%20Guide
> API: http://www.dotnetrdf.org/api/
> ================================================================
>
>
>
>

Comments on SPARQL 1.1

2009-11-13T02:46:45Z

Hi all

Here are my comments/questions on the draft. Some of this is based on
personal opinion and some of this is based on starting to try and implement
some of this stuff in my SPARQL implementation in my latest development
builds of dotNetRDF (early Alpha release .Net RDF library).

Aggregates
- I definitely support the use of a separate HAVING keyword, I agree with
Leigh that it makes clear that the type of constraint is different and helps
SQL programmers make the move to SPARQL by making the syntax familiar

- The proposed grammar has a couple of things which bug me:

1. It allows for a Having clause without a Group By clause which doesn't
necessarily make sense depending on the constraint given. I assume that if
you have a Having clause without a Group By then it should act as a Filter
on the result set?
The only reason I can see for allowing this is if the having clause uses
an aggregate then this makes sense in some circumstances e.g. seeing whether
there are a given number of Triples matching a pattern:

SELECT * WHERE {?s a ?type} HAVING (COUNT(?s) > 10)

2. What is the intended meaning of grouping by an aggregate since the
grammar permits this? Should this be permitted at all?

- On the subject of which aggregates to include I think the ones currently
proposed are all useful and I can't think of any obvious missing ones. With
regards to how MIN and MAX should operate would it be reasonable to suggest
that since SPARQL defines a partial ordering over values that MIN/MAX should
return the minimum/maximum based on that ordering. This is much easier to
implement than doing some form of type detection or having some complex
algorithm for how they operate over mixed datatypes. While it does have the
disadvantage or potentially returning different results depending on how
exactly the SPARQL engine orders values I would be happy with this
behaviour. If people really need type specific minima/maxima then they can
use appropriate FILTERs in their queries or possibly extra aggregates could
be introduced eg. NMIN/NMAX (Numeric minimum/maximum)

Subqueries
- I would appreciate some clearer guidance on variable scoping as I don't
feel comfortable attempting to implement these until I have a better idea of
how this should work.

Projection Expressions
- I like these very much and have implemented these already. I personally
don't like the idea of a LET keyword, it certainly restricts the ability of
the SPARQL processor to decide in what order it wishes to execute the query.
Plus to my mind it starts to make SPARQL into the equivalent of Transact SQL
(and other vendor specific SQL stored procedure languages) which feels wrong
to me.

Rob Vesse
dotNetRDF Lead Developer
================================================================
Developer Discussion & Feature Request -
dotnetrdf-develop@...
Bug Reports - dotnetrdf-bugs@...
User Help & Support - dotnetrdf-support@...

Website: http://www.dotnetrdf.org
User Guide: http://www.dotnetrdf.org/content.asp?pageID=User%20Guide
API: http://www.dotnetrdf.org/api/
================================================================

Inferencing on graph patterns

2009-11-09T04:06:41Z

Hi,

In trying to understand SPARQL 1.1 Entailment Regimes [1] I was wondering if inferencing will ever take place on the RDF data in graph patterns themselves. Consider the following data in a store:

(1) ex:C1 rdfs:subClassOf ex:C2 .
(2) :myC1 a ex:C1 .

Now when you do the following query at the endpoint for the store:

SELECT ?type WHERE { ?x a ex:C1 , ?type }

and it supports the RDFS entailment regime the result would be ex:C1 and ex:C2 because the store would infer :myC1 to be of type ex:C2, either when the data got added or on-the-fly.

But what if the store didn't contain triple (2) and so no data about any instances of those two classes? Is there any way that it could make inferences about the data in the WHERE clause and therefore infer that ?x would be of type ex:C2 as well? I guess not since this is querying by matching patterns rather than "query by example". But I can see various uses for this. So if this is not how it's currently defined to work, have you considered this?

Thanks,
Simon

[1] http://www.w3.org/TR/2009/WD-sparql11-entailment-20091022/

Some comments on the SPARQL 1.1 draft documents

2009-11-05T13:00:56Z

I posted some comments about the 1.1 draft on my blog, which Axel
Polleres asked that I post here. I will not paste the entire
contents, as I suffer from a clinical lack of conciseness. Instead, I
will say my two main points and link.

The first, smaller point was a disappointment with the choice of
RDF/XML as the one standard required to be supported by the new RDF
graph HTTP management API. The protocol has issues marked as
'postponed' for years. Other easy to parse protocols have existed for
years. Why the continued support of RDF/XML as the canonical
standard?

The larger point was that the sizable syntax extensions make a
protocol with limited adoption even more difficult to implement.
Implementations do not exist for several popular web development
languages, and enlarging the syntax only increases the range of
features that are 'required' to be supported.

I suggested that a better route would be to establish a more
machine-friendly format as the standard, and define SPARQL in terms of
how it compiles into that. That algebra, already published alongside
the syntax, is readably expressible as S-expressions, and those should
be the protocol, with the human-readable form as an addendum. The end
result is that *every* language feature can easily be subject to
extension, service discovery, and the incremental implementation that
small-scale open source projects need, instead of only those features
expressible by extension functions.

Further, it's worth noting that SQL has spent the last 20 years being
abstracted away by necessarily complicated libraries, and having the
structure of a query language be well-defined in a machine-readable
format seems to be more forward-looking than the incidental details of
a human-readable version.

The service discovery feature was an excellent step in the right
direction, and some of us would love to see it apply to everything
SPARQL can do, not just part of it.

The long-winded version is at:

http://bhuga.net/2009/11/w3c-going-wrong-direction-sparql-11

Ben Lavender

Re: Concerning LET or AS

2009-11-03T22:58:05Z

As a personal comment, (sorry you are probably sick of me now). I don't
particularly seek a response to this comment.

I am surprised that there is concern that the LET single assignment
construct may mislead users into having an incorrect processing model in
their heads that might be overly procedural.

This surprise is because the whole point about having a declarative
semantics is that the processing model is irrelevant. Thus, with a
declarative language, we expect, perhaps even desire, that users have
incorrect processing models. Each implementation is free to use their
own processing model, and the user works with their own. For example,
when running an XSLT script, if it has some side effect of writing a
message to the console, it is often surprising when these messages get
written. This is because the easiest way to think of the XSLT processing
model is top-down left-to-right, but good implementations tend to be
lazy. This mismatch between the users model and the implementor's
reality is desirable because:
a) it makes it easier for the user to understand the language
b) it allows the implementor to efficiently implement the language
c) the declarative language design ensures that it doesn't matter that
these two views of the processing model differ, perhaps radically.

So, I think an advantage of the term LET as opposed to BIND (say) is
that LET reminds some users of procedural programming in BASIC, and
allows them to reuse that programming model. Now, while the details of
the execution flow are very different in SPARQL than in BASIC, it seems
that this apparent familiarity has pedagogical advantages.

==

For the record, TopQuadrant's position is we don't care what word is
used, whether it is LET or BIND or something else.

A further aside is that the latest release of TopBraid Composer includes
a SPARQL debugger function that exposes some aspects of the insides of
the SPARQL processing (I haven't used it). But I guess that using such a
tool would quickly disabuse you of any incorrect notions of the
processing model. (LET, I believe, is one of the SPARQL extensions
supported by the tool).

Jeremy

Re: Concerning LET or AS

2009-11-03T22:43:48Z

I made a couple of mistakes in my previous text.

Please allow me to withdraw that text and try again.

The errors were:
1) I missed the { } around the subselect
2) my modification to rule 43 lost the SubSelect expansion
3) Removed too many '(' ')' in rewrite rule

I made some modifications for clarity too.

Also as a very minor comment, rule [43] etc combined with the gramar
rules from SPARQL 1.0 do not seem to expand to the example query
immediately above.

I take the intent of rule 43 to be:

GroupGraphPattern ::= '{' ( SubSelect | GroupGraphPatternSub )+ '}'

(Without the +, the rule matches either a single subselect or a SPARQL
1.0 body,
but not a combination of both.)

Here is modified text:

===============================

'LET' is specified as a macro-expansion, in terms of subselect queries.
In the FPWD of Query 1.1 we modify rule 43 for GroupGraphPattern as follows:

[43*] GroupGraphPattern ::= '{' GroupGraphPatternLetSub '}'

[A] GroupGraphPatternLetSub ::= ( GroupGraphPatternLetSub LetExpr '.'?
)? GroupGraphPatternNoLetSub

[B] LetExpr ::= 'LET' '(' Var ':=' Expression ( ',' Var ':='
Expression )* ')'

[C] GroupGraphPatternNoLetSub ::= ( SubSelect | GroupGraphPatternSub )+

Rules [A] and [B] are interpreted by rewriting queries involving LET
into queries not involving LET.
We will use phi(x) to be the rewritten query of x.
For clarity of exposition we will expand the two alternative readings of
[A] as
[A.1] GroupGraphPatternLetSub ::= GroupGraphPatternNoLetSub
[A.2] GroupGraphPatternLetSub ::= GroupGraphPatternLetSub LetExpr '.'?
GroupGraphPatternNoLetSub
Expressions matching rule [A.1] are not rewritten.

If x matches rule B, then:

phi(x) = 'SELECT' * '(' Expression 'AS' Var ')' ( '(' Expression 'AS'
Var ')' )*

(with the variables matching respectively).

If x matches rule A.2 with y matching GroupGraphPatternLetSub on the
R.H.S.,
z matching LetExpr, and w matching GroupGraphPatternNoLetSub then

phi(x) =
'{' phi(z)
'WHERE' '{'
phi(y)
'}'
'}'
w

After this rewrite is applied to all instances matching rules [A] and
[B], the rewritten query does not involve 'LET' and its meaning is as
given in
the rest of the specification.
======================

Re: Concerning LET or AS

2009-11-03T14:51:40Z

off list Steve and Lee encouraged me to be clearer about what I thought
LET as a keyword means.

Here is my attempt at specifying it (based on the WG Wiki page).
Please note that I am not responsible for TopQuadrant's SPARQL work;
Holger is our expert, and we tend to be dependent on Andy's implementation.
So, I am happy with any corrections from Andy.

It is not important how the word LET is spelt (i.e. as far as I know,
TopQuadrant has no particular attachment to 'LET' rather than 'BIND' for
example).

================================

In the FPWD of Query 1.1 we modify rule 43 for GroupGraphPattern as follows:

[43*] GroupGraphPattern ::= '{' GroupGraphPatternLetSub '}'

[A] GroupGraphPatternLetSub ::= ( GroupGraphPatternLetSub Let '.'? )?
GroupGraphPatternSub

[B] Let ::= 'LET' '(' Var ':=' Expression ( ',' Var ':=' Expression
)* ')'

Rules [43*] [A] and [B] are interpreted by rewriting queries involving
LET into queries
not involving LET.
We will use phi(x) to be the written query of x.

If x matches rule B, then:

phi(x) = 'SELECT' '(' * '(' Expression 'AS' Var ')' ( '(' Expression
'AS' Var ')' )* ')'

(with the variables matching respectively).

If x matches rule A then

phi(x) =
phi(Let)
'WHERE' '{'
phi(GroupGraphPatternLetSub)
'}'
GroupGraphPatternSub

The rest of the specification then applies.
==================

(Note this is a fine recipe for implementing as well).
Specifically, this prohibits forward references.
Being a macro expansion into a declarative form, this is declarative.

Jeremy

Re: Concerning LET or AS

2009-11-03T09:13:25Z

I thought I should share a couple of the comments I have had on this
topic from TopQuadrant colleagues:

[[
Speaking as a person who teaches this stuff, more than our own
reputation is at stake. Perhaps this is something to add to our objection.

Technology adoption is the goal of a standard. SPARQL fights an uphill
battle for a couple of reasons: (1) It's Not SQL. (2) Pattern-based
retrieval is weird (witness the fabulous popularity of PROLOG as a
software engineering language). In short, many people are looking for
reasons not to adopt it, and to stay with familiarity.
In short, the semantic web is faced with a huge hurdle in SPARQL. And
while I applaud the "small standard" policy (which in general is a boon
to teaching and adoption), it is only good when it serves ease of adoption.

SPARQL 1.0 has some huge problems, that give SQL fans great ammunition
when it comes to saying "it's not ready" - negation and aggregates are
the biggies here, and both of them have been fixed.

Adding in another complex idiom (like nested subqueries) for something
simple (LET) will be repeating the mistake of negation. Another reason
to say, "Wait for SPARQL 3".
I would go so far as to use the "small standard" argument the other
way. Subqueries are difficult - in fact, in my course, I say, 'The
reason SPARQL doesn't have subquery is because it is not needed. The
sorts of things that you use them for in SQL are done easily in a
pattern language, without resorted to a complex construct like a
subquery". I challenge the room to prove me wrong. One person was able
to do so. No SQL programmer can do it - subqueries are error prone and
confusing.

So - faced with a confusing, difficult, unsuccessful idea from SQL
(subqueries) vs a well-accepted idea from BASIC, which one fits the
"small standard" mantra better? I can speak confidently as an educator
in this stuff. LET wins hands-down.
]]

and

[[
Also look at our mailing list to see what our customers are doing. [X]'s
message from yesterday contains:

SELECT ?stringPredicate ?stringObject ?stringAttributesNodeName ?
stringAttributeNodeName
WHERE {
CQ:GatheredData ?predicate ?object .
LET (?stringPredicate := smf:cast(smf:name(?predicate),
xsd:string)) .
LET (?stringObject := smf:cast(smf:name(?object), xsd:string)) .
LET (?uuid := smf:generateUUID()) .
?attributes a CQ:attributes .
LET (?attributesNodeName := smf:qname(?attributes)) .
LET (?stringAttributesNodeName := smf:cast(smf:name(?
attributesNodeName), xsd:string)) .
LET (?stringAttributeNodeName := smf:buildString("{?
stringAttributesNodeName}-{?uuid}")) .
}

Will make a nice chain of sub-SELECTs...

]]

Re: Concerning LET or AS

2009-11-02T23:39:15Z

PS

My example come from putting together the following thoughts ...
A suggestion that what people don't like about LET is that it is
'procedural'.
However single-assignment is declarative, and that perceiving LET as
procedural is a failure of understanding.
Ordering constraints can be declarative or procedural, LET introduces
declarative ordering constraints.
The ordering constraints appear because of the shape of the problem: for
example if you compute an end date from a start date and a duration, you
need to know the start date and the duration.
Each LET declaratively but concisely introduces an ordering constraint.
The alternative SELECT AS WHERE construct declaratively and verbosely
introduces an ordering constraint.

There is a natural ordering to do with the flow of information. This
isn't necessarily the order of computation, but it is an order in which
it is easier for the query author to think about the query. The LET
syntax follows this natural ordering, the AS syntax does not.

Concerning LET or AS

2009-11-02T23:30:23Z

Thank you for your time and attention at the WG meeting today.

TopQuadrant would like Holger's earlier comment [1] to be treated as a
formal comment. (i.e. with an official WG response on this mailing list).

My understanding from today's meeting is that that is likely to be that
the WG has already considered the LET design and believes the AS design
to be adequate.
(LET is merely an abbreviated form for certain AS constructs). I also do
not believe that TopQuadrant is bringing any new information that was
not considered at your f2f meeting [2].

We however feel strongly about this, and are likely to raise a formal
objection (in the sense that we believe it would be better for the WG to
take a few weeks longer over SPARQL 1.1 and get this right, than to
deliver SPARQL 1.1 on schedule without this feature).

Thinking through particularly Steve's comments, I tried to come up with
an example illustrating how the ordering of operations that is sometimes
required is better articulated with LET than with AS.
This example is not as polished as I would like, since I believe it is
more helpful to contribute during your F2F meeting.

First I wish to clarify that this is not about whether or not assignment
should be in SPARQL 1.1. Assignment is in already, with the AS construct
that was discussed under item 39. This issue is purely about the syntax
and scoping rules for the single assignment capability.

Many of the sort of processing tasks that we and are customers have
involve mapping several legacy sources together, merging them into one
RDF graph, and then doing some processing.
A frequent problem is that different legacy sources represent the same
data in different ways, e.g. with different case conventions, in
different units, or whatever. In these cases, data laundry of one sort
or another is necessary. One option for laundry is using functions and
assignment within SPARQL.

So for my example, I am taking information about alumni at a college and
trying to find the appropriate year photo for them.
I will simplify the name problem to a name consist of a first name and a
last name, (no middle initial), but people change their last name from
time to time.

The data sources that I have include:
- a current mailing database, with full-names, e-mail addresses, and
addresses
a:fullName a:email a:address
_:w a:fullName "John Smith" .
_:w a:email <mailto:john.smith@...>.

- a database with students first names and last names and former last names
to simplify processing I just use two properties
b:firstName
b:lastName

for example:
_:x b:firstName "John" .
_:x b:lastName "Doe" .
_:x b:lastName "Smith".

shows that the person known as John Doe and the person known as John
Smith are one and the same, without clarifying the chronology of the
name change.

- a database with date of matriculation, and years of study, by full
name at time of matriculation
c:matriculationDate c:studyYears c:fullName
_:y c:fullName "John Doe" .
_:y c:studyYears "P1Y"^^xs:yearMonthDuration .
_:y c:matriculationDate "1988-09-01"^^xsd:date.

- and a list of graduation photo names by year.
d:year d:fileName
_:z d:year "1988"^^xsd:date
_:z d:fileName "classOf88"

- I have arranged these photos as jpg files on the web at
http://www.example.org/photos
http://www.example.org/photos/classOf88.jpg

SELECT ?eMail ?image
WHERE
{ ?a a:email ?eMail .
?a e:fullName ?fullName
LET ( ?fullNameSpaceNormalized=normalize-space(?fullName) ) [A]
LET ( ?firstName=substring-before(?fullNameSpaceNormalized," ") [B]
?lastName=substring-after(?fullNameSpaceNormalized," ") )
?b b:firstName ?firstName .
?b b:lastName ?lastName .
?b b:lastName ?altLastName . [C]
LET ( ?altName=concat(?firstName, " ", ?altLastName ) )
?c c:fullName ?a;tName .
?c c:studyYears ?lengthOfCourse .
?c c:matriculationDate ?matriculate .
LET (?endDate=|year-from-date(add-yearMonthDuration-to-date(?matriculate,?lengthOfCourse)) )
?d d:year ?endDate .
?d d:fileName ?imageFile .
LET ( ?image = xs:anyURI(concat("http://www.example.org/photos", ?imageFile, ".jpg" ) ) )|
}

Notes:
[A] for robustness against leading/trailing space and/or double space in
the middle
[B] cannot be combined with [A] because of rules discussed under issue 39
[C] ?altLastName can be the same as ?lastName

I believe the WG is considering recommending that this query should be
written as follows.

SELECT ?eMail,
xs:anyURI(concat("http://www.example.org/photos", ?imageFile,
".jpg" ) ) as ?image
WHERE {
SELECT ( *
year-from-date(add-yearMonthDuration-to-date(?matriculate,?lengthOfCourse))

AS ?endDate )
WHERE {
SELECT ( * concat(?firstName, " ", ?altLastName ) AS ?altName )
WHERE {
SELECT (* substring-before(?fullNameSpaceNormalized," ")
AS ?firstName,
substring-after(?fullNameSpaceNormalized," ") AS ?lastName )
WHERE {
SELECT (* normalize-space(?fullName) as ?fullNameSpaceNormalized)
WHERE {
?a a:email ?eMail .
?a e:fullName ?fullName .
}
}
?b b:firstName ?firstName .
?b b:lastName ?lastName .
?b b:lastName ?altLastName .
}
?c c:fullName ?a;tName .
?c c:studyYears ?lengthOfCourse .
?c c:matriculationDate ?matriculate .
}
?d d:year ?endDate .
?d d:fileName ?imageFile .
}

(Using the equivalence from [3])
We believe that this is inferior.
Harder to write, harder to read, harder to understand, and that the cost
of complicating the language by having two ways to say the same thing is
well worth it.

Jeremy Carroll
AC Rep, TopQuadrant.

[1]
http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Oct/0003
[2]
http://www.w3.org/2009/sparql/meeting/2009-05-06#ProjectExpressions___26___20_Assignment

[3]
http://www.w3.org/2009/sparql/wiki/Feature:Assignment#Equivalence_with_SubSelects_and_ProjectExpressions

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-30T11:12:05Z

On Oct 28, 2009, at 11:13 PM, Richard Newman wrote:

> I believe that a large portion of SPARQL users (maybe all of the non-
> experts) think procedurally when writing queries. They're not
> thinking about satisfying clauses, they're thinking about "fetch all
> the subjects with this object, then fetch all their names, then
> filter out the ones with...".
>
> This is why they're surprised at unexpected results, or unexpected
> performance: the algebraic interpretation of their queries is very
> different to what they think they've written.
>
> We're all far too close to RDF query languages to remember how non-
> implementors think.
>
> My wife is a UX person. In that field it's considered wise to never
> think of the user being wrong: if they've come to the incorrect
> conclusion, it's very likely because of something you've done or not
> done, and it's the software that should change, not the user. It
> would be interesting to run a user test of SPARQL; I'm sure we'd
> learn a huge amount about the assumptions and pain points of people
> actually trying to solve problems with it.

I could not agree more with this. But the problems go further than
just perception. Even if I had fully understood all the details of the
SPARQL algebra, I still do not know what kind of reorderings will
happen inside the query engine. The engine may reorder FILTERs because
of some heuristics. These heuristics may be unsuitable, or they may be
misinformed because statistical data about the triple store is not
always available. In those cases, the system should just use the order
in which the user has specified the clauses.

> I think Holger's point is that SPARQL as specified loses a lot of
> the information that the query writer has encoded in the query. (He
> surely knows that FILTERs are not order dependent: that's what he's
> lamenting.)
>
> Most people do not think in an order-independent fashion,
> particularly when other language constructs such as OPTIONAL *are*
> ordered (after a fashion).
>
> I see users interspersing FILTERs throughout their queries all the
> time. Very often they do it because they know it's the best way to
> run the query. The query language then says "pull out all the
> FILTERs", and the implementation then has to decide how to run
> them... and it might not have as much information as does the user.
> (For example, when the execution of a custom FILTER function is very
> expensive, and you need to trick the planner to execute it later or
> earlier.)
>
> Put another way: I've never *ever* seen a user write something like
>
> SELECT * {
> FILTER (?name ...)
> ?x foaf:name ?name .
> ...
> }
>
> even though it's meaningful SPARQL. Perhaps it shouldn't be
> meaningful.
>
> This problem gets worse when you consider subqueries, remote
> queries, computed properties...
>
> Perhaps order-dependence is actually an intuitive, reasonable
> default for a language? Imperative programming language compilers
> have done a pretty good job starting with ordered statements, and
> figuring out when they can disregard that to get better parallelism.
> That's an optimization, not the default.
>
> Devil's advocacy over :)

+1

In order to not break backward compatibility (albeit debatable), maybe
a new keyword such as SELECT ... WHERE PROCEDURAL { ... } (not a nice
name yet, but you get the idea) could be introduced to help the engine
decide whether a query has been written by someone with a procedural
background or by someone who comes from the SQL world, and who expects
the engine to do the reordering for him.

Holger

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-30T10:49:25Z

Thanks for pointing this out, Andy. Its good to hear there is already some control over the order of execution this way, and this mechanism would make the job easier for a LET macro.

But this also dramatically highlights how unattractive the notation below would be for real world usage. Not only is it way too verbose, but it also requires a mind set that the average user does not have - at least my intuition was misleading. Related working groups (in particular OWL) have made similar mistakes in the past, trying to design a language that is theoretically sound and clear, but unintuitive to use as it does not align with the traditional imperative/object-oriented way of thinking. I have a long list of customer stories (both for SPARQL and OWL) where people were disappointed because the language did not produce the results they expected. In particular the lack of reliable ordering of FILTER clauses often leads to very very slow queries and people believe the whole semantic web stack is not ready for prime time yet.

Holger

On Oct 30, 2009, at 3:32 AM, Seaborne, Andy wrote:

I think you need to write it as:

SELECT ?newLiteral
WHERE {
      { SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral)
        WHERE
        {
          SELECT ((?fromValue * ?rate) AS ?newValue)
          WHERE
          {
            SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue)
            WHERE
            {
             SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate)
               WHERE {}
            }
        }
    }
}

To put the inner calculations into the scope of the outer ones.

Whereas

SELECT ?newLiteral
WHERE {
                {
                                SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {}
                }
                {
                              SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}
                }
                {
                                SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}
                }
                {
                                SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}
                }
}

evaluates each SELECT separately and joins the results together. So for example:

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

Is

------------
| newValue |
============
|          |
------------

And does not see ?rate from the other SELECT

                Andy

From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-comments-request@...] On Behalf Of Holger Knublauch
Sent: 26 October 2009 23:56
To: public-rdf-dawg-comments@...
Subject: Re: Missing LET (Assignment) in SPARQL 1.1

Thanks, Lee. I appreciate you taking the time to assemble all this information.

I have made some experiments with the proposal to use sub-selects plus project expressions in a random sample of some of my typical queries. You can see three cases below. Without understanding all implications from a SPARQL engine and algebra point of view, my impression is that the mapping appears to be straight forward, but that it leads to very verbose code. And I did not even try to find the really bad cases.

I am therefore wondering whether LET can be introduced as syntactic sugar similar to some of the new OWL 2 extensions that do not change the semantics but only provide additional mappings from syntax to semantics - this is hopefully easier to manage for the WG?

Thanks,
Holger

----

From the currency conversion example on my blog

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

The original current query is

SELECT ?newLiteral
WHERE {
  LET (?fromCurrency := datatype(?arg1)) .
  LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .
  LET (?fromValue := xsd:double(?arg1)) .
  LET (?newValue := (?fromValue * ?rate)) .
  LET (?newLiteral := smf:cast(?newValue, ?arg2)) .
}

Using nested queries with well-meaning formatting would create something like

SELECT ?newLiteral
WHERE {
              {
                              SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {}
              }
              {
                            SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}
              }
              {
                              SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}
              }
              {
                              SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}
              }
}

Using a single expression would be

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)
WHERE {
}

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

---

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

SELECT ?value
WHERE {
  ?arg2 qud:conversionMultiplier ?M1 .
  ?arg2 qud:conversionOffset ?O1 .
  ?arg3 qud:conversionMultiplier ?M2 .
  ?arg3 qud:conversionOffset ?O2 .
  LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .
}

would become

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)
WHERE {
  ?arg2 qud:conversionMultiplier ?M1 .
  ?arg2 qud:conversionOffset ?O1 .
  ?arg3 qud:conversionMultiplier ?M2 .
  ?arg3 qud:conversionOffset ?O2 .
}

---

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

# Rule1: Collect and replace diamond if possible
CONSTRUCT {
  ?diamond spinbox:replaceWith spinbox:Space .
  ?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
  ?world spinbox:field ?this .
  ?world spinbox:keyDirection ?direction .
  LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .
  ?diamond a boulders:Diamond .
  ?world spinbox:field ?this .
  ?world boulders:diamondsCollected ?oldDiamondsCount .
  LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .
}

This would become

# Rule1: Collect and replace diamond if possible
CONSTRUCT {
  ?diamond spinbox:replaceWith spinbox:Space .
  ?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
  ?world spinbox:field ?this .
  ?world spinbox:keyDirection ?direction .
              {
                              SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)
                              WHERE {
                              }
              }
  ?diamond a boulders:Diamond .
  ?world spinbox:field ?this .
  ?world boulders:diamondsCollected ?oldDiamondsCount .
              {
                            SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)
                              WHERE {
                              }
              }
}

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee

[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

Dear WG,
reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.
Problem 1: How to create new values for CONSTRUCT queries
Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.
Problem 2: Verbosity
We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.
*SELECT* ?x ?r
*WHERE* {
?x ex:property ?y .
*LET* (?helper1 := ex:function1(?y)) .
*LET* (?helper2 := ex:function2(?helper1)) .
*LET* (?r := ex:function3(?helper2)) .
}
The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.
I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.
Thanks
Holger
PS: For a parallel thread on jena-dev (with Andy's response), see
http://tech.groups.yahoo.com/group/jena-dev/message/41903

RE: SPARQL 1.1: project expressions and type errors

2009-10-30T08:33:48Z

> -----Original Message-----
> From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-
> comments-request@...] On Behalf Of Arjohn Kampman
> Sent: 30 October 2009 14:17
> To: public-rdf-dawg-comments@...
> Subject: SPARQL 1.1: project expressions and type errors
>
> Dear WG,
>
> I was wondering how a SPARQL query engine should handle project
> expressions that apply an operator to an incompatible argument. For
> example with the following query with ?p bound to a URI:
>
> select ?s, datatype(?p) as ?q
> where { ?s ?p ?o }
>
> I can think of two ways to handle this:
> 1) Produce a solution with ?q unbound
> 2) Do not produce solutions for these cases
>
> What would be the best way to handle this?

Yes - this is one of issue that needs to be nailed down. We need to go for a consistent error handling approach with, say, aggregates. Another choice - fail the whole query - would seem out of keeping. For me, your choice (2) accords closely with errors in FILTERs.

Andy

>
> Regards,
>
> --
> Arjohn Kampman, Senior Software Engineer
> Aduna - Semantic Power
> www.aduna-software.com

SPARQL 1.1: project expressions and type errors

2009-10-30T07:16:39Z

Dear WG,

I was wondering how a SPARQL query engine should handle project
expressions that apply an operator to an incompatible argument. For
example with the following query with ?p bound to a URI:

select ?s, datatype(?p) as ?q
where { ?s ?p ?o }

I can think of two ways to handle this:
1) Produce a solution with ?q unbound
2) Do not produce solutions for these cases

What would be the best way to handle this?

Regards,

--
Arjohn Kampman, Senior Software Engineer
Aduna - Semantic Power
www.aduna-software.com

RE: Missing LET (Assignment) in SPARQL 1.1

2009-10-30T05:37:06Z

> -----Original Message-----
> From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-
> comments-request@...] On Behalf Of Lee Feigenbaum
> Sent: 27 October 2009 13:28
> To: Holger Knublauch
> Cc: SPARQL Working Group Comments
> Subject: Re: Missing LET (Assignment) in SPARQL 1.1
>
> Holger Knublauch wrote:
> >> This example actually is a good example of one thing that concerns me
> >> about assignment (and remember that my implementation is one that does
> >> support LET expressions): I'm concerned whenever a new SPARQL
> >> construct has an order-dependence. SPARQL is already order-dependent
> >> in cases involving OPTIONAL, but I prefer to keep as much of SPARQL
> >> order-independent as is possible. The above collection of assignments
> >> reads OK because of the order they're presented in, but if you switch
> >> the order around it's not at all clear to me what the proper algebraic
> >> expectations would be.
> >
> > Yes, LET assignments will (have to) be order dependent. And yes, this is
> > a good thing. Sure, it may not be perfect from some theoretical point of
> > view, but without ordering the whole approach would not work, and we
> > would throw out the baby with the bath water. Even the solution with
> > nested sub-selects is order dependent. Giving users the ability to
> > specify the order in a reliable way has not been a problem with any
> > other mainstream computer language, so why should SPARQL be different?
>
> SPARQL is a query language, and my understanding of previous discussions
> is that there is concern that an assignment construct turns a (mostly)
> declarative language into a (somewhat) imperative language, which is (at
> least) a different mind set for users. Again, I'm just repeating what I
> believe I've heard from WG members.
>
> Also, for what it's worth, I don't think that LET need be ordered - the
> Open Anzo implementation is not, and it's (nevertheless) very useful for us.
>
> Also, as currently specified in our Working Drafts, subqueries are not
> order dependent. Andy or Steve will correct me if I'm wrong, I'm sure. :-)

You are right elements in the same group combine with join so they are not order dependent. But the example was wrong - the SELECTs should nested so variables are in-scope.

There are two dimensions of order dependency here:

The first:

...pattern ...
LET (?z := ?y + ?x)
LET (?a := 2+?z )

which, practically, is best defined as order dependent but it happens by nesting:

SELECT ?z , (2+?z AS ?a) # Projecting ?z is necessary
{
SELECT (?y + ?x AS ?z)
{
...pattern...
}
}

and we don't regard that order dependency as a problem.

AS is assignment and the same issues arise in SELECT expressions with AS:

Is this
SELECT (?y + ?x AS ?z) , (2+?z AS ?a)
legal? It can be made to with a left-to-right reading.

The second order dependence issue is around:

{ ?s ?p ?o LET (?o := <x> ) }
{ LET (?o := <x> ) ?s ?p ?o }

This can be made to work (they do the same thing) - a solution is acceptable if an assignment occurs or the variable already has that value. If it's a different value, then the solution is rejected - this is assignment as a necessary constraint. It can be useful in transformations of the algebra for optimization. The other design is that variable names must not have been mentioned beforehand so case 1 is illegal - it can be a static check.

Andy

See also
http://lists.w3.org/Archives/Public/public-rdf-dawg/2009OctDec/0270.html

RE: Missing LET (Assignment) in SPARQL 1.1

2009-10-30T03:32:41Z

I think you need to write it as:

SELECT ?newLiteral

WHERE {

{ SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral)

WHERE

{

SELECT ((?fromValue * ?rate) AS ?newValue)

WHERE

{

SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue)

WHERE

{

SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate)

WHERE {}

}

To put the inner calculations into the scope of the outer ones.

Whereas

SELECT ?newLiteral

WHERE {

{

SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {}

}

{

SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

}

{

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

}

{

SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

}

evaluates each SELECT separately and joins the results together. So for example:

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

------------

| newValue |

============

| |

------------

And does not see ?rate from the other SELECT

Andy

From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-comments-request@...] On Behalf Of Holger Knublauch
Sent: 26 October 2009 23:56
To: public-rdf-dawg-comments@...
Subject: Re: Missing LET (Assignment) in SPARQL 1.1

Thanks, Lee. I appreciate you taking the time to assemble all this information.

I have made some experiments with the proposal to use sub-selects plus project expressions in a random sample of some of my typical queries. You can see three cases below. Without understanding all implications from a SPARQL engine and algebra point of view, my impression is that the mapping appears to be straight forward, but that it leads to very verbose code. And I did not even try to find the really bad cases.

I am therefore wondering whether LET can be introduced as syntactic sugar similar to some of the new OWL 2 extensions that do not change the semantics but only provide additional mappings from syntax to semantics - this is hopefully easier to manage for the WG?

Thanks,

Holger

----

From the currency conversion example on my blog

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

The original current query is

SELECT ?newLiteral

WHERE {

LET (?fromCurrency := datatype(?arg1)) .

LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .

LET (?fromValue := xsd:double(?arg1)) .

LET (?newValue := (?fromValue * ?rate)) .

LET (?newLiteral := smf:cast(?newValue, ?arg2)) .

}

Using nested queries with well-meaning formatting would create something like

SELECT ?newLiteral

WHERE {

{

SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {}

}

{

SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

}

{

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

}

{

SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

}

Using a single expression would be

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)

WHERE {

}

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

---

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

SELECT ?value

WHERE {

?arg2 qud:conversionMultiplier ?M1 .

?arg2 qud:conversionOffset ?O1 .

?arg3 qud:conversionMultiplier ?M2 .

?arg3 qud:conversionOffset ?O2 .

LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .

}

would become

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)

WHERE {

?arg2 qud:conversionMultiplier ?M1 .

?arg2 qud:conversionOffset ?O1 .

?arg3 qud:conversionMultiplier ?M2 .

?arg3 qud:conversionOffset ?O2 .

}

---

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

?diamond spinbox:replaceWith spinbox:Space .

?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

?world spinbox:field ?this .

?world spinbox:keyDirection ?direction .

LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

?diamond a boulders:Diamond .

?world spinbox:field ?this .

?world boulders:diamondsCollected ?oldDiamondsCount .

LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

This would become

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

?diamond spinbox:replaceWith spinbox:Space .

?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

?world spinbox:field ?this .

?world spinbox:keyDirection ?direction .

{

SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)

WHERE {

}

?diamond a boulders:Diamond .

?world spinbox:field ?this .

?world boulders:diamondsCollected ?oldDiamondsCount .

{

SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)

WHERE {

}

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee

[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

Dear WG,

reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.

Problem 1: How to create new values for CONSTRUCT queries

Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.

Problem 2: Verbosity

We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.

*SELECT* ?x ?r

*WHERE* {

?x ex:property ?y .

*LET* (?helper1 := ex:function1(?y)) .

*LET* (?helper2 := ex:function2(?helper1)) .

*LET* (?r := ex:function3(?helper2)) .

}

The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.

I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.

Thanks

Holger

PS: For a parallel thread on jena-dev (with Andy's response), see

http://tech.groups.yahoo.com/group/jena-dev/message/41903

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-28T23:13:49Z

>> Yes, LET assignments will (have to) be order dependent. And yes,
>> this is a good thing. Sure, it may not be perfect from some
>> theoretical point of view, but without ordering the whole approach
>> would not work, and we would throw out the baby with the bath
>> water. Even the solution with nested sub-selects is order
>> dependent. Giving users the ability to specify the order in a
>> reliable way has not been a problem with any other mainstream
>> computer language, so why should SPARQL be different?
>
> SPARQL is a query language, and my understanding of previous
> discussions is that there is concern that an assignment construct
> turns a (mostly) declarative language into a (somewhat) imperative
> language, which is (at least) a different mind set for users. Again,
> I'm just repeating what I believe I've heard from WG members.

I believe that a large portion of SPARQL users (maybe all of the non-
experts) think procedurally when writing queries. They're not thinking
about satisfying clauses, they're thinking about "fetch all the
subjects with this object, then fetch all their names, then filter out
the ones with...".

This is why they're surprised at unexpected results, or unexpected
performance: the algebraic interpretation of their queries is very
different to what they think they've written.

We're all far too close to RDF query languages to remember how non-
implementors think.

My wife is a UX person. In that field it's considered wise to never
think of the user being wrong: if they've come to the incorrect
conclusion, it's very likely because of something you've done or not
done, and it's the software that should change, not the user. It would
be interesting to run a user test of SPARQL; I'm sure we'd learn a
huge amount about the assumptions and pain points of people actually
trying to solve problems with it.

> Also, for what it's worth, I don't think that LET need be ordered -
> the Open Anzo implementation is not, and it's (nevertheless) very
> useful for us.

Holger's usage seems to suggest using LET for intermediate results,
which at least allows efficient reuse of calculated values. That
requires ordering, implicit or not.

If the ordering is implicit, I guarantee that a customer will at some
point ask for a "warning mode" that tells you when a variable is used
before it's assigned to. You can specify behavior all you like, but
that doesn't change how people think.

The small set of Prolog users who are writing SPARQL will be pleased,
of course :)

> Also, as currently specified in our Working Drafts, subqueries are
> not order dependent. Andy or Steve will correct me if I'm wrong, I'm
> sure. :-)

If subqueries can either draw bindings from the enclosing query, or
return them back (surely both being required to make the feature
useful), then strictly controlling the order of their execution would
seem a smart thing to do. Imagine a remote query or subquery which
returns one result if ?x is bound, or a million different ?x bindings
if it's not... it's not always possible to figure out when that'll
happen.

"Sufficiently smart compiler" is not an adequate response. There's a
continuum along which software should gracefully cede control back to
the user.

>> Same with FILTERs - often the query designer knows very well where
>> he wants the FILTERing to take place. Why should an engine be
>> required to do the re-ordering automatically and possibly mess up
>> any performance expectations? But that's a separate topic :)
>
> FILTERs are not order dependent in SPARQL. They are attached
> (conceptually) to either the optional pattern or the group pattern
> in which they occur.

Just to play devil's advocate for a moment:

I think Holger's point is that SPARQL as specified loses a lot of the
information that the query writer has encoded in the query. (He surely
knows that FILTERs are not order dependent: that's what he's lamenting.)

Most people do not think in an order-independent fashion, particularly
when other language constructs such as OPTIONAL *are* ordered (after a
fashion).

I see users interspersing FILTERs throughout their queries all the
time. Very often they do it because they know it's the best way to run
the query. The query language then says "pull out all the FILTERs",
and the implementation then has to decide how to run them... and it
might not have as much information as does the user. (For example,
when the execution of a custom FILTER function is very expensive, and
you need to trick the planner to execute it later or earlier.)

Put another way: I've never *ever* seen a user write something like

SELECT * {
FILTER (?name ...)
?x foaf:name ?name .
...
}

even though it's meaningful SPARQL. Perhaps it shouldn't be meaningful.

This problem gets worse when you consider subqueries, remote queries,
computed properties...

Perhaps order-dependence is actually an intuitive, reasonable default
for a language? Imperative programming language compilers have done a
pretty good job starting with ordered statements, and figuring out
when they can disregard that to get better parallelism. That's an
optimization, not the default.

Devil's advocacy over :)

-R

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-27T10:08:51Z

Hi Danny,

But as Holger points out, LET could likely be order-dependent, but -
it doesn't have to be a drain. CONSTANT, no? The first assignation is
the only assignation to care about.

Whenever the right hand side of the LET contains variables and these variables change their binding, then the LET would need to be recomputed. So just computing them once won't work.

However, there might be a nice compromise here that also preserves the declarative purity of SPARQL: only compute LETs on demand. In this scenario, LET would need to have an unbound variable on the left, and any algebraic operation that refers to that variable (e.g. in a BGP) would first call the function on the right to compute the value. Engines can then easily optimize the performance and avoid duplicate computations as long as the input variables have not changed yet. What would be the problems with this approach?

Holger

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-27T06:27:43Z

Holger Knublauch wrote:

>> This example actually is a good example of one thing that concerns me
>> about assignment (and remember that my implementation is one that does
>> support LET expressions): I'm concerned whenever a new SPARQL
>> construct has an order-dependence. SPARQL is already order-dependent
>> in cases involving OPTIONAL, but I prefer to keep as much of SPARQL
>> order-independent as is possible. The above collection of assignments
>> reads OK because of the order they're presented in, but if you switch
>> the order around it's not at all clear to me what the proper algebraic
>> expectations would be.
>
> Yes, LET assignments will (have to) be order dependent. And yes, this is
> a good thing. Sure, it may not be perfect from some theoretical point of
> view, but without ordering the whole approach would not work, and we
> would throw out the baby with the bath water. Even the solution with
> nested sub-selects is order dependent. Giving users the ability to
> specify the order in a reliable way has not been a problem with any
> other mainstream computer language, so why should SPARQL be different?

SPARQL is a query language, and my understanding of previous discussions
is that there is concern that an assignment construct turns a (mostly)
declarative language into a (somewhat) imperative language, which is (at
least) a different mind set for users. Again, I'm just repeating what I
believe I've heard from WG members.

Also, for what it's worth, I don't think that LET need be ordered - the
Open Anzo implementation is not, and it's (nevertheless) very useful for us.

Also, as currently specified in our Working Drafts, subqueries are not
order dependent. Andy or Steve will correct me if I'm wrong, I'm sure. :-)

> Same with FILTERs - often the query designer knows very well where he
> wants the FILTERing to take place. Why should an engine be required to
> do the re-ordering automatically and possibly mess up any performance
> expectations? But that's a separate topic :)

FILTERs are not order dependent in SPARQL. They are attached
(conceptually) to either the optional pattern or the group pattern in
which they occur.

>> You can project multiple expressions from a single subquery, so I'm
>> not sure that's a concern?
>
> The problem is readability. Imagine a couple of expressions in the same
> SELECT row...

I don't see any problem here...

>>
>> Again, why not put both calculations in a single subquery? I don't
>> know what "*?this*" is, but I'd expect this query to be much easier to
>> read as:
>>
>> CONSTRUCT {
>> ?diamond spinbox:replaceWith spinbox:Space .
>> ?world boulders:diamondsCollected ?newDiamondsCount .
>> }
>> WHERE {
>> SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
>> (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
>> ?world spinbox:field ?this .
>> ?world spinbox:keyDirection ?direction .
>> ?diamond a boulders:Diamond .
>> ?world spinbox:field ?this .
>> ?world boulders:diamondsCollected ?oldDiamondsCount .
>> }
>> }
>
> Because I am using the variable ?diamond in the WHERE clause,

Ah, yes, I had not seen that.

Lee

and not

> just in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a
> boulders:Diamond match, otherwise the whole query does not make sense.
> Compare the original version below
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
> ?diamond spinbox:replaceWith spinbox:Space .
> ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
> ?world spinbox:field *?this* .
> ?world spinbox:keyDirection ?direction .
> *LET* (?diamond := spinbox:getNeighbor(*?this*, ?direction)) .
> ?diamond a boulders:Diamond .
> ?world spinbox:field *?this* .
> ?world boulders:diamondsCollected ?oldDiamondsCount .
> *LET* (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
>
> Regards,
> Holger
>

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-26T21:18:01Z

Perhaps (on a sunny day) I might have something to add to the
discussion, but please forgive me, I missed the point in which
order-dependence arrived. I've not been keeping up and frankly I can't
see how you can sensibly do it to OPTIONALs.

But as Holger points out, LET could likely be order-dependent, but -
it doesn't have to be a drain. CONSTANT, no? The first assignation is
the only assignation to care about.

Scoping of such things shouldn't be a big issue - we have brackets...

2009/10/27 Holger Knublauch <yahoo@...>:

> Same with FILTERs - often the query designer knows very well where he wants
> the FILTERing to take place. Why should an engine be required to do the
> re-ordering automatically and possibly mess up any performance expectations?
> But that's a separate topic :)

Absolutely. A FILTER is a filter, how it's implemented is irrelevant.

> You can project multiple expressions from a single subquery, so I'm not sure
> that's a concern?
>
> The problem is readability. Imagine a couple of expressions in the same
> SELECT row...

I suspect (apologies again for not following relevant discussions)
that's flogging a dead horse - if you need projection, use CONSTRUCT,

> Again, why not put both calculations in a single subquery? I don't know what
> "*?this*" is, but I'd expect this query to be much easier to read as:
>
> CONSTRUCT {
>    ?diamond spinbox:replaceWith spinbox:Space .
>    ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
> SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
> (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
>    ?world spinbox:field ?this .
>    ?world spinbox:keyDirection ?direction .
>    ?diamond a boulders:Diamond .
>    ?world spinbox:field ?this .
>    ?world boulders:diamondsCollected ?oldDiamondsCount .
> }
> }
>
> Because I am using the variable ?diamond in the WHERE clause, and not just
> in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a
> boulders:Diamond match, otherwise the whole query does not make sense.

Surely ?diamond would be there catching something even in version 1.001?

> Compare the original version below
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>   ?diamond spinbox:replaceWith spinbox:Space .
>   ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>   ?world spinbox:field ?this .
>   ?world spinbox:keyDirection ?direction .
>   LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .
>   ?diamond a boulders:Diamond .
>   ?world spinbox:field ?this .
>   ?world boulders:diamondsCollected ?oldDiamondsCount .
>   LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }

The order of things appears to make things very messy, perhaps an
argument for avoiding it.
Maybe LETs should only appear at the start of a clause.

belated Cheers,
Danny.

--
http://danny.ayers.name

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-26T20:35:24Z

This example actually is a good example of one thing that concerns me about assignment (and remember that my implementation is one that does support LET expressions): I'm concerned whenever a new SPARQL construct has an order-dependence. SPARQL is already order-dependent in cases involving OPTIONAL, but I prefer to keep as much of SPARQL order-independent as is possible. The above collection of assignments reads OK because of the order they're presented in, but if you switch the order around it's not at all clear to me what the proper algebraic expectations would be.

Yes, LET assignments will (have to) be order dependent. And yes, this is a good thing. Sure, it may not be perfect from some theoretical point of view, but without ordering the whole approach would not work, and we would throw out the baby with the bath water. Even the solution with nested sub-selects is order dependent. Giving users the ability to specify the order in a reliable way has not been a problem with any other mainstream computer language, so why should SPARQL be different?

Same with FILTERs - often the query designer knows very well where he wants the FILTERing to take place. Why should an engine be required to do the re-ordering automatically and possibly mess up any performance expectations? But that's a separate topic :)

You can project multiple expressions from a single subquery, so I'm not sure that's a concern?

The problem is readability. Imagine a couple of expressions in the same SELECT row...

Again, why not put both calculations in a single subquery? I don't know what "*?this*" is, but I'd expect this query to be much easier to read as:

CONSTRUCT {
   ?diamond spinbox:replaceWith spinbox:Space .
   ?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount) (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
   ?world spinbox:field ?this .
   ?world spinbox:keyDirection ?direction .
   ?diamond a boulders:Diamond .
   ?world spinbox:field ?this .
   ?world boulders:diamondsCollected ?oldDiamondsCount .
}
}

Because I am using the variable ?diamond in the WHERE clause, and not just in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a boulders:Diamond match, otherwise the whole query does not make sense. Compare the original version below

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

?diamond spinbox:replaceWith spinbox:Space .

?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

?world spinbox:field ?this .

?world spinbox:keyDirection ?direction .

LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

?diamond a boulders:Diamond .

?world spinbox:field ?this .

?world boulders:diamondsCollected ?oldDiamondsCount .

LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

Regards,

Holger

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-26T18:31:05Z

Holger Knublauch wrote:

> Thanks, Lee. I appreciate you taking the time to assemble all this
> information.
>
> I have made some experiments with the proposal to use sub-selects plus
> project expressions in a random sample of some of my typical queries.
> You can see three cases below. Without understanding all implications
> from a SPARQL engine and algebra point of view, my impression is that
> the mapping appears to be straight forward, but that it leads to very
> verbose code. And I did not even try to find the really bad cases.
>
> I am therefore wondering whether LET can be introduced as syntactic
> sugar similar to some of the new OWL 2 extensions that do not change the
> semantics but only provide additional mappings from syntax to semantics
> - this is hopefully easier to manage for the WG?

Hi Holger,

In short, the group is free under its charter to consider this as a
purely syntactic sugar extension, but given the process we've been
through to this point, I don't expect it to happen.

That said, I have some comments / questions below in the interests of
discussion :-)

>
> Thanks,
> Holger
>
>
>
> ----
>
> From the currency conversion example on my blog
>
> http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html
>
> The original current query is
>
> *SELECT* ?newLiteral
> *WHERE* {
> *LET* (?fromCurrency := datatype(?arg1)) .
> *LET* (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .
> *LET* (?fromValue := xsd:double(?arg1)) .
> *LET* (?newValue := (?fromValue * ?rate)) .
> *LET* (?newLiteral := smf:cast(?newValue, ?arg2)) .
> }
>
> Using nested queries with well-meaning formatting would create something
> like
>
> *SELECT* ?newLiteral
> *WHERE* {
> {
> *SELECT* (datatype(?arg1) *AS* ?fromCurrency)
> (xsd:double(?arg1) *AS* ?fromValue) *WHERE* {}
> }
> {
>
> *SELECT* (currencies:getRateByCurrencies(?fromCurrency, ?arg2) *AS* ?rate) *WHERE* {}
> }
> {
> *SELECT* ((?fromValue * ?rate) *AS* ?newValue) *WHERE* {}
> }
> {
> *SELECT* (smf:cast(?newValue, ?arg2) *AS* ?newLiteral) *WHERE* {}
> }
> }

This example actually is a good example of one thing that concerns me
about assignment (and remember that my implementation is one that does
support LET expressions): I'm concerned whenever a new SPARQL construct
has an order-dependence. SPARQL is already order-dependent in cases
involving OPTIONAL, but I prefer to keep as much of SPARQL
order-independent as is possible. The above collection of assignments
reads OK because of the order they're presented in, but if you switch
the order around it's not at all clear to me what the proper algebraic
expectations would be.

>
> Using a single expression would be
>
> *SELECT* (smf:cast((xsd:double(?arg1) *
> currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) *AS* ?newLiteral)
> *WHERE* {
> }
>
> The example is a bit atypical because it exclusively uses LETs, and not
> even a triple match. It also uses externally pre-bound variables. But
> still it gives some insights.
>
>
> ---
>
> In the following function body, project expressions work actually ok,
> but keep fingers crossed that you do not have to return multiple of such
> computed values in the SELECT:

You can project multiple expressions from a single subquery, so I'm not
sure that's a concern?

> *SELECT* ?value
> *WHERE* {
> *?arg2* qud:conversionMultiplier ?M1 .
> *?arg2* qud:conversionOffset ?O1 .
> *?arg3* qud:conversionMultiplier ?M2 .
> *?arg3* qud:conversionOffset ?O2 .
> *LET* (?value := ((((*?arg1* * ?M1) + ?O1) - ?O2) / ?M2)) .
> }
>
> would become
>
> *SELECT* (((((*?arg1* * ?M1) + ?O1) - ?O2) / ?M2) *AS* ?value)
> *WHERE* {
> *?arg2* qud:conversionMultiplier ?M1 .
> *?arg2* qud:conversionOffset ?O1 .
> *?arg3* qud:conversionMultiplier ?M2 .
> *?arg3* qud:conversionOffset ?O2 .
> }
>
> ---
>
> Here is an example from the SPIN box computer game, using LET in SPARQL
> rules. This is a very typical use case actually:
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
> ?diamond spinbox:replaceWith spinbox:Space .
> ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
> ?world spinbox:field *?this* .
> ?world spinbox:keyDirection ?direction .
> *LET* (?diamond := spinbox:getNeighbor(*?this*, ?direction)) .
> ?diamond a boulders:Diamond .
> ?world spinbox:field *?this* .
> ?world boulders:diamondsCollected ?oldDiamondsCount .
> *LET* (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
>
> This would become
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
> ?diamond spinbox:replaceWith spinbox:Space .
> ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
> ?world spinbox:field *?this* .
> ?world spinbox:keyDirection ?direction .
> {
> *SELECT* (spinbox:getNeighbor(*?this*, ?direction) *AS* ?diamond)
> *WHERE* {
> }
> }
> ?diamond a boulders:Diamond .
> ?world spinbox:field *?this* .
> ?world boulders:diamondsCollected ?oldDiamondsCount .
> {
> *SELECT* ((?oldDiamondsCount + 1) *AS* ?newDiamondsCount)
> *WHERE* {
> }
> }
> }
>

Again, why not put both calculations in a single subquery? I don't know
what "*?this*" is, but I'd expect this query to be much easier to read as:

CONSTRUCT {
?diamond spinbox:replaceWith spinbox:Space .
?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
(spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
?world spinbox:field ?this .
?world spinbox:keyDirection ?direction .
?diamond a boulders:Diamond .
?world spinbox:field ?this .
?world boulders:diamondsCollected ?oldDiamondsCount .
}
}

Lee

> On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:
>
>> Hi Holger,
>>
>> Thanks for the feedback. Unfortunately, assignment is not on the
>> current Working Group's road map for standardization at this time.
>> Here's how we got to this point:
>>
>> From roughly March through May, the WG considered around 40 potential
>> new features[1] for the SPARQL landscape, including assignment[2]. At
>> the time, we documented two implementations (ARQ and Open Anzo) and
>> the support that you expressed for the feature back in March[3].
>>
>> In going through the features, the WG discussed Assignment in our
>> March, 31 teleconference. You can see the discussion at the time at
>> [4], the results of which was a straw poll result of 7/6/3, indicating
>> some support and several notes of concern.
>>
>> Later in the process, we took a survey of WG member's prioritized
>> preferences of the proposed features. Steve Harris whipped up the
>> Condorcet results of the survey which you can see at [5]. Assignment
>> was in the middle of the pack.
>>
>> At the group's first face-to-face meeting in May, assignment was
>> discussed once more[6], with significant concerns expressed from
>> Garlik and OpenLink Software, strong support from Clark & Parsia, and
>> expressions ranging from indifference to mild support from other WG
>> members (as I read the minutes & recollect the conversation). In the
>> end, the group resolved to accept the list of feature proposals at
>> [7], and Kendall's concerns about the relationship between assignment
>> and projected expressions was addressed by Steve H at [8].
>>
>> Since then, the group has been rechartered with a specific mandate to
>> work on the features decided during the first phase of the group's
>> lifetime[9]. It's my hope & belief that if projected expressions do
>> not end up fulfilling most users' needs, that implementors will extend
>> their SPARQL implementations with assignment or a similar capability,
>> and we will then revisit this in the next round of SPARQL standardization.
>>
>> hope this is helpful,
>> Lee
>>
>>
>> [1] http://www.w3.org/2009/sparql/wiki/Category:Features
>> [2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
>> [3]
>> http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
>> [4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
>> [5] http://plugin.org.uk/misc/votes2.svg
>> [6] http://www.w3.org/2009/sparql/meeting/2009-05-06
>> [7]
>> http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
>> <http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744>
>>
>> [8]
>> http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
>> [9] http://www.w3.org/2009/05/sparql-phase-II-charter
>>
>> Holger Knublauch wrote:
>>> Dear WG,
>>> reading through the drafts (great to have them already!) I am
>>> confused about the future of Assignments (LET keyword in Jena) which
>>> has proven to be absolutely essential for many of our customers
>>> projects. The SPARQL 1.1 working group seems to have converged in
>>> favor of supporting Project expressions and subqueries only, but
>>> these IMHO fail to address the requirements below.
>>> Problem 1: How to create new values for CONSTRUCT queries
>>> Project expressions solve some problems for SELECT queries, but the
>>> major use cases of LET have been in CONSTRUCT queries. I only see
>>> subqueries as a (poor) way of creating new values for use in the
>>> CONSTRUCT clause. Creating a subquery for every LET looks like a very
>>> user unfriendly mechanism to me.
>>> Problem 2: Verbosity
>>> We often work with complex transformations such as string operations
>>> that are best split into multiple steps. Project expressions do not
>>> allow using intermediate variables, such as below and would force
>>> users to chain together very long spaghetti expressions such as
>>> SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this
>>> with some more complex expressions and it quickly becomes completely
>>> unreadable. Also, consider you would want to reuse intermediate
>>> values in multiple places, to avoid duplicate processing.
>>> *SELECT* ?x ?r
>>> *WHERE* {
>>> ?x ex:property ?y .
>>> *LET* (?helper1 := ex:function1(?y)) .
>>> *LET* (?helper2 := ex:function2(?helper1)) .
>>> *LET* (?r := ex:function3(?helper2)) .
>>> }
>>> The LET keyword has solved both problems nicely and in the most
>>> general way, and would make project expressions superfluous.
>>> I would appreciate pointers to the discussions that led to the
>>> decision to not support Assignments at this stage.
>>> Thanks
>>> Holger
>>> PS: For a parallel thread on jena-dev (with Andy's response), see
>>> http://tech.groups.yahoo.com/group/jena-dev/message/41903
>

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-26T17:35:41Z

Hi Holger,

While I agree with Lee's assessment, I also think that you're right in
that simple variable assignment is important.

I'd like to point out that almost every implementation is already
doing this, and I believe that they're using identical (or nearly
identical) syntax. The fact that so many parse with ARQ is a good
start. I also know that some non-ARQ versions (like mine) have just
duplicated the ARQ syntax.

So while I don't think it will make the standard (and am disappointed
that it won't) it may be possible to get a note published about it to
recommend best practice. I believe that the only impediment to this is
that it requires someone with the time to write it up.

Regards,
Paul Gearon

On Mon, Oct 26, 2009 at 7:56 PM, Holger Knublauch <yahoo@...> wrote:

> Thanks, Lee. I appreciate you taking the time to assemble all this
> information.
> I have made some experiments with the proposal to use sub-selects plus
> project expressions in a random sample of some of my typical queries. You
> can see three cases below. Without understanding all implications from a
> SPARQL engine and algebra point of view, my impression is that the mapping
> appears to be straight forward, but that it leads to very verbose code. And
> I did not even try to find the really bad cases.
> I am therefore wondering whether LET can be introduced as syntactic sugar
> similar to some of the new OWL 2 extensions that do not change the semantics
> but only provide additional mappings from syntax to semantics - this is
> hopefully easier to manage for the WG?
> Thanks,
> Holger
>
>
> ----
> From the currency conversion example on my blog
> http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html
>
> The original current query is
> SELECT ?newLiteral
> WHERE {
>   LET (?fromCurrency := datatype(?arg1)) .
>   LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .
>   LET (?fromValue := xsd:double(?arg1)) .
>   LET (?newValue := (?fromValue * ?rate)) .
>   LET (?newLiteral := smf:cast(?newValue, ?arg2)) .
> }
> Using nested queries with well-meaning formatting would create something
> like
> SELECT ?newLiteral
> WHERE {
> {
> SELECT (datatype(?arg1) AS ?fromCurrency)
> (xsd:double(?arg1) AS ?fromValue) WHERE {}
> }
> {
>
> SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}
> }
> {
> SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}
> }
> {
> SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}
> }
> }
> Using a single expression would be
> SELECT (smf:cast((xsd:double(?arg1) *
> currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)
> WHERE {
> }
> The example is a bit atypical because it exclusively uses LETs, and not even
> a triple match. It also uses externally pre-bound variables. But still it
> gives some insights.
>
> ---
> In the following function body, project expressions work actually ok, but
> keep fingers crossed that you do not have to return multiple of such
> computed values in the SELECT:
> SELECT ?value
> WHERE {
>   ?arg2 qud:conversionMultiplier ?M1 .
>   ?arg2 qud:conversionOffset ?O1 .
>   ?arg3 qud:conversionMultiplier ?M2 .
>   ?arg3 qud:conversionOffset ?O2 .
>   LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .
> }
> would become
> SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)
> WHERE {
>   ?arg2 qud:conversionMultiplier ?M1 .
>   ?arg2 qud:conversionOffset ?O1 .
>   ?arg3 qud:conversionMultiplier ?M2 .
>   ?arg3 qud:conversionOffset ?O2 .
> }
> ---
> Here is an example from the SPIN box computer game, using LET in SPARQL
> rules. This is a very typical use case actually:
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>   ?diamond spinbox:replaceWith spinbox:Space .
>   ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>   ?world spinbox:field ?this .
>   ?world spinbox:keyDirection ?direction .
>   LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .
>   ?diamond a boulders:Diamond .
>   ?world spinbox:field ?this .
>   ?world boulders:diamondsCollected ?oldDiamondsCount .
>   LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
> This would become
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>   ?diamond spinbox:replaceWith spinbox:Space .
>   ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>   ?world spinbox:field ?this .
>   ?world spinbox:keyDirection ?direction .
> {
> SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)
> WHERE {
> }
> }
>   ?diamond a boulders:Diamond .
>   ?world spinbox:field ?this .
>   ?world boulders:diamondsCollected ?oldDiamondsCount .
> {
>    SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)
> WHERE {
> }
> }
> }
>
> On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:
>
> Hi Holger,
>
> Thanks for the feedback. Unfortunately, assignment is not on the current
> Working Group's road map for standardization at this time. Here's how we got
> to this point:
>
> From roughly March through May, the WG considered around 40 potential new
> features[1] for the SPARQL landscape, including assignment[2]. At the time,
> we documented two implementations (ARQ and Open Anzo) and the support that
> you expressed for the feature back in March[3].
>
> In going through the features, the WG discussed Assignment in our March, 31
> teleconference. You can see the discussion at the time at [4], the results
> of which was a straw poll result of 7/6/3, indicating some support and
> several notes of concern.
>
> Later in the process, we took a survey of WG member's prioritized
> preferences of the proposed features. Steve Harris whipped up the Condorcet
> results of the survey which you can see at [5]. Assignment was in the middle
> of the pack.
>
> At the group's first face-to-face meeting in May, assignment was discussed
> once more[6], with significant concerns expressed from Garlik and OpenLink
> Software, strong support from Clark & Parsia, and expressions ranging from
> indifference to mild support from other WG members (as I read the minutes &
> recollect the conversation). In the end, the group resolved to accept the
> list of feature proposals at [7], and Kendall's concerns about the
> relationship between assignment and projected expressions was addressed by
> Steve H at [8].
>
> Since then, the group has been rechartered with a specific mandate to work
> on the features decided during the first phase of the group's lifetime[9].
> It's my hope & belief that if projected expressions do not end up fulfilling
> most users' needs, that implementors will extend their SPARQL
> implementations with assignment or a similar capability, and we will then
> revisit this in the next round of SPARQL standardization.
>
> hope this is helpful,
> Lee
>
>
> [1] http://www.w3.org/2009/sparql/wiki/Category:Features
> [2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
> [3]
> http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
> [4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
> [5] http://plugin.org.uk/misc/votes2.svg
> [6] http://www.w3.org/2009/sparql/meeting/2009-05-06
> [7]
> http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
> [8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
> [9] http://www.w3.org/2009/05/sparql-phase-II-charter
>
> Holger Knublauch wrote:
>
> Dear WG,
>
> reading through the drafts (great to have them already!) I am confused about
> the future of Assignments (LET keyword in Jena) which has proven to be
> absolutely essential for many of our customers projects. The SPARQL 1.1
> working group seems to have converged in favor of supporting Project
> expressions and subqueries only, but these IMHO fail to address the
> requirements below.
>
> Problem 1: How to create new values for CONSTRUCT queries
>
> Project expressions solve some problems for SELECT queries, but the major
> use cases of LET have been in CONSTRUCT queries. I only see subqueries as a
> (poor) way of creating new values for use in the CONSTRUCT clause. Creating
> a subquery for every LET looks like a very user unfriendly mechanism to me.
>
> Problem 2: Verbosity
>
> We often work with complex transformations such as string operations that
> are best split into multiple steps. Project expressions do not allow using
> intermediate variables, such as below and would force users to chain
> together very long spaghetti expressions such as SELECT (?x
> ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more
> complex expressions and it quickly becomes completely unreadable. Also,
> consider you would want to reuse intermediate values in multiple places, to
> avoid duplicate processing.
>
> *SELECT* ?x ?r
>
> *WHERE* {
>
> ?x ex:property ?y .
>
> *LET* (?helper1 := ex:function1(?y)) .
>
> *LET* (?helper2 := ex:function2(?helper1)) .
>
> *LET* (?r := ex:function3(?helper2)) .
>
> }
>
> The LET keyword has solved both problems nicely and in the most general way,
> and would make project expressions superfluous.
>
> I would appreciate pointers to the discussions that led to the decision to
> not support Assignments at this stage.
>
> Thanks
>
> Holger
>
> PS: For a parallel thread on jena-dev (with Andy's response), see
>
> http://tech.groups.yahoo.com/group/jena-dev/message/41903
>
>

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-26T16:56:08Z

Thanks, Lee. I appreciate you taking the time to assemble all this information.

Thanks,

Holger

----

From the currency conversion example on my blog

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

The original current query is

SELECT ?newLiteral

WHERE {

LET (?fromCurrency := datatype(?arg1)) .

LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .

LET (?fromValue := xsd:double(?arg1)) .

LET (?newValue := (?fromValue * ?rate)) .

LET (?newLiteral := smf:cast(?newValue, ?arg2)) .

}

Using nested queries with well-meaning formatting would create something like

SELECT ?newLiteral

WHERE {

{

SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {}

}

{

SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

}

{

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

}

{

SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

}

Using a single expression would be

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)

WHERE {

}

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

---

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

SELECT ?value

WHERE {

?arg2 qud:conversionMultiplier ?M1 .

?arg2 qud:conversionOffset ?O1 .

?arg3 qud:conversionMultiplier ?M2 .

?arg3 qud:conversionOffset ?O2 .

LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .

}

would become

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)

WHERE {

?arg2 qud:conversionMultiplier ?M1 .

?arg2 qud:conversionOffset ?O1 .

?arg3 qud:conversionMultiplier ?M2 .

?arg3 qud:conversionOffset ?O2 .

}

---

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

?diamond spinbox:replaceWith spinbox:Space .

?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

?world spinbox:field ?this .

?world spinbox:keyDirection ?direction .

LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

?diamond a boulders:Diamond .

?world spinbox:field ?this .

?world boulders:diamondsCollected ?oldDiamondsCount .

LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

This would become

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

?diamond spinbox:replaceWith spinbox:Space .

?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

?world spinbox:field ?this .

?world spinbox:keyDirection ?direction .

{

SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)

WHERE {

}

?diamond a boulders:Diamond .

?world spinbox:field ?this .

?world boulders:diamondsCollected ?oldDiamondsCount .

{

SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)

WHERE {

}

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee

[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:
Dear WG,
reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.
Problem 1: How to create new values for CONSTRUCT queries
Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.
Problem 2: Verbosity
We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.
*SELECT* ?x ?r
*WHERE* {
?x ex:property ?y .
*LET* (?helper1 := ex:function1(?y)) .
*LET* (?helper2 := ex:function2(?helper1)) .
*LET* (?r := ex:function3(?helper2)) .
}
The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.
I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.
Thanks
Holger
PS: For a parallel thread on jena-dev (with Andy's response), see
http://tech.groups.yahoo.com/group/jena-dev/message/41903

Re: Missing LET (Assignment) in SPARQL 1.1

2009-10-25T20:34:00Z

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current
Working Group's road map for standardization at this time. Here's how we
got to this point:

From roughly March through May, the WG considered around 40 potential
new features[1] for the SPARQL landscape, including assignment[2]. At
the time, we documented two implementations (ARQ and Open Anzo) and the
support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March,
31 teleconference. You can see the discussion at the time at [4], the
results of which was a straw poll result of 7/6/3, indicating some
support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized
preferences of the proposed features. Steve Harris whipped up the
Condorcet results of the survey which you can see at [5]. Assignment was
in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was
discussed once more[6], with significant concerns expressed from Garlik
and OpenLink Software, strong support from Clark & Parsia, and
expressions ranging from indifference to mild support from other WG
members (as I read the minutes & recollect the conversation). In the
end, the group resolved to accept the list of feature proposals at [7],
and Kendall's concerns about the relationship between assignment and
projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to
work on the features decided during the first phase of the group's
lifetime[9]. It's my hope & belief that if projected expressions do not
end up fulfilling most users' needs, that implementors will extend their
SPARQL implementations with assignment or a similar capability, and we
will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee

[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3]
http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7]
http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744

[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

> Dear WG,
>
> reading through the drafts (great to have them already!) I am confused
> about the future of Assignments (LET keyword in Jena) which has proven
> to be absolutely essential for many of our customers projects. The
> SPARQL 1.1 working group seems to have converged in favor of supporting
> Project expressions and subqueries only, but these IMHO fail to address
> the requirements below.
>
>
> Problem 1: How to create new values for CONSTRUCT queries
> Project expressions solve some problems for SELECT queries, but the
> major use cases of LET have been in CONSTRUCT queries. I only see
> subqueries as a (poor) way of creating new values for use in the
> CONSTRUCT clause. Creating a subquery for every LET looks like a very
> user unfriendly mechanism to me.
>
> Problem 2: Verbosity
> We often work with complex transformations such as string operations
> that are best split into multiple steps. Project expressions do not
> allow using intermediate variables, such as below and would force users
> to chain together very long spaghetti expressions such as SELECT (?x
> ex:function3(ex:function2(ex:function1(?y))). Imagine this with some
> more complex expressions and it quickly becomes completely unreadable.
> Also, consider you would want to reuse intermediate values in multiple
> places, to avoid duplicate processing.
>
> *SELECT* ?x ?r
> *WHERE* {
> ?x ex:property ?y .
> *LET* (?helper1 := ex:function1(?y)) .
> *LET* (?helper2 := ex:function2(?helper1)) .
> *LET* (?r := ex:function3(?helper2)) .
> }
>
> The LET keyword has solved both problems nicely and in the most general
> way, and would make project expressions superfluous.
>
> I would appreciate pointers to the discussions that led to the decision
> to not support Assignments at this stage.
>
> Thanks
> Holger
>
> PS: For a parallel thread on jena-dev (with Andy's response), see
>
> http://tech.groups.yahoo.com/group/jena-dev/message/41903