Missing LET (Assignment) in SPARQL 1.1

[Holger Knublauch-3]

Dear WG,

reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.

Problem 1: How to create new values for CONSTRUCT queries

Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.

Problem 2: Verbosity

We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.

SELECT ?x ?r

WHERE {

?x ex:property ?y .

LET (?helper1 := ex:function1(?y)) .

LET (?helper2 := ex:function2(?helper1)) .

LET (?r := ex:function3(?helper2)) .

}

The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.

I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.

Thanks

Holger

PS: For a parallel thread on jena-dev (with Andy's response), see

http://tech.groups.yahoo.com/group/jena-dev/message/41903

[Lee Feigenbaum-2]

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current
Working Group's road map for standardization at this time. Here's how we
got to this point:

 From roughly March through May, the WG considered around 40 potential
new features[1] for the SPARQL landscape, including assignment[2]. At
the time, we documented two implementations (ARQ and Open Anzo) and the
support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March,
31 teleconference. You can see the discussion at the time at [4], the
results of which was a straw poll result of 7/6/3, indicating some
support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized
preferences of the proposed features. Steve Harris whipped up the
Condorcet results of the survey which you can see at [5]. Assignment was
in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was
discussed once more[6], with significant concerns expressed from Garlik
and OpenLink Software, strong support from Clark & Parsia, and
expressions ranging from indifference to mild support from other WG
members (as I read the minutes & recollect the conversation). In the
end, the group resolved to accept the list of feature proposals at [7],
and Kendall's concerns about the relationship between assignment and
projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to
work on the features decided during the first phase of the group's
lifetime[9]. It's my hope & belief that if projected expressions do not
end up fulfilling most users' needs, that implementors will extend their
SPARQL implementations with assignment or a similar capability, and we
will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee


[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3]
http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7]
http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744 

[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

> Dear WG,
>
> reading through the drafts (great to have them already!) I am confused
> about the future of Assignments (LET keyword in Jena) which has proven
> to be absolutely essential for many of our customers projects. The
> SPARQL 1.1 working group seems to have converged in favor of supporting
> Project expressions and subqueries only, but these IMHO fail to address
> the requirements below.
>
>
> Problem 1: How to create new values for CONSTRUCT queries
> Project expressions solve some problems for SELECT queries, but the
> major use cases of LET have been in CONSTRUCT queries. I only see
> subqueries as a (poor) way of creating new values for use in the
> CONSTRUCT clause. Creating a subquery for every LET looks like a very
> user unfriendly mechanism to me.
>
> Problem 2: Verbosity
> We often work with complex transformations such as string operations
> that are best split into multiple steps. Project expressions do not
> allow using intermediate variables, such as below and would force users
> to chain together very long spaghetti expressions such as SELECT (?x
> ex:function3(ex:function2(ex:function1(?y))). Imagine this with some
> more complex expressions and it quickly becomes completely unreadable.
> Also, consider you would want to reuse intermediate values in multiple
> places, to avoid duplicate processing.
>
> *SELECT* ?x ?r
> *WHERE* {
> ?x ex:property ?y .
> *LET* (?helper1 := ex:function1(?y)) .
> *LET* (?helper2 := ex:function2(?helper1)) .
> *LET* (?r := ex:function3(?helper2)) .
> }
>
> The LET keyword has solved both problems nicely and in the most general
> way, and would make project expressions superfluous.
>
> I would appreciate pointers to the discussions that led to the decision
> to not support Assignments at this stage.
>
> Thanks
> Holger
>
> PS: For a parallel thread on jena-dev (with Andy's response), see
>
> http://tech.groups.yahoo.com/group/jena-dev/message/41903

[Holger Knublauch-3]

Thanks, Lee. I appreciate you taking the time to assemble all this information.

I have made some experiments with the proposal to use sub-selects plus project expressions in a random sample of some of my typical queries. You can see three cases below. Without understanding all implications from a SPARQL engine and algebra point of view, my impression is that the mapping appears to be straight forward, but that it leads to very verbose code. And I did not even try to find the really bad cases.

I am therefore wondering whether LET can be introduced as syntactic sugar similar to some of the new OWL 2 extensions that do not change the semantics but only provide additional mappings from syntax to semantics - this is hopefully easier to manage for the WG?

Thanks,

Holger

----

From the currency conversion example on my blog

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

The original current query is

SELECT ?newLiteral

WHERE {

    LET (?fromCurrency := datatype(?arg1)) .

    LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .

    LET (?fromValue := xsd:double(?arg1)) .

    LET (?newValue := (?fromValue * ?rate)) .

    LET (?newLiteral := smf:cast(?newValue, ?arg2)) .

}

Using nested queries with well-meaning formatting would create something like

SELECT ?newLiteral

WHERE {

{

SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {} 

}

{

     SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

}

{

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

}

{

SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

}

}

Using a single expression would be

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)

WHERE {

}

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

---

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

SELECT ?value

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

    LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .

}

would become

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

}

---

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .

    LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

    LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

This would become

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .

{

SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)

WHERE {

}

}

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

{

     SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)

WHERE {

}

}

}

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee


[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

Dear WG,

reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.

Problem 1: How to create new values for CONSTRUCT queries

Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.

Problem 2: Verbosity

We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.

*SELECT* ?x ?r

*WHERE* {

?x ex:property ?y .

*LET* (?helper1 := ex:function1(?y)) .

*LET* (?helper2 := ex:function2(?helper1)) .

*LET* (?r := ex:function3(?helper2)) .

}

The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.

I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.

Thanks

Holger

PS: For a parallel thread on jena-dev (with Andy's response), see

http://tech.groups.yahoo.com/group/jena-dev/message/41903

[Paul Gearon]

Hi Holger,

While I agree with Lee's assessment, I also think that you're right in
that simple variable assignment is important.

I'd like to point out that almost every implementation is already
doing this, and I believe that they're using identical (or nearly
identical) syntax. The fact that so many parse with ARQ is a good
start. I also know that some non-ARQ versions (like mine) have just
duplicated the ARQ syntax.

So while I don't think it will make the standard (and am disappointed
that it won't) it may be possible to get a note published about it to
recommend best practice. I believe that the only impediment to this is
that it requires someone with the time to write it up.

Regards,
Paul Gearon

On Mon, Oct 26, 2009 at 7:56 PM, Holger Knublauch <yahoo@...> wrote:

> Thanks, Lee. I appreciate you taking the time to assemble all this
> information.
> I have made some experiments with the proposal to use sub-selects plus
> project expressions in a random sample of some of my typical queries. You
> can see three cases below. Without understanding all implications from a
> SPARQL engine and algebra point of view, my impression is that the mapping
> appears to be straight forward, but that it leads to very verbose code. And
> I did not even try to find the really bad cases.
> I am therefore wondering whether LET can be introduced as syntactic sugar
> similar to some of the new OWL 2 extensions that do not change the semantics
> but only provide additional mappings from syntax to semantics - this is
> hopefully easier to manage for the WG?
> Thanks,
> Holger
>
>
> ----
> From the currency conversion example on my blog
> http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html
>
> The original current query is
> SELECT ?newLiteral
> WHERE {
>     LET (?fromCurrency := datatype(?arg1)) .
>     LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .
>     LET (?fromValue := xsd:double(?arg1)) .
>     LET (?newValue := (?fromValue * ?rate)) .
>     LET (?newLiteral := smf:cast(?newValue, ?arg2)) .
> }
> Using nested queries with well-meaning formatting would create something
> like
> SELECT ?newLiteral
> WHERE {
> {
> SELECT (datatype(?arg1) AS ?fromCurrency)
> (xsd:double(?arg1) AS ?fromValue) WHERE {}
> }
> {
>
> SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}
> }
> {
> SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}
> }
> {
> SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}
> }
> }
> Using a single expression would be
> SELECT (smf:cast((xsd:double(?arg1) *
> currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)
> WHERE {
> }
> The example is a bit atypical because it exclusively uses LETs, and not even
> a triple match. It also uses externally pre-bound variables. But still it
> gives some insights.
>
> ---
> In the following function body, project expressions work actually ok, but
> keep fingers crossed that you do not have to return multiple of such
> computed values in the SELECT:
> SELECT ?value
> WHERE {
>     ?arg2 qud:conversionMultiplier ?M1 .
>     ?arg2 qud:conversionOffset ?O1 .
>     ?arg3 qud:conversionMultiplier ?M2 .
>     ?arg3 qud:conversionOffset ?O2 .
>     LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .
> }
> would become
> SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)
> WHERE {
>     ?arg2 qud:conversionMultiplier ?M1 .
>     ?arg2 qud:conversionOffset ?O1 .
>     ?arg3 qud:conversionMultiplier ?M2 .
>     ?arg3 qud:conversionOffset ?O2 .
> }
> ---
> Here is an example from the SPIN box computer game, using LET in SPARQL
> rules. This is a very typical use case actually:
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>     ?world spinbox:field ?this .
>     ?world spinbox:keyDirection ?direction .
>     LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field ?this .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
>     LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
> This would become
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>     ?world spinbox:field ?this .
>     ?world spinbox:keyDirection ?direction .
> {
> SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)
> WHERE {
> }
> }
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field ?this .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
> {
>      SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)
> WHERE {
> }
> }
> }
>
> On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:
>
> Hi Holger,
>
> Thanks for the feedback. Unfortunately, assignment is not on the current
> Working Group's road map for standardization at this time. Here's how we got
> to this point:
>
> From roughly March through May, the WG considered around 40 potential new
> features[1] for the SPARQL landscape, including assignment[2]. At the time,
> we documented two implementations (ARQ and Open Anzo) and the support that
> you expressed for the feature back in March[3].
>
> In going through the features, the WG discussed Assignment in our March, 31
> teleconference. You can see the discussion at the time at [4], the results
> of which was a straw poll result of 7/6/3, indicating some support and
> several notes of concern.
>
> Later in the process, we took a survey of WG member's prioritized
> preferences of the proposed features. Steve Harris whipped up the Condorcet
> results of the survey which you can see at [5]. Assignment was in the middle
> of the pack.
>
> At the group's first face-to-face meeting in May, assignment was discussed
> once more[6], with significant concerns expressed from Garlik and OpenLink
> Software, strong support from Clark & Parsia, and expressions ranging from
> indifference to mild support from other WG members (as I read the minutes &
> recollect the conversation). In the end, the group resolved to accept the
> list of feature proposals at [7], and Kendall's concerns about the
> relationship between assignment and projected expressions was addressed by
> Steve H at [8].
>
> Since then, the group has been rechartered with a specific mandate to work
> on the features decided during the first phase of the group's lifetime[9].
> It's my hope & belief that if projected expressions do not end up fulfilling
> most users' needs, that implementors will extend their SPARQL
> implementations with assignment or a similar capability, and we will then
> revisit this in the next round of SPARQL standardization.
>
> hope this is helpful,
> Lee
>
>
> [1] http://www.w3.org/2009/sparql/wiki/Category:Features
> [2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
> [3]
> http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
> [4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
> [5] http://plugin.org.uk/misc/votes2.svg
> [6] http://www.w3.org/2009/sparql/meeting/2009-05-06
> [7]
> http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
> [8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
> [9] http://www.w3.org/2009/05/sparql-phase-II-charter
>
> Holger Knublauch wrote:
>
> Dear WG,
>
> reading through the drafts (great to have them already!) I am confused about
> the future of Assignments (LET keyword in Jena) which has proven to be
> absolutely essential for many of our customers projects. The SPARQL 1.1
> working group seems to have converged in favor of supporting Project
> expressions and subqueries only, but these IMHO fail to address the
> requirements below.
>
> Problem 1: How to create new values for CONSTRUCT queries
>
> Project expressions solve some problems for SELECT queries, but the major
> use cases of LET have been in CONSTRUCT queries. I only see subqueries as a
> (poor) way of creating new values for use in the CONSTRUCT clause. Creating
> a subquery for every LET looks like a very user unfriendly mechanism to me.
>
> Problem 2: Verbosity
>
> We often work with complex transformations such as string operations that
> are best split into multiple steps. Project expressions do not allow using
> intermediate variables, such as below and would force users to chain
> together very long spaghetti expressions such as SELECT (?x
> ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more
> complex expressions and it quickly becomes completely unreadable. Also,
> consider you would want to reuse intermediate values in multiple places, to
> avoid duplicate processing.
>
> *SELECT* ?x ?r
>
> *WHERE* {
>
> ?x ex:property ?y .
>
> *LET* (?helper1 := ex:function1(?y)) .
>
> *LET* (?helper2 := ex:function2(?helper1)) .
>
> *LET* (?r := ex:function3(?helper2)) .
>
> }
>
> The LET keyword has solved both problems nicely and in the most general way,
> and would make project expressions superfluous.
>
> I would appreciate pointers to the discussions that led to the decision to
> not support Assignments at this stage.
>
> Thanks
>
> Holger
>
> PS: For a parallel thread on jena-dev (with Andy's response), see
>
> http://tech.groups.yahoo.com/group/jena-dev/message/41903
>
>

[Lee Feigenbaum-2]

Holger Knublauch wrote:

> Thanks, Lee. I appreciate you taking the time to assemble all this
> information.
>
> I have made some experiments with the proposal to use sub-selects plus
> project expressions in a random sample of some of my typical queries.
> You can see three cases below. Without understanding all implications
> from a SPARQL engine and algebra point of view, my impression is that
> the mapping appears to be straight forward, but that it leads to very
> verbose code. And I did not even try to find the really bad cases.
>
> I am therefore wondering whether LET can be introduced as syntactic
> sugar similar to some of the new OWL 2 extensions that do not change the
> semantics but only provide additional mappings from syntax to semantics
> - this is hopefully easier to manage for the WG?

Hi Holger,

In short, the group is free under its charter to consider this as a
purely syntactic sugar extension, but given the process we've been
through to this point, I don't expect it to happen.

That said, I have some comments / questions below in the interests of
discussion :-)

>
> Thanks,
> Holger
>
>
>
> ----
>
>  From the currency conversion example on my blog
>
> http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html
>
> The original current query is
>
> *SELECT* ?newLiteral
> *WHERE* {
>     *LET* (?fromCurrency := datatype(?arg1)) .
>     *LET* (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .
>     *LET* (?fromValue := xsd:double(?arg1)) .
>     *LET* (?newValue := (?fromValue * ?rate)) .
>     *LET* (?newLiteral := smf:cast(?newValue, ?arg2)) .
> }
>
> Using nested queries with well-meaning formatting would create something
> like
>
> *SELECT* ?newLiteral
> *WHERE* {
> {
> *SELECT* (datatype(?arg1) *AS* ?fromCurrency)
> (xsd:double(?arg1) *AS* ?fromValue) *WHERE* {}
> }
> {
>      
> *SELECT* (currencies:getRateByCurrencies(?fromCurrency, ?arg2) *AS* ?rate) *WHERE* {}
> }
> {
> *SELECT* ((?fromValue * ?rate) *AS* ?newValue) *WHERE* {}
> }
> {
> *SELECT* (smf:cast(?newValue, ?arg2) *AS* ?newLiteral) *WHERE* {}
> }
> }

This example actually is a good example of one thing that concerns me
about assignment (and remember that my implementation is one that does
support LET expressions): I'm concerned whenever a new SPARQL construct
has an order-dependence. SPARQL is already order-dependent in cases
involving OPTIONAL, but I prefer to keep as much of SPARQL
order-independent as is possible. The above collection of assignments
reads OK because of the order they're presented in, but if you switch
the order around it's not at all clear to me what the proper algebraic
expectations would be.

>
> Using a single expression would be
>
> *SELECT* (smf:cast((xsd:double(?arg1) *
> currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) *AS* ?newLiteral)
> *WHERE* {
> }
>
> The example is a bit atypical because it exclusively uses LETs, and not
> even a triple match. It also uses externally pre-bound variables. But
> still it gives some insights.
>
>
> ---
>
> In the following function body, project expressions work actually ok,
> but keep fingers crossed that you do not have to return multiple of such
> computed values in the SELECT:

You can project multiple expressions from a single subquery, so I'm not
sure that's a concern?

> *SELECT* ?value
> *WHERE* {
>     *?arg2* qud:conversionMultiplier ?M1 .
>     *?arg2* qud:conversionOffset ?O1 .
>     *?arg3* qud:conversionMultiplier ?M2 .
>     *?arg3* qud:conversionOffset ?O2 .
>     *LET* (?value := ((((*?arg1* * ?M1) + ?O1) - ?O2) / ?M2)) .
> }
>
> would become
>
> *SELECT* (((((*?arg1* * ?M1) + ?O1) - ?O2) / ?M2) *AS* ?value)
> *WHERE* {
>     *?arg2* qud:conversionMultiplier ?M1 .
>     *?arg2* qud:conversionOffset ?O1 .
>     *?arg3* qud:conversionMultiplier ?M2 .
>     *?arg3* qud:conversionOffset ?O2 .
> }
>
> ---
>
> Here is an example from the SPIN box computer game, using LET in SPARQL
> rules. This is a very typical use case actually:
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
>     ?world spinbox:field *?this* .
>     ?world spinbox:keyDirection ?direction .
>     *LET* (?diamond := spinbox:getNeighbor(*?this*, ?direction)) .
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field *?this* .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
>     *LET* (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
>
> This would become
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
>     ?world spinbox:field *?this* .
>     ?world spinbox:keyDirection ?direction .
> {
> *SELECT* (spinbox:getNeighbor(*?this*, ?direction) *AS* ?diamond)
> *WHERE* {
> }
> }
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field *?this* .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
> {
>      *SELECT* ((?oldDiamondsCount + 1) *AS* ?newDiamondsCount)
> *WHERE* {
> }
> }
> }
>

Again, why not put both calculations in a single subquery? I don't know
what "*?this*" is, but I'd expect this query to be much easier to read as:

CONSTRUCT {
     ?diamond spinbox:replaceWith spinbox:Space .
     ?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
   SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
(spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
     ?world spinbox:field ?this .
     ?world spinbox:keyDirection ?direction .
     ?diamond a boulders:Diamond .
     ?world spinbox:field ?this .
     ?world boulders:diamondsCollected ?oldDiamondsCount .
   }
}

Lee

> On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:
>
>> Hi Holger,
>>
>> Thanks for the feedback. Unfortunately, assignment is not on the
>> current Working Group's road map for standardization at this time.
>> Here's how we got to this point:
>>
>> From roughly March through May, the WG considered around 40 potential
>> new features[1] for the SPARQL landscape, including assignment[2]. At
>> the time, we documented two implementations (ARQ and Open Anzo) and
>> the support that you expressed for the feature back in March[3].
>>
>> In going through the features, the WG discussed Assignment in our
>> March, 31 teleconference. You can see the discussion at the time at
>> [4], the results of which was a straw poll result of 7/6/3, indicating
>> some support and several notes of concern.
>>
>> Later in the process, we took a survey of WG member's prioritized
>> preferences of the proposed features. Steve Harris whipped up the
>> Condorcet results of the survey which you can see at [5]. Assignment
>> was in the middle of the pack.
>>
>> At the group's first face-to-face meeting in May, assignment was
>> discussed once more[6], with significant concerns expressed from
>> Garlik and OpenLink Software, strong support from Clark & Parsia, and
>> expressions ranging from indifference to mild support from other WG
>> members (as I read the minutes & recollect the conversation). In the
>> end, the group resolved to accept the list of feature proposals at
>> [7], and Kendall's concerns about the relationship between assignment
>> and projected expressions was addressed by Steve H at [8].
>>
>> Since then, the group has been rechartered with a specific mandate to
>> work on the features decided during the first phase of the group's
>> lifetime[9]. It's my hope & belief that if projected expressions do
>> not end up fulfilling most users' needs, that implementors will extend
>> their SPARQL implementations with assignment or a similar capability,
>> and we will then revisit this in the next round of SPARQL standardization.
>>
>> hope this is helpful,
>> Lee
>>
>>
>> [1] http://www.w3.org/2009/sparql/wiki/Category:Features
>> [2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
>> [3]
>> http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
>> [4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
>> [5] http://plugin.org.uk/misc/votes2.svg
>> [6] http://www.w3.org/2009/sparql/meeting/2009-05-06
>> [7]
>> http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744 
>> <http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744>
>>
>> [8]
>> http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
>> [9] http://www.w3.org/2009/05/sparql-phase-II-charter
>>
>> Holger Knublauch wrote:
>>> Dear WG,
>>> reading through the drafts (great to have them already!) I am
>>> confused about the future of Assignments (LET keyword in Jena) which
>>> has proven to be absolutely essential for many of our customers
>>> projects. The SPARQL 1.1 working group seems to have converged in
>>> favor of supporting Project expressions and subqueries only, but
>>> these IMHO fail to address the requirements below.
>>> Problem 1: How to create new values for CONSTRUCT queries
>>> Project expressions solve some problems for SELECT queries, but the
>>> major use cases of LET have been in CONSTRUCT queries. I only see
>>> subqueries as a (poor) way of creating new values for use in the
>>> CONSTRUCT clause. Creating a subquery for every LET looks like a very
>>> user unfriendly mechanism to me.
>>> Problem 2: Verbosity
>>> We often work with complex transformations such as string operations
>>> that are best split into multiple steps. Project expressions do not
>>> allow using intermediate variables, such as below and would force
>>> users to chain together very long spaghetti expressions such as
>>> SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this
>>> with some more complex expressions and it quickly becomes completely
>>> unreadable. Also, consider you would want to reuse intermediate
>>> values in multiple places, to avoid duplicate processing.
>>> *SELECT* ?x ?r
>>> *WHERE* {
>>> ?x ex:property ?y .
>>> *LET* (?helper1 := ex:function1(?y)) .
>>> *LET* (?helper2 := ex:function2(?helper1)) .
>>> *LET* (?r := ex:function3(?helper2)) .
>>> }
>>> The LET keyword has solved both problems nicely and in the most
>>> general way, and would make project expressions superfluous.
>>> I would appreciate pointers to the discussions that led to the
>>> decision to not support Assignments at this stage.
>>> Thanks
>>> Holger
>>> PS: For a parallel thread on jena-dev (with Andy's response), see
>>> http://tech.groups.yahoo.com/group/jena-dev/message/41903
>

[Holger Knublauch-3]

This example actually is a good example of one thing that concerns me about assignment (and remember that my implementation is one that does support LET expressions): I'm concerned whenever a new SPARQL construct has an order-dependence. SPARQL is already order-dependent in cases involving OPTIONAL, but I prefer to keep as much of SPARQL order-independent as is possible. The above collection of assignments reads OK because of the order they're presented in, but if you switch the order around it's not at all clear to me what the proper algebraic expectations would be.

Yes, LET assignments will (have to) be order dependent. And yes, this is a good thing. Sure, it may not be perfect from some theoretical point of view, but without ordering the whole approach would not work, and we would throw out the baby with the bath water. Even the solution with nested sub-selects is order dependent. Giving users the ability to specify the order in a reliable way has not been a problem with any other mainstream computer language, so why should SPARQL be different?

Same with FILTERs - often the query designer knows very well where he wants the FILTERing to take place. Why should an engine be required to do the re-ordering automatically and possibly mess up any performance expectations? But that's a separate topic :)

You can project multiple expressions from a single subquery, so I'm not sure that's a concern?

The problem is readability. Imagine a couple of expressions in the same SELECT row...

Again, why not put both calculations in a single subquery? I don't know what "*?this*" is, but I'd expect this query to be much easier to read as:

CONSTRUCT {
   ?diamond spinbox:replaceWith spinbox:Space .
   ?world boulders:diamondsCollected ?newDiamondsCount .
}
WHERE {
 SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount) (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
   ?world spinbox:field ?this .
   ?world spinbox:keyDirection ?direction .
   ?diamond a boulders:Diamond .
   ?world spinbox:field ?this .
   ?world boulders:diamondsCollected ?oldDiamondsCount .
 }
}

Because I am using the variable ?diamond in the WHERE clause, and not just in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a boulders:Diamond match, otherwise the whole query does not make sense. Compare the original version below

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .

    LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

    LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

Regards,

Holger

[Danny Ayers]

Perhaps (on a sunny day) I might have something to add to the
discussion, but please forgive me, I missed the point in which
order-dependence arrived. I've not been keeping up and frankly I can't
see how you can sensibly do it to OPTIONALs.

But as Holger points out, LET could likely be order-dependent, but -
it doesn't have to be a drain. CONSTANT, no? The first assignation is
the only assignation to care about.

Scoping of such things shouldn't be a big issue - we have brackets...

2009/10/27 Holger Knublauch <yahoo@...>:

> Same with FILTERs - often the query designer knows very well where he wants
> the FILTERing to take place. Why should an engine be required to do the
> re-ordering automatically and possibly mess up any performance expectations?
> But that's a separate topic :)

Absolutely. A FILTER is a filter, how it's implemented is irrelevant.

> You can project multiple expressions from a single subquery, so I'm not sure
> that's a concern?
>
> The problem is readability. Imagine a couple of expressions in the same
> SELECT row...

I suspect (apologies again for not following relevant discussions)
that's flogging a dead horse - if you need projection, use CONSTRUCT,

> Again, why not put both calculations in a single subquery? I don't know what
> "*?this*" is, but I'd expect this query to be much easier to read as:
>
> CONSTRUCT {
>    ?diamond spinbox:replaceWith spinbox:Space .
>    ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>  SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
> (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
>    ?world spinbox:field ?this .
>    ?world spinbox:keyDirection ?direction .
>    ?diamond a boulders:Diamond .
>    ?world spinbox:field ?this .
>    ?world boulders:diamondsCollected ?oldDiamondsCount .
>  }
> }
>
> Because I am using the variable ?diamond in the WHERE clause, and not just
> in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a
> boulders:Diamond match, otherwise the whole query does not make sense.

Surely ?diamond would be there catching something even in version 1.001?

> Compare the original version below
> # Rule1: Collect and replace diamond if possible
> CONSTRUCT {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> WHERE {
>     ?world spinbox:field ?this .
>     ?world spinbox:keyDirection ?direction .
>     LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field ?this .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
>     LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }

The order of things appears to make things very messy, perhaps an
argument for avoiding it.
Maybe LETs should only appear at the start of a clause.

belated Cheers,
Danny.


--
http://danny.ayers.name

[Lee Feigenbaum-2]

Holger Knublauch wrote:

>> This example actually is a good example of one thing that concerns me
>> about assignment (and remember that my implementation is one that does
>> support LET expressions): I'm concerned whenever a new SPARQL
>> construct has an order-dependence. SPARQL is already order-dependent
>> in cases involving OPTIONAL, but I prefer to keep as much of SPARQL
>> order-independent as is possible. The above collection of assignments
>> reads OK because of the order they're presented in, but if you switch
>> the order around it's not at all clear to me what the proper algebraic
>> expectations would be.
>
> Yes, LET assignments will (have to) be order dependent. And yes, this is
> a good thing. Sure, it may not be perfect from some theoretical point of
> view, but without ordering the whole approach would not work, and we
> would throw out the baby with the bath water. Even the solution with
> nested sub-selects is order dependent. Giving users the ability to
> specify the order in a reliable way has not been a problem with any
> other mainstream computer language, so why should SPARQL be different?

SPARQL is a query language, and my understanding of previous discussions
is that there is concern that an assignment construct turns a (mostly)
declarative language into a (somewhat) imperative language, which is (at
least) a different mind set for users. Again, I'm just repeating what I
believe I've heard from WG members.

Also, for what it's worth, I don't think that LET need be ordered - the
Open Anzo implementation is not, and it's (nevertheless) very useful for us.

Also, as currently specified in our Working Drafts, subqueries are not
order dependent. Andy or Steve will correct me if I'm wrong, I'm sure. :-)

> Same with FILTERs - often the query designer knows very well where he
> wants the FILTERing to take place. Why should an engine be required to
> do the re-ordering automatically and possibly mess up any performance
> expectations? But that's a separate topic :)

FILTERs are not order dependent in SPARQL. They are attached
(conceptually) to either the optional pattern or the group pattern in
which they occur.

>> You can project multiple expressions from a single subquery, so I'm
>> not sure that's a concern?
>
> The problem is readability. Imagine a couple of expressions in the same
> SELECT row...

I don't see any problem here...

>>
>> Again, why not put both calculations in a single subquery? I don't
>> know what "*?this*" is, but I'd expect this query to be much easier to
>> read as:
>>
>> CONSTRUCT {
>>    ?diamond spinbox:replaceWith spinbox:Space .
>>    ?world boulders:diamondsCollected ?newDiamondsCount .
>> }
>> WHERE {
>>  SELECT ?world (?oldDiamondCount + 1 AS ?newDiamongCount)
>> (spinbox:getNeighbor(?this, ?direction) AS ?diamond) {
>>    ?world spinbox:field ?this .
>>    ?world spinbox:keyDirection ?direction .
>>    ?diamond a boulders:Diamond .
>>    ?world spinbox:field ?this .
>>    ?world boulders:diamondsCollected ?oldDiamondsCount .
>>  }
>> }
>
> Because I am using the variable ?diamond in the WHERE clause,

Ah, yes, I had not seen that.

Lee

and not

> just in the CONSTRUCT. ?diamond will have to be bound in the ?diamond a
> boulders:Diamond match, otherwise the whole query does not make sense.
> Compare the original version below
>
> # Rule1: Collect and replace diamond if possible
> *CONSTRUCT* {
>     ?diamond spinbox:replaceWith spinbox:Space .
>     ?world boulders:diamondsCollected ?newDiamondsCount .
> }
> *WHERE* {
>     ?world spinbox:field *?this* .
>     ?world spinbox:keyDirection ?direction .
>     *LET* (?diamond := spinbox:getNeighbor(*?this*, ?direction)) .
>     ?diamond a boulders:Diamond .
>     ?world spinbox:field *?this* .
>     ?world boulders:diamondsCollected ?oldDiamondsCount .
>     *LET* (?newDiamondsCount := (?oldDiamondsCount + 1)) .
> }
>
> Regards,
> Holger
>

[Holger Knublauch-3]

Hi Danny,

But as Holger points out, LET could likely be order-dependent, but -
it doesn't have to be a drain. CONSTANT, no? The first assignation is
the only assignation to care about.

Whenever the right hand side of the LET contains variables and these variables change their binding, then the LET would need to be recomputed. So just computing them once won't work.

However, there might be a nice compromise here that also preserves the declarative purity of SPARQL: only compute LETs on demand. In this scenario, LET would need to have an unbound variable on the left, and any algebraic operation that refers to that variable (e.g. in a BGP) would first call the function on the right to compute the value. Engines can then easily optimize the performance and avoid duplicate computations as long as the input variables have not changed yet. What would be the problems with this approach?

Holger

[Richard Newman-8]

>> Yes, LET assignments will (have to) be order dependent. And yes,  
>> this is a good thing. Sure, it may not be perfect from some  
>> theoretical point of view, but without ordering the whole approach  
>> would not work, and we would throw out the baby with the bath  
>> water. Even the solution with nested sub-selects is order  
>> dependent. Giving users the ability to specify the order in a  
>> reliable way has not been a problem with any other mainstream  
>> computer language, so why should SPARQL be different?
>
> SPARQL is a query language, and my understanding of previous  
> discussions is that there is concern that an assignment construct  
> turns a (mostly) declarative language into a (somewhat) imperative  
> language, which is (at least) a different mind set for users. Again,  
> I'm just repeating what I believe I've heard from WG members.

I believe that a large portion of SPARQL users (maybe all of the non-
experts) think procedurally when writing queries. They're not thinking  
about satisfying clauses, they're thinking about "fetch all the  
subjects with this object, then fetch all their names, then filter out  
the ones with...".

This is why they're surprised at unexpected results, or unexpected  
performance: the algebraic interpretation of their queries is very  
different to what they think they've written.

We're all far too close to RDF query languages to remember how non-
implementors think.

My wife is a UX person. In that field it's considered wise to never  
think of the user being wrong: if they've come to the incorrect  
conclusion, it's very likely because of something you've done or not  
done, and it's the software that should change, not the user. It would  
be interesting to run a user test of SPARQL; I'm sure we'd learn a  
huge amount about the assumptions and pain points of people actually  
trying to solve problems with it.


> Also, for what it's worth, I don't think that LET need be ordered -  
> the Open Anzo implementation is not, and it's (nevertheless) very  
> useful for us.

Holger's usage seems to suggest using LET for intermediate results,  
which at least allows efficient reuse of calculated values. That  
requires ordering, implicit or not.

If the ordering is implicit, I guarantee that a customer will at some  
point ask for a "warning mode" that tells you when a variable is used  
before it's assigned to. You can specify behavior all you like, but  
that doesn't change how people think.

The small set of Prolog users who are writing SPARQL will be pleased,  
of course :)


> Also, as currently specified in our Working Drafts, subqueries are  
> not order dependent. Andy or Steve will correct me if I'm wrong, I'm  
> sure. :-)

If subqueries can either draw bindings from the enclosing query, or  
return them back (surely both being required to make the feature  
useful), then strictly controlling the order of their execution would  
seem a smart thing to do. Imagine a remote query or subquery which  
returns one result if ?x is bound, or a million different ?x bindings  
if it's not... it's not always possible to figure out when that'll  
happen.

"Sufficiently smart compiler" is not an adequate response. There's a  
continuum along which software should gracefully cede control back to  
the user.


>> Same with FILTERs - often the query designer knows very well where  
>> he wants the FILTERing to take place. Why should an engine be  
>> required to do the re-ordering automatically and possibly mess up  
>> any performance expectations? But that's a separate topic :)
>
> FILTERs are not order dependent in SPARQL. They are attached  
> (conceptually) to either the optional pattern or the group pattern  
> in which they occur.

Just to play devil's advocate for a moment:

I think Holger's point is that SPARQL as specified loses a lot of the  
information that the query writer has encoded in the query. (He surely  
knows that FILTERs are not order dependent: that's what he's lamenting.)

Most people do not think in an order-independent fashion, particularly  
when other language constructs such as OPTIONAL *are* ordered (after a  
fashion).

I see users interspersing FILTERs throughout their queries all the  
time. Very often they do it because they know it's the best way to run  
the query. The query language then says "pull out all the FILTERs",  
and the implementation then has to decide how to run them... and it  
might not have as much information as does the user. (For example,  
when the execution of a custom FILTER function is very expensive, and  
you need to trick the planner to execute it later or earlier.)

Put another way: I've never *ever* seen a user write something like

   SELECT * {
     FILTER (?name ...)
     ?x foaf:name ?name .
     ...
   }

even though it's meaningful SPARQL. Perhaps it shouldn't be meaningful.

This problem gets worse when you consider subqueries, remote queries,  
computed properties...

Perhaps order-dependence is actually an intuitive, reasonable default  
for a language? Imperative programming language compilers have done a  
pretty good job starting with ordered statements, and figuring out  
when they can disregard that to get better parallelism. That's an  
optimization, not the default.

Devil's advocacy over :)

-R

[Seaborne, Andy]

Some parts of this message have been removed. Learn more about Nabble's security policy.

I think you need to write it as:

 

SELECT ?newLiteral

WHERE {

      { SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral)

        WHERE

        {

          SELECT ((?fromValue * ?rate) AS ?newValue)

          WHERE

          {

            SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue)

            WHERE

            {

               SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate)

               WHERE {}

            }

        }

    }

}

 

To put the inner calculations into the scope of the outer ones.

 

Whereas

 

SELECT ?newLiteral

WHERE {

                {

                                SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {} 

                }

                {

                                SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

                }

                {

                                SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

                }

                {

                                SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

                }

}

 

evaluates each SELECT separately and joins the results together.  So for example:

 

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

 

Is

 

------------

| newValue |

============

|          |

------------

 

And does not see ?rate from the other SELECT

 

                Andy

 

From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-comments-request@...] On Behalf Of Holger Knublauch
Sent: 26 October 2009 23:56
To: public-rdf-dawg-comments@...
Subject: Re: Missing LET (Assignment) in SPARQL 1.1

 

Thanks, Lee. I appreciate you taking the time to assemble all this information.

 

I have made some experiments with the proposal to use sub-selects plus project expressions in a random sample of some of my typical queries. You can see three cases below. Without understanding all implications from a SPARQL engine and algebra point of view, my impression is that the mapping appears to be straight forward, but that it leads to very verbose code. And I did not even try to find the really bad cases.

 

I am therefore wondering whether LET can be introduced as syntactic sugar similar to some of the new OWL 2 extensions that do not change the semantics but only provide additional mappings from syntax to semantics - this is hopefully easier to manage for the WG?

 

Thanks,

Holger

 

 

 

----

 

From the currency conversion example on my blog

 

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

 

The original current query is

 

SELECT ?newLiteral

WHERE {

    LET (?fromCurrency := datatype(?arg1)) .

    LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .

    LET (?fromValue := xsd:double(?arg1)) .

    LET (?newValue := (?fromValue * ?rate)) .

    LET (?newLiteral := smf:cast(?newValue, ?arg2)) .

}

 

Using nested queries with well-meaning formatting would create something like

 

SELECT ?newLiteral

WHERE {

              {

                              SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {} 

              }

              {

                              SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

              }

              {

                              SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

              }

              {

                              SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

              }

}

 

Using a single expression would be

 

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)

WHERE {

}

 

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

 

 

---

 

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

 

SELECT ?value

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

    LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .

}

 

would become

 

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

}

 

---

 

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

 

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .

    LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

    LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

 

This would become

 

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .         

              {

                              SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)

                              WHERE {

                              }

              }

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

              {

                              SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)

                              WHERE {

                              }

              }

}

 

 

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee


[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

Dear WG,

reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.

Problem 1: How to create new values for CONSTRUCT queries

Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.

Problem 2: Verbosity

We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.

*SELECT* ?x ?r

*WHERE* {

?x ex:property ?y .

*LET* (?helper1 := ex:function1(?y)) .

*LET* (?helper2 := ex:function2(?helper1)) .

*LET* (?r := ex:function3(?helper2)) .

}

The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.

I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.

Thanks

Holger

PS: For a parallel thread on jena-dev (with Andy's response), see

http://tech.groups.yahoo.com/group/jena-dev/message/41903

 

[Seaborne, Andy]

> -----Original Message-----
> From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-
> comments-request@...] On Behalf Of Lee Feigenbaum
> Sent: 27 October 2009 13:28
> To: Holger Knublauch
> Cc: SPARQL Working Group Comments
> Subject: Re: Missing LET (Assignment) in SPARQL 1.1
>
> Holger Knublauch wrote:
> >> This example actually is a good example of one thing that concerns me
> >> about assignment (and remember that my implementation is one that does
> >> support LET expressions): I'm concerned whenever a new SPARQL
> >> construct has an order-dependence. SPARQL is already order-dependent
> >> in cases involving OPTIONAL, but I prefer to keep as much of SPARQL
> >> order-independent as is possible. The above collection of assignments
> >> reads OK because of the order they're presented in, but if you switch
> >> the order around it's not at all clear to me what the proper algebraic
> >> expectations would be.
> >
> > Yes, LET assignments will (have to) be order dependent. And yes, this is
> > a good thing. Sure, it may not be perfect from some theoretical point of
> > view, but without ordering the whole approach would not work, and we
> > would throw out the baby with the bath water. Even the solution with
> > nested sub-selects is order dependent. Giving users the ability to
> > specify the order in a reliable way has not been a problem with any
> > other mainstream computer language, so why should SPARQL be different?
>
> SPARQL is a query language, and my understanding of previous discussions
> is that there is concern that an assignment construct turns a (mostly)
> declarative language into a (somewhat) imperative language, which is (at
> least) a different mind set for users. Again, I'm just repeating what I
> believe I've heard from WG members.
>
> Also, for what it's worth, I don't think that LET need be ordered - the
> Open Anzo implementation is not, and it's (nevertheless) very useful for us.
>
> Also, as currently specified in our Working Drafts, subqueries are not
> order dependent. Andy or Steve will correct me if I'm wrong, I'm sure. :-)

You are right elements in the same group combine with join so they are not order dependent.  But the example was wrong - the SELECTs should nested so variables are in-scope.

There are two dimensions of order dependency here:

The first:

...pattern ...
LET (?z := ?y + ?x)
LET (?a := 2+?z )

which, practically, is best defined as order dependent but it happens by nesting:

SELECT ?z , (2+?z AS ?a) # Projecting ?z is necessary
{
   SELECT (?y + ?x AS ?z)
  {
     ...pattern...
  }
}

and we don't regard that order dependency as a problem.

AS is assignment and the same issues arise in SELECT expressions with AS:

Is this
  SELECT (?y + ?x AS ?z) , (2+?z AS ?a)
legal? It can be made to with a left-to-right reading.


The second order dependence issue is around:

{ ?s ?p ?o LET (?o := <x> ) }
{ LET (?o := <x> ) ?s ?p ?o }

This can be made to work (they do the same thing) - a solution is acceptable if an assignment occurs or the variable already has that value.  If it's a different value, then the solution is rejected - this is assignment as a necessary constraint.  It can be useful in transformations of the algebra for optimization.  The other design is that variable names must not have been mentioned beforehand so case 1 is illegal - it can be a static check.

        Andy

See also
http://lists.w3.org/Archives/Public/public-rdf-dawg/2009OctDec/0270.html

[Holger Knublauch-3]

Thanks for pointing this out, Andy. Its good to hear there is already some control over the order of execution this way, and this mechanism would make the job easier for a LET macro.

But this also dramatically highlights how unattractive the notation below would be for real world usage. Not only is it way too verbose, but it also requires a mind set that the average user does not have - at least my intuition was misleading. Related working groups (in particular OWL) have made similar mistakes in the past, trying to design a language that is theoretically sound and clear, but unintuitive to use as it does not align with the traditional imperative/object-oriented way of thinking. I have a long list of customer stories (both for SPARQL and OWL) where people were disappointed because the language did not produce the results they expected. In particular the lack of reliable ordering of FILTER clauses often leads to very very slow queries and people believe the whole semantic web stack is not ready for prime time yet.

Holger

On Oct 30, 2009, at 3:32 AM, Seaborne, Andy wrote:

I think you need to write it as:

 

SELECT ?newLiteral

WHERE {

      { SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral)

        WHERE

        {

          SELECT ((?fromValue * ?rate) AS ?newValue)

          WHERE

          {

            SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue)

            WHERE

            {

               SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate)

               WHERE {}

            }

        }

    }

}

 

To put the inner calculations into the scope of the outer ones.

 

Whereas

 

SELECT ?newLiteral

WHERE {

                {

                                SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {} 

                }

                {

                                SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

                }

                {

                                SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

                }

                {

                                SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

                }

}

 

evaluates each SELECT separately and joins the results together.  So for example:

 

SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

 

Is

 

------------

| newValue |

============

|          |

------------

 

And does not see ?rate from the other SELECT

 

                Andy

 

From: public-rdf-dawg-comments-request@... [mailto:public-rdf-dawg-comments-request@...] On Behalf Of Holger Knublauch
Sent: 26 October 2009 23:56
To: public-rdf-dawg-comments@...
Subject: Re: Missing LET (Assignment) in SPARQL 1.1

 

Thanks, Lee. I appreciate you taking the time to assemble all this information.

 

I have made some experiments with the proposal to use sub-selects plus project expressions in a random sample of some of my typical queries. You can see three cases below. Without understanding all implications from a SPARQL engine and algebra point of view, my impression is that the mapping appears to be straight forward, but that it leads to very verbose code. And I did not even try to find the really bad cases.

 

I am therefore wondering whether LET can be introduced as syntactic sugar similar to some of the new OWL 2 extensions that do not change the semantics but only provide additional mappings from syntax to semantics - this is hopefully easier to manage for the WG?

 

Thanks,

Holger

 

 

 

----

 

From the currency conversion example on my blog

 

http://composing-the-semantic-web.blogspot.com/2009/09/currency-conversion-with-units-ontology.html

 

The original current query is

 

SELECT ?newLiteral

WHERE {

    LET (?fromCurrency := datatype(?arg1)) .

    LET (?rate := currencies:getRateByCurrencies(?fromCurrency, ?arg2)) .

    LET (?fromValue := xsd:double(?arg1)) .

    LET (?newValue := (?fromValue * ?rate)) .

    LET (?newLiteral := smf:cast(?newValue, ?arg2)) .

}

 

Using nested queries with well-meaning formatting would create something like

 

SELECT ?newLiteral

WHERE {

              {

                              SELECT (datatype(?arg1) AS ?fromCurrency) (xsd:double(?arg1) AS ?fromValue) WHERE {} 

              }

              {

                              SELECT (currencies:getRateByCurrencies(?fromCurrency, ?arg2) AS ?rate) WHERE {}

              }

              {

                              SELECT ((?fromValue * ?rate) AS ?newValue) WHERE {}

              }

              {

                              SELECT (smf:cast(?newValue, ?arg2) AS ?newLiteral) WHERE {}

              }

}

 

Using a single expression would be

 

SELECT (smf:cast((xsd:double(?arg1) * currencies:getRateByCurrencies(datatype(?arg1), ?arg2)), ?arg2) AS ?newLiteral)

WHERE {

}

 

The example is a bit atypical because it exclusively uses LETs, and not even a triple match. It also uses externally pre-bound variables. But still it gives some insights.

 

 

---

 

In the following function body, project expressions work actually ok, but keep fingers crossed that you do not have to return multiple of such computed values in the SELECT:

 

SELECT ?value

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

    LET (?value := ((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2)) .

}

 

would become

 

SELECT (((((?arg1 * ?M1) + ?O1) - ?O2) / ?M2) AS ?value)

WHERE {

    ?arg2 qud:conversionMultiplier ?M1 .

    ?arg2 qud:conversionOffset ?O1 .

    ?arg3 qud:conversionMultiplier ?M2 .

    ?arg3 qud:conversionOffset ?O2 .

}

 

---

 

Here is an example from the SPIN box computer game, using LET in SPARQL rules. This is a very typical use case actually:

 

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .

    LET (?diamond := spinbox:getNeighbor(?this, ?direction)) .

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

    LET (?newDiamondsCount := (?oldDiamondsCount + 1)) .

}

 

This would become

 

# Rule1: Collect and replace diamond if possible

CONSTRUCT {

    ?diamond spinbox:replaceWith spinbox:Space .

    ?world boulders:diamondsCollected ?newDiamondsCount .

}

WHERE {

    ?world spinbox:field ?this .

    ?world spinbox:keyDirection ?direction .         

              {

                              SELECT (spinbox:getNeighbor(?this, ?direction) AS ?diamond)

                              WHERE {

                              }

              }

    ?diamond a boulders:Diamond .

    ?world spinbox:field ?this .

    ?world boulders:diamondsCollected ?oldDiamondsCount .

              {

                              SELECT ((?oldDiamondsCount + 1) AS ?newDiamondsCount)

                              WHERE {

                              }

              }

}

 

 

On Oct 25, 2009, at 8:34 PM, Lee Feigenbaum wrote:

Hi Holger,

Thanks for the feedback. Unfortunately, assignment is not on the current Working Group's road map for standardization at this time. Here's how we got to this point:

From roughly March through May, the WG considered around 40 potential new features[1] for the SPARQL landscape, including assignment[2]. At the time, we documented two implementations (ARQ and Open Anzo) and the support that you expressed for the feature back in March[3].

In going through the features, the WG discussed Assignment in our March, 31 teleconference. You can see the discussion at the time at [4], the results of which was a straw poll result of 7/6/3, indicating some support and several notes of concern.

Later in the process, we took a survey of WG member's prioritized preferences of the proposed features. Steve Harris whipped up the Condorcet results of the survey which you can see at [5]. Assignment was in the middle of the pack.

At the group's first face-to-face meeting in May, assignment was discussed once more[6], with significant concerns expressed from Garlik and OpenLink Software, strong support from Clark & Parsia, and expressions ranging from indifference to mild support from other WG members (as I read the minutes & recollect the conversation). In the end, the group resolved to accept the list of feature proposals at [7], and Kendall's concerns about the relationship between assignment and projected expressions was addressed by Steve H at [8].

Since then, the group has been rechartered with a specific mandate to work on the features decided during the first phase of the group's lifetime[9]. It's my hope & belief that if projected expressions do not end up fulfilling most users' needs, that implementors will extend their SPARQL implementations with assignment or a similar capability, and we will then revisit this in the next round of SPARQL standardization.

hope this is helpful,
Lee


[1] http://www.w3.org/2009/sparql/wiki/Category:Features
[2] http://www.w3.org/2009/sparql/wiki/Feature:Assignment
[3] http://lists.w3.org/Archives/Public/public-rdf-dawg-comments/2009Mar/0009.html
[4] http://www.w3.org/2009/sparql/meeting/2009-03-31#assignment
[5] http://plugin.org.uk/misc/votes2.svg
[6] http://www.w3.org/2009/sparql/meeting/2009-05-06
[7] http://www.w3.org/2009/sparql/wiki/index.php?title=FeatureProposal&oldid=744 
[8] http://lists.w3.org/Archives/Public/public-rdf-dawg/2009AprJun/0231.html
[9] http://www.w3.org/2009/05/sparql-phase-II-charter

Holger Knublauch wrote:

Dear WG,

reading through the drafts (great to have them already!) I am confused about the future of Assignments (LET keyword in Jena) which has proven to be absolutely essential for many of our customers projects. The SPARQL 1.1 working group seems to have converged in favor of supporting Project expressions and subqueries only, but these IMHO fail to address the requirements below.

Problem 1: How to create new values for CONSTRUCT queries

Project expressions solve some problems for SELECT queries, but the major use cases of LET have been in CONSTRUCT queries. I only see subqueries as a (poor) way of creating new values for use in the CONSTRUCT clause. Creating a subquery for every LET looks like a very user unfriendly mechanism to me.

Problem 2: Verbosity

We often work with complex transformations such as string operations that are best split into multiple steps. Project expressions do not allow using intermediate variables, such as below and would force users to chain together very long spaghetti expressions such as SELECT (?x ex:function3(ex:function2(ex:function1(?y))). Imagine this with some more complex expressions and it quickly becomes completely unreadable. Also, consider you would want to reuse intermediate values in multiple places, to avoid duplicate processing.

*SELECT* ?x ?r

*WHERE* {

?x ex:property ?y .

*LET* (?helper1 := ex:function1(?y)) .

*LET* (?helper2 := ex:function2(?helper1)) .

*LET* (?r := ex:function3(?helper2)) .

}

The LET keyword has solved both problems nicely and in the most general way, and would make project expressions superfluous.

I would appreciate pointers to the discussions that led to the decision to not support Assignments at this stage.

Thanks

Holger

PS: For a parallel thread on jena-dev (with Andy's response), see

http://tech.groups.yahoo.com/group/jena-dev/message/41903

 

[Holger Knublauch-2]

On Oct 28, 2009, at 11:13 PM, Richard Newman wrote:

> I believe that a large portion of SPARQL users (maybe all of the non-
> experts) think procedurally when writing queries. They're not  
> thinking about satisfying clauses, they're thinking about "fetch all  
> the subjects with this object, then fetch all their names, then  
> filter out the ones with...".
>
> This is why they're surprised at unexpected results, or unexpected  
> performance: the algebraic interpretation of their queries is very  
> different to what they think they've written.
>
> We're all far too close to RDF query languages to remember how non-
> implementors think.
>
> My wife is a UX person. In that field it's considered wise to never  
> think of the user being wrong: if they've come to the incorrect  
> conclusion, it's very likely because of something you've done or not  
> done, and it's the software that should change, not the user. It  
> would be interesting to run a user test of SPARQL; I'm sure we'd  
> learn a huge amount about the assumptions and pain points of people  
> actually trying to solve problems with it.

I could not agree more with this. But the problems go further than  
just perception. Even if I had fully understood all the details of the  
SPARQL algebra, I still do not know what kind of reorderings will  
happen inside the query engine. The engine may reorder FILTERs because  
of some heuristics. These heuristics may be unsuitable, or they may be  
misinformed because statistical data about the triple store is not  
always available. In those cases, the system should just use the order  
in which the user has specified the clauses.

> I think Holger's point is that SPARQL as specified loses a lot of  
> the information that the query writer has encoded in the query. (He  
> surely knows that FILTERs are not order dependent: that's what he's  
> lamenting.)
>
> Most people do not think in an order-independent fashion,  
> particularly when other language constructs such as OPTIONAL *are*  
> ordered (after a fashion).
>
> I see users interspersing FILTERs throughout their queries all the  
> time. Very often they do it because they know it's the best way to  
> run the query. The query language then says "pull out all the  
> FILTERs", and the implementation then has to decide how to run  
> them... and it might not have as much information as does the user.  
> (For example, when the execution of a custom FILTER function is very  
> expensive, and you need to trick the planner to execute it later or  
> earlier.)
>
> Put another way: I've never *ever* seen a user write something like
>
>  SELECT * {
>    FILTER (?name ...)
>    ?x foaf:name ?name .
>    ...
>  }
>
> even though it's meaningful SPARQL. Perhaps it shouldn't be  
> meaningful.
>
> This problem gets worse when you consider subqueries, remote  
> queries, computed properties...
>
> Perhaps order-dependence is actually an intuitive, reasonable  
> default for a language? Imperative programming language compilers  
> have done a pretty good job starting with ordered statements, and  
> figuring out when they can disregard that to get better parallelism.  
> That's an optimization, not the default.
>
> Devil's advocacy over :)

+1

In order to not break backward compatibility (albeit debatable), maybe  
a new keyword such as SELECT ... WHERE PROCEDURAL { ... } (not a nice  
name yet, but you get the idea) could be introduced to help the engine  
decide whether a query has been written by someone with a procedural  
background or by someone who comes from the SQL world, and who expects  
the engine to do the reordering for him.

Holger