[xquery-talk] XQuery 3.0 Grouping - Sufficiently baffled by ordering

[Ihe Onwuka]

xquery version "3.0";
The task is to display things in descending order of the number of times
they occur.

New Effort

import module namespace sp="urn:xxxxxxxxxx.com" at 'common.xqm';

<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
group by $name:=$thing/@name
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>

In New Effort, my post-grouping tuple should consist of the grouping key
and a sequence of n $things where n is the number of of $thing for that
grouping key.
I aim to capture that in the let so that I can order by it when outputing.

However the ordering does is ineffective and I get the same output as
Original effort below which makes no attempt to order.

Original Effort

xquery version "3.0";

import module namespace sp="urn:xxxxxxxxxxxxxx.com" at 'common.xqm';

<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
group by $name:=$thing/@name

return <thing name="{$name}" count="{count($thing)}" />}
</counts>

[Michael Kay]

You need to cut out that middle "return". You've written two nested FLWOR expressions, and an "order by" clause on the inner FLWOR expression isn't going to affect the order of the outer FLWOR expression.

Michael Kay
Saxonica

Post by Ihe Onwuka
xquery version "3.0";
The task is to display things in descending order of the number of times they occur.
New Effort
import module namespace sp="urn:xxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>
In New Effort, my post-grouping tuple should consist of the grouping key and a sequence of n $things where n is the number of of $thing for that grouping key.
I aim to capture that in the let so that I can order by it when outputing.
However the ordering does is ineffective and I get the same output as Original effort below which makes no attempt to order.
Original Effort
xquery version "3.0";
import module namespace sp="urn:xxxxxxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return <thing name="{$name}" count="{count($thing)}" />}
</counts>
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

Tried that already but that will give me a let that follows a group by
which is not allowed so I put the return back.

When I moved the count up before the group by (still with no return) to
counter that eXist gave me


err:XPDY0002 variable '$count' is not set. [at line 11, column

Post by Michael Kay
You need to cut out that middle "return". You've written two nested FLWOR
expressions, and an "order by" clause on the inner FLWOR expression isn't
going to affect the order of the outer FLWOR expression.
Michael Kay
Saxonica
xquery version "3.0";
The task is to display things in descending order of the number of times they occur.
New Effort
import module namespace sp="urn:xxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>
In New Effort, my post-grouping tuple should consist of the grouping key
and a sequence of n $things where n is the number of of $thing for that
grouping key.
I aim to capture that in the let so that I can order by it when outputing.
However the ordering does is ineffective and I get the same output as
Original effort below which makes no attempt to order.
Original Effort
xquery version "3.0";
import module namespace sp="urn:xxxxxxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return <thing name="{$name}" count="{count($thing)}" />}
</counts>
_______________________________________________
http://x-query.com/mailman/listinfo/talk
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

Post by Ihe Onwuka
Tried that already but that will give me a let that follows a group by
which is not allowed so I put the return back.
When I moved the count up before the group by (still with no return) to
counter that eXist gave me

when I moved the let statement up before the group ........

Post by Ihe Onwuka
err:XPDY0002 variable '$count' is not set. [at line 11, column

Post by Michael Kay
You need to cut out that middle "return". You've written two nested FLWOR
expressions, and an "order by" clause on the inner FLWOR expression isn't
going to affect the order of the outer FLWOR expression.
Michael Kay
Saxonica
xquery version "3.0";
The task is to display things in descending order of the number of times they occur.
New Effort
import module namespace sp="urn:xxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>
In New Effort, my post-grouping tuple should consist of the grouping key
and a sequence of n $things where n is the number of of $thing for that
grouping key.
I aim to capture that in the let so that I can order by it when outputing.
However the ordering does is ineffective and I get the same output as
Original effort below which makes no attempt to order.
Original Effort
xquery version "3.0";
import module namespace sp="urn:xxxxxxxxxxxxxx.com" at 'common.xqm';
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return <thing name="{$name}" count="{count($thing)}" />}
</counts>
_______________________________________________
http://x-query.com/mailman/listinfo/talk
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Michael Kay]

Try

Post by Ihe Onwuka
<counts>{
for $thing in subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
order by count($thing) descending
return <thing name="{$name}" count="count($thing)}" />}
</counts>

Michael Kay
Saxonica

Post by Ihe Onwuka

Post by Ihe Onwuka
<counts>{
for $thing in subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>

[Ihe Onwuka]

I am most distressed that this worked, firstly

section 3.10.8 of the spec says

An order by clause contains one or more ordering specifications, called
orderspecs <http://www.w3.org/TR/xquery-30/#doc-xquery30-OrderSpec>, as
shown in the grammar. For each tuple in the input tuple stream, the
orderspecs are evaluated, using the variable bindings in that tuple.

but for that query to have worked it must have applied the orderspec to
each tuple in the post-grouping tuple stream.

It also suggests that the count($thing) has to be evaluated twice (in the
order by and the return - perhaps this gets optimised) because of the
prohibition of the let - following the group by.

So it worked, but I'm still baffled.

section 3.10.8 of the spec says

Post by Ihe Onwuka
Try
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
order by count($thing) descending
return <thing name="{$name}" count="count($thing)}" />}
</counts>
Michael Kay
Saxonica
<counts>{
for $thing in
subsequence(collection($sp:reviewsColl)/descendant::reviews[1]//thing,1,50000)
return
let $count:=count($thing)
order by $count descending
return <thing name="{$name}" count="{$count}" />}
</counts>
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Michael Kay]

Post by Ihe Onwuka
I am most distressed that this worked, firstly
section 3.10.8 of the spec says
An order by clause contains one or more ordering specifications, called orderspecs, as shown in the grammar. For each tuple in the input tuple stream, the orderspecs are evaluated, using the variable bindings in that tuple.
but for that query to have worked it must have applied the orderspec to each tuple in the post-grouping tuple stream.

Yes, the output tuple stream of the group-by is the input tuple stream for the order-by.

Post by Ihe Onwuka
It also suggests that the count($thing) has to be evaluated twice (in the order by and the return - perhaps this gets optimised) because of the prohibition of the let - following the group by.

Yes, I was surprised to see that a "let" clause isn't allowed after an order-by or group-by. I don't remember the reasons for that.

Please don't ask me to defend the design. I wanted grouping to work on a sequence of items (like windowing does), not on a stream of tuples, but I was heavily outvoted.

Michael Kay
Saxonica

[John Snelson]

Post by Michael Kay
Yes, I was surprised to see that a "let" clause isn't allowed after an
order-by or group-by. I don't remember the reasons for that.

A let clause _can_ be used after an order by or group by clause in
XQuery 3.0:

http://www.w3.org/TR/xquery-30/#prod-xquery30-FLWORExpr

I suspect this is just unimplemented in eXist.

John

[Ihe Onwuka]

That does not allow for an initial clause to follow an intermediate clause.

Post by Michael Kay
Yes, I was surprised to see that a "let" clause isn't allowed after an
order-by or group-by. I don't remember the reasons for that.

A let clause _can_ be used after an order by or group by clause in XQuery
http://www.w3.org/TR/xquery-30/#prod-xquery30-FLWORExpr
I suspect this is just unimplemented in eXist.
John
--
John Snelson, Lead Engineer http://twitter.com/jpcs
MarkLogic Corporation http://www.marklogic.com
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

Hang on I might have read that wrong ...not sure now.

Post by Ihe Onwuka
That does not allow for an initial clause to follow an intermediate clause.

Post by Michael Kay
Yes, I was surprised to see that a "let" clause isn't allowed after an
order-by or group-by. I don't remember the reasons for that.

A let clause _can_ be used after an order by or group by clause in XQuery
http://www.w3.org/TR/xquery-30/#prod-xquery30-FLWORExpr
I suspect this is just unimplemented in eXist.
John
--
John Snelson, Lead Engineer http://twitter.com/jpcs
MarkLogic Corporation http://www.marklogic.com
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

So the devil lies in the fact that an initial clause is an intermediate
clause.

Well eXist certainly doesn't allow the let.


unexpected token: let (while expecting closing tag for element constructor:
counts)

Post by Ihe Onwuka
Hang on I might have read that wrong ...not sure now.

Post by Ihe Onwuka
That does not allow for an initial clause to follow an intermediate clause.

Post by John Snelson

Post by Michael Kay
Yes, I was surprised to see that a "let" clause isn't allowed after an
order-by or group-by. I don't remember the reasons for that.

A let clause _can_ be used after an order by or group by clause in
http://www.w3.org/TR/xquery-30/#prod-xquery30-FLWORExpr
I suspect this is just unimplemented in eXist.
John
--
John Snelson, Lead Engineer http://twitter.com/jpcs
MarkLogic Corporation http://www.marklogic.com
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

Post by Ihe Onwuka
I am most distressed that this worked, firstly
section 3.10.8 of the spec says
An order by clause contains one or more ordering specifications, called
orderspecs <http://www.w3.org/TR/xquery-30/#doc-xquery30-OrderSpec>, as
shown in the grammar. For each tuple in the input tuple stream, the
orderspecs are evaluated, using the variable bindings in that tuple.
but for that query to have worked it must have applied the orderspec to
each tuple in the post-grouping tuple stream.
Yes, the output tuple stream of the group-by is the input tuple stream for the order-by.

But that is the entire source of the befuddlement. There is nothing I can
see that says that the input tuple stream of the order by is sourced from
the group by (unless there is some presumption consequent on the
juxtaposition of the statements). So I read it as meaning that the input
tuple stream to the order by must be the pre-grouping tuple stream.

Post by Ihe Onwuka
It also suggests that the count($thing) has to be evaluated twice (in the
order by and the return - perhaps this gets optimised) because of the
prohibition of the let - following the group by.
Yes, I was surprised to see that a "let" clause isn't allowed after an
order-by or group-by. I don't remember the reasons for that.
Please don't ask me to defend the design. I wanted grouping to work on a
sequence of items (like windowing does), not on a stream of tuples, but I
was heavily outvoted.
Michael Kay
Saxonica
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Michael Kay]

But that is the entire source of the befuddlement. There is nothing I can see that says that the input tuple stream of the order by is sourced from the group by (unless there is some presumption consequent on the juxtaposition of the statements). So I read it as meaning that the input tuple stream to the order by must be the pre-grouping tuple stream.

The relevant statement is in the introduction to FLWOR expressions in 3.10:

Conceptually, the initial clause generates a tuple stream. Each intermediate clause takes the tuple stream generated by the previous clause as input and generates a (possibly different) tuple stream as output.

However, it is possibly confusing that in this sentence, "the initial clause" means "the first clause", while "Each intermediate clause" means each clause after the first, before the "return". That is, they don't mean the same as InitialClause and IntermediateClause in the grammar.

I'll raise a bug on this.

Michael Kay
Saxonica