[xquery-talk] Recursion or not - how to (re)query an API

[Bridger Dyson-Smith]

Hi all,

I'm struggling with getting a working example of the following problem: I'm
sending a query to an API that returns some data and, if the data set is
large enough, a resumption token to requery the API.

I'm stuck at the point of how to generate the new query URL, while still
processing the new data. Is the best idea to create a function that will
update a new request to the API, or is there a different approach that can
be used?

I've made several attempts with functions -- the latest non-working
examples are below -- and I'm missing... something, but I don't know what.

Thanks for any guidance you can provide.
Cheers,
Bridger

(: sample query :)
(:
http://server:8080/fedora/objects?query=pid~{$collection}*&resultFormat=xml&pid=true
:)

declare variable $query external := '
http://our.server:8080/fedora/objects?query=pid~gamble*&resultFormat=xml&pid=true
';
declare variable $sessionToken := '&sessionToken=';
declare variable $resToken := '';

(:functions:)
(:doesn't work:)
declare function local:getPID($in as document-node()) as xs:anyAtomicType*
{
data($in/*:resultList/*:objectFields/*:pid)
};

(:doesn't work:)
declare function local:reQuery(
$in as document-node(),
$rQ as xs:untypedAtomic*) as document-node()
{
let $rQ := $in/*:listSession/*:token/text()
return (
if ($rQ) then
local:getPID(doc("$in" || "&sessionToken=" || "$rQ"))
else ()
)
};

(: this works processing the descendant elements but doesn't recurse :)
for $result in fn:doc($query)/*:result
let $resToken := $result/*:listSession/*:token/text()
let $pid := $result/*:resultList/*:objectFields/*:pid/text()
return (
$result,
$resToken,
$pid
)

(: sample returned from query; e.g. $result :)
<result xmlns="http://www.fedora.info/definitions/1/0/types/" xmlns:types="
http://www.fedora.info/definitions/1/0/types/" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://www.fedora.info/definitions/1/0/types/
http://localhost:8080/fedora/schema/findObjects.xsd">
<listSession>
<token>63c0762b5eb3d4b46c58add843b7b3e6</token>
<cursor>0</cursor>
<expirationDate>2016-01-18T01:40:05.381Z</expirationDate>
</listSession>
<resultList>
<objectFields>
<pid>gamble:1</pid>
</objectFields>
<objectFields>
<pid>gamble:10</pid>
</objectFields>
</resultList>
</result>

[Christian Grün]

Hi Bridger,

I know too little about the API you are working, and your query
processor (parts of the semantics of fn:doc are
implementation-defined), but I noticed that the URL construction might
need to be revised:

OLD:
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))

NEW:
local:getPID(doc($in || "&amp;sessionToken=" || encode-for-uri($rQ)))

Possibly, $in (or a part of it) needs to be encoded as well.

Hope this helps,
Christian



On Mon, Jan 18, 2016 at 5:09 PM, Bridger Dyson-Smith

Post by Bridger Dyson-Smith
Hi all,
I'm struggling with getting a working example of the following problem: I'm
sending a query to an API that returns some data and, if the data set is
large enough, a resumption token to requery the API.
I'm stuck at the point of how to generate the new query URL, while still
processing the new data. Is the best idea to create a function that will
update a new request to the API, or is there a different approach that can
be used?
I've made several attempts with functions -- the latest non-working examples
are below -- and I'm missing... something, but I don't know what.
Thanks for any guidance you can provide.
Cheers,
Bridger
(: sample query :)
http://server:8080/fedora/objects?query=pid~{$collection}*&resultFormat=xml&pid=true
:)
declare variable $query external :=
'http://our.server:8080/fedora/objects?query=pid~gamble*&amp;resultFormat=xml&amp;pid=true';
declare variable $sessionToken := '&amp;sessionToken=';
declare variable $resToken := '';
(:functions:)
(:doesn't work:)
declare function local:getPID($in as document-node()) as xs:anyAtomicType*
{
data($in/*:resultList/*:objectFields/*:pid)
};
(:doesn't work:)
declare function local:reQuery(
$in as document-node(),
$rQ as xs:untypedAtomic*) as document-node()
{
let $rQ := $in/*:listSession/*:token/text()
return (
if ($rQ) then
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))
else ()
)
};
(: this works processing the descendant elements but doesn't recurse :)
for $result in fn:doc($query)/*:result
let $resToken := $result/*:listSession/*:token/text()
let $pid := $result/*:resultList/*:objectFields/*:pid/text()
return (
$result,
$resToken,
$pid
)
(: sample returned from query; e.g. $result :)
<result xmlns="http://www.fedora.info/definitions/1/0/types/"
xmlns:types="http://www.fedora.info/definitions/1/0/types/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.fedora.info/definitions/1/0/types/
http://localhost:8080/fedora/schema/findObjects.xsd">
<listSession>
<token>63c0762b5eb3d4b46c58add843b7b3e6</token>
<cursor>0</cursor>
<expirationDate>2016-01-18T01:40:05.381Z</expirationDate>
</listSession>
<resultList>
<objectFields>
<pid>gamble:1</pid>
</objectFields>
<objectFields>
<pid>gamble:10</pid>
</objectFields>
</resultList>
</result>
_______________________________________________
http://x-query.com/mailman/listinfo/talk

_______________________________________________
***@x-query.com
http://x-query.com/mailman/listinfo/talk

[Bridger Dyson-Smith]

Hi Christian,

thank you for your help. I'm using BaseX 8.3.1 as my query processor. I'm
guilty, I'm afraid, of presenting a poorly framed question to the list. I
need to mull over the problem a little more.

Thanks again!
Bridger

Post by Christian Grün
Hi Bridger,
I know too little about the API you are working, and your query
processor (parts of the semantics of fn:doc are
implementation-defined), but I noticed that the URL construction might
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))
local:getPID(doc($in || "&amp;sessionToken=" || encode-for-uri($rQ)))
Possibly, $in (or a part of it) needs to be encoded as well.
Hope this helps,
Christian
On Mon, Jan 18, 2016 at 5:09 PM, Bridger Dyson-Smith

Post by Bridger Dyson-Smith
Hi all,

I'm

Post by Bridger Dyson-Smith
sending a query to an API that returns some data and, if the data set is
large enough, a resumption token to requery the API.
I'm stuck at the point of how to generate the new query URL, while still
processing the new data. Is the best idea to create a function that will
update a new request to the API, or is there a different approach that

can

Post by Bridger Dyson-Smith
be used?
I've made several attempts with functions -- the latest non-working

examples

Post by Bridger Dyson-Smith
are below -- and I'm missing... something, but I don't know what.
Thanks for any guidance you can provide.
Cheers,
Bridger
(: sample query :)

http://server:8080/fedora/objects?query=pid~{$collection}*&resultFormat=xml&pid=true

Post by Bridger Dyson-Smith
:)
declare variable $query external :=
'

http://our.server:8080/fedora/objects?query=pid~gamble*&amp;resultFormat=xml&amp;pid=true
';

Post by Bridger Dyson-Smith
declare variable $sessionToken := '&amp;sessionToken=';
declare variable $resToken := '';
(:functions:)
(:doesn't work:)
declare function local:getPID($in as document-node()) as

xs:anyAtomicType*

Post by Bridger Dyson-Smith
{
data($in/*:resultList/*:objectFields/*:pid)
};
(:doesn't work:)
declare function local:reQuery(
$in as document-node(),
$rQ as xs:untypedAtomic*) as document-node()
{
let $rQ := $in/*:listSession/*:token/text()
return (
if ($rQ) then
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))
else ()
)
};
(: this works processing the descendant elements but doesn't recurse :)
for $result in fn:doc($query)/*:result
let $resToken := $result/*:listSession/*:token/text()
let $pid := $result/*:resultList/*:objectFields/*:pid/text()
return (
$result,
$resToken,
$pid
)
(: sample returned from query; e.g. $result :)
<result xmlns="http://www.fedora.info/definitions/1/0/types/"
xmlns:types="http://www.fedora.info/definitions/1/0/types/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.fedora.info/definitions/1/0/types/
http://localhost:8080/fedora/schema/findObjects.xsd">
<listSession>
<token>63c0762b5eb3d4b46c58add843b7b3e6</token>
<cursor>0</cursor>
<expirationDate>2016-01-18T01:40:05.381Z</expirationDate>
</listSession>
<resultList>
<objectFields>
<pid>gamble:1</pid>
</objectFields>
<objectFields>
<pid>gamble:10</pid>
</objectFields>
</resultList>
</result>
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Bridger Dyson-Smith]

Winona,
this sounds like an interesting direction -- I'm mostly in the dark on
modules, so this is really neat.

Thanks!
Bridger

Have you considered using the EXPath http client rather then the doc()
function, I find it to be a more flexible option for handling API
interactions? http://expath.org/modules/http-client/samples
You can then run your query, and base subsequent action on the http status
code, do a if/else for http status code 200, if 200, then test for a
resumption token. If there is a resumption token, resubmit same query with
resumption token parameter added otherwise (if no resumption token) your
query is done.
Does that help? Or are you looking for something a little more step by
step?
-Winona
On Mon, Jan 18, 2016 at 11:58 AM, Bridger Dyson-Smith <

Post by Bridger Dyson-Smith
Hi Christian,
thank you for your help. I'm using BaseX 8.3.1 as my query processor. I'm
guilty, I'm afraid, of presenting a poorly framed question to the list. I
need to mull over the problem a little more.
Thanks again!
Bridger
On Mon, Jan 18, 2016 at 11:32 AM, Christian GrÃŒn <

Post by Christian Grün
Hi Bridger,
I know too little about the API you are working, and your query
processor (parts of the semantics of fn:doc are
implementation-defined), but I noticed that the URL construction might
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))
local:getPID(doc($in || "&amp;sessionToken=" || encode-for-uri($rQ)))
Possibly, $in (or a part of it) needs to be encoded as well.
Hope this helps,
Christian
On Mon, Jan 18, 2016 at 5:09 PM, Bridger Dyson-Smith

Post by Bridger Dyson-Smith
Hi all,
I'm struggling with getting a working example of the following

problem: I'm

Post by Bridger Dyson-Smith
sending a query to an API that returns some data and, if the data set

is

Post by Bridger Dyson-Smith
large enough, a resumption token to requery the API.
I'm stuck at the point of how to generate the new query URL, while

still

Post by Bridger Dyson-Smith
processing the new data. Is the best idea to create a function that

will

Post by Bridger Dyson-Smith
update a new request to the API, or is there a different approach that

can

Post by Bridger Dyson-Smith
be used?
I've made several attempts with functions -- the latest non-working

examples

Post by Bridger Dyson-Smith
are below -- and I'm missing... something, but I don't know what.
Thanks for any guidance you can provide.
Cheers,
Bridger
(: sample query :)

http://server:8080/fedora/objects?query=pid~{$collection}*&resultFormat=xml&pid=true

Post by Bridger Dyson-Smith
:)
declare variable $query external :=
'

http://our.server:8080/fedora/objects?query=pid~gamble*&amp;resultFormat=xml&amp;pid=true
';

Post by Bridger Dyson-Smith
declare variable $sessionToken := '&amp;sessionToken=';
declare variable $resToken := '';
(:functions:)
(:doesn't work:)
declare function local:getPID($in as document-node()) as

xs:anyAtomicType*

Post by Bridger Dyson-Smith
{
data($in/*:resultList/*:objectFields/*:pid)
};
(:doesn't work:)
declare function local:reQuery(
$in as document-node(),
$rQ as xs:untypedAtomic*) as document-node()
{
let $rQ := $in/*:listSession/*:token/text()
return (
if ($rQ) then
local:getPID(doc("$in" || "&amp;sessionToken=" || "$rQ"))
else ()
)
};
(: this works processing the descendant elements but doesn't recurse :)
for $result in fn:doc($query)/*:result
let $resToken := $result/*:listSession/*:token/text()
let $pid := $result/*:resultList/*:objectFields/*:pid/text()
return (
$result,
$resToken,
$pid
)
(: sample returned from query; e.g. $result :)
<result xmlns="http://www.fedora.info/definitions/1/0/types/"
xmlns:types="http://www.fedora.info/definitions/1/0/types/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.fedora.info/definitions/1/0/types/
http://localhost:8080/fedora/schema/findObjects.xsd">
<listSession>
<token>63c0762b5eb3d4b46c58add843b7b3e6</token>
<cursor>0</cursor>
<expirationDate>2016-01-18T01:40:05.381Z</expirationDate>
</listSession>
<resultList>
<objectFields>
<pid>gamble:1</pid>
</objectFields>
<objectFields>
<pid>gamble:10</pid>
</objectFields>
</resultList>
</result>
_______________________________________________
http://x-query.com/mailman/listinfo/talk

_______________________________________________
http://x-query.com/mailman/listinfo/talk