unknown
1970-01-01 00:00:00 UTC
We declare a variable for our query.
declare variable $mqlQuery {'[{
"type": "/film/film",
"name": null,
"netflix_id": []
}]'};
declare variable $freebase {'https://www.googleapis.com/freebase/v1/mqlread'};
declare variable $key {obtain an API key from freebase and puts it's
value here'};
Here is a link to a blog that describes the process of obtaining a
Freebase API key
http://anchetawern.github.io/blog/2013/02/11/getting-started-with-freebase-api/
Since we are going to be doing tail recursion we need to put the API
call in a function. Lets start with the function signature.
declare function local:freebaseCall($cursor as xs:string,$i as xs:integer)
2 parameters the first is the cursor and the second an integer which I
use to provision an auto-incremented unique file name an to tell me
how many records were loaded at the end - since there are a 100
records per API call it's ( $i - 1) * 100 + the number of records
returned by the final cursor. This function will make the API call and
store the results in the db
{
if ($cursor eq 'false')
termination condition
then $i || ' pages loaded' else
let $params := ('query=' || encode-for-uri($mqlQuery), 'key=' ||
$key, 'cursor=' || encode-for-uri($cursor))
Above uri encodes the parameters to the API call - we have three the
MQL query, the API key and the cursor
let $href := $freebase || '?' || string-join($params, '&')
This constructs the API call - again thanks to Michael Westbay for
showing the correct way to do this by string joining the parameters
with a separator of &
let $responses :=
http:send-request(<http:request href="{$href}" method="get"/>)
Make the API call.
return if ($responses[1]/@status ne '200')
then <failure
href="{xmldb:decode-uri(xs:anyURI($href))}">{$responses[1]}</failure>
else let $jsonResponse:= util:base64-decode($responses[2])
Standard EXPATH http error checking - don't forget to base64 decode
the body of the response.
let $freebaseXML:= xqjson:parse-json($jsonResponse)
Convert the returned JSON to XML because we are going to construct an
http PUT to store it in our xml db.
let $movieData := http:send-request(<http:request
href="{concat(path to store the data in your repostiory,$i,'.xml')}"
username="username"
password="password"
auth-method="basic"
send-authorization="true"
method="put">
<http:body
media-type="application/xml"/>
</http:request>,
(),
<batch
cursor="{$cursor}">{transform:transform($freebaseXML,doc(identity.xsl'),())}</batch>)
Standard EXPATH PUT request. On the last line we are wrapping the
returned XML with an element that carries the value of the cursor that
was used to obtain the page. Identity.xsl is of course the standard
XSLT identity transform, you can use it as a placeholder for the
insertion of your own custom transform.
return
local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1)
Finally the tail recursive call. We extract the cursor from the
returned JSON for parameter 1 and increment $i to give us a unique
document name for the next page to store.
};
Don't forget your closing curly brace and the attendant semi colon
One last thing to kick it all off pass the null string as the initial
cursor value and initialise your counter
local:freebaseCall('',1)
One last thing. I experienced repeated eXist crashes when running
this. Note that you can prime the function call with a cursor to pick
up from where you left off. That way you'll get to the end.
return
local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1)
};
declare variable $mqlQuery {'[{
"type": "/film/film",
"name": null,
"netflix_id": []
}]'};
declare variable $freebase {'https://www.googleapis.com/freebase/v1/mqlread'};
declare variable $key {obtain an API key from freebase and puts it's
value here'};
Here is a link to a blog that describes the process of obtaining a
Freebase API key
http://anchetawern.github.io/blog/2013/02/11/getting-started-with-freebase-api/
Since we are going to be doing tail recursion we need to put the API
call in a function. Lets start with the function signature.
declare function local:freebaseCall($cursor as xs:string,$i as xs:integer)
2 parameters the first is the cursor and the second an integer which I
use to provision an auto-incremented unique file name an to tell me
how many records were loaded at the end - since there are a 100
records per API call it's ( $i - 1) * 100 + the number of records
returned by the final cursor. This function will make the API call and
store the results in the db
{
if ($cursor eq 'false')
termination condition
then $i || ' pages loaded' else
let $params := ('query=' || encode-for-uri($mqlQuery), 'key=' ||
$key, 'cursor=' || encode-for-uri($cursor))
Above uri encodes the parameters to the API call - we have three the
MQL query, the API key and the cursor
let $href := $freebase || '?' || string-join($params, '&')
This constructs the API call - again thanks to Michael Westbay for
showing the correct way to do this by string joining the parameters
with a separator of &
let $responses :=
http:send-request(<http:request href="{$href}" method="get"/>)
Make the API call.
return if ($responses[1]/@status ne '200')
then <failure
href="{xmldb:decode-uri(xs:anyURI($href))}">{$responses[1]}</failure>
else let $jsonResponse:= util:base64-decode($responses[2])
Standard EXPATH http error checking - don't forget to base64 decode
the body of the response.
let $freebaseXML:= xqjson:parse-json($jsonResponse)
Convert the returned JSON to XML because we are going to construct an
http PUT to store it in our xml db.
let $movieData := http:send-request(<http:request
href="{concat(path to store the data in your repostiory,$i,'.xml')}"
username="username"
password="password"
auth-method="basic"
send-authorization="true"
method="put">
<http:body
media-type="application/xml"/>
</http:request>,
(),
<batch
cursor="{$cursor}">{transform:transform($freebaseXML,doc(identity.xsl'),())}</batch>)
Standard EXPATH PUT request. On the last line we are wrapping the
returned XML with an element that carries the value of the cursor that
was used to obtain the page. Identity.xsl is of course the standard
XSLT identity transform, you can use it as a placeholder for the
insertion of your own custom transform.
return
local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1)
Finally the tail recursive call. We extract the cursor from the
returned JSON for parameter 1 and increment $i to give us a unique
document name for the next page to store.
};
Don't forget your closing curly brace and the attendant semi colon
One last thing to kick it all off pass the null string as the initial
cursor value and initialise your counter
local:freebaseCall('',1)
One last thing. I experienced repeated eXist crashes when running
this. Note that you can prime the function call with a cursor to pick
up from where you left off. That way you'll get to the end.
return
local:freebaseCall($freebaseXML//data(pair[@name="cursor"]), $i + 1)
};