tor sparql queen ✅does anyone have a good example / bit of code i can look over for using spark + pref python to iterate over a large number of HTTP calls?
use @@cm_harlow are you doing some sort of massive crawl?
@ekansa ... yeah :-/ i'm trying to get a bunch of MARC records outta LoC via 4 diff api routes
@cm_harlow And their terms of service are OK with that?
Sorry, I've never done anything with Spark. Sounds like you're doing a huge job though. I wonder if the LoC could just give you a big data dump?
@ekansa terms of service for 1 of the services have a query time limit, others are open.
their data dumps are not the representation i need, a lossy representation at that, and nearly a year out of date.
and they're not willing to generate new data dumps for me at this moment and send me to the various request options.
(but i could pay a vendor for what i need)
@cm_harlow Wow! Seesh.
So you doing some sort of analysis on all these MARC records? Or is this to buildup your own data for retrieval services? Or something else.
Sorry, I can't help but I'm super intrigued by the scale of your project!
@ekansa heh, nw. i'm looking to retrieve a full dump of the Authorities to then serve up + manage in a few different ways - as a Git repo, as a ResourceSync source, as a IPFS repo.
@ekansa + perform a more granular conversion to RDF, enhance with reconciliation, + serve/publish in same mechanisms.
@ekansa the management aspect means checking on an Atom feed + other spots for notification of records added or updates, then pulling in less intensive fashion. But I need to get over the hump of a preliminary pull of all the data.
@cm_harlow Cool use of Atom, an old but great kind of API.
@ekansa for sure.
i'm basically doing whatever I can to get these datasets better published + shared. I want to explore what forking of large auth. datasets could look like but can't wait on LoC to move towards something other than Voyager / Z39.50 / SRU / MarkLogic for their data services
@cm_harlow Interesting. I've had trouble with really big Git repos before (memory issues), but you can probably divide into a number of smaller repos.
@ekansa yeah - a la who's on first
@ekansa i like what WOF is doing, find their kinda complete separation from other efforts in terms of data models a bit disconcerting though tbh
@cm_harlow what is WOF?
@ekansa sorry, who's on first - https://whosonfirst.mapzen.com/