tor sparql queen ✅does anyone have a good example / bit of code i can look over for using spark + pref python to iterate over a large number of HTTP calls?
use @@cm_harlow are you doing some sort of massive crawl?
@ekansa ... yeah :-/ i'm trying to get a bunch of MARC records outta LoC via 4 diff api routes
@cm_harlow And their terms of service are OK with that?
Sorry, I've never done anything with Spark. Sounds like you're doing a huge job though. I wonder if the LoC could just give you a big data dump?
@ekansa terms of service for 1 of the services have a query time limit, others are open.
their data dumps are not the representation i need, a lossy representation at that, and nearly a year out of date.
and they're not willing to generate new data dumps for me at this moment and send me to the various request options.
(but i could pay a vendor for what i need)
@cm_harlow Wow! Seesh.
So you doing some sort of analysis on all these MARC records? Or is this to buildup your own data for retrieval services? Or something else.
Sorry, I can't help but I'm super intrigued by the scale of your project!
@ekansa heh, nw. i'm looking to retrieve a full dump of the Authorities to then serve up + manage in a few different ways - as a Git repo, as a ResourceSync source, as a IPFS repo.
@ekansa + perform a more granular conversion to RDF, enhance with reconciliation, + serve/publish in same mechanisms.
@cm_harlow wow! That sounds really awesome. For what resources? All the LoC or some?
@ekansa all the LoC Name + Subject Authorities - at least at first.
@cm_harlow That sounds super cool. Would be really interesting to link up with some other linked datasets, esp. gazetteers.
@ekansa for sure. geo data is one of the things that suffers the worst from the current LoC data dumps in RDF bc the conversion is lossy