Ive said it a million times before: in research, if open data mandates mean that all data needs to exist on some cloud server, we are in for a catastrophe of the highest order where cloud providers will be in a position to siphon off however much public funding they want - the "free open data" programs are bait. The corollary is the continual rise and sudden collapse of archives who miss one grant cycle or cant keep up with the infinitely expanding cloud bills.

Its not just a disaster for private data, but public data and the whole of the web. If researchers want their disciplines to continue to exist, to contribute to a healthy information ecosystem, and to realize the promise of the web for science, they should be investing in and demanding p2p infrastructure.

@jonny I get the principle, & it may work for relatively mainstream data (sort of like OpenStreetMap labelling) but a _lot_ more challenging for specialist / niche data where the learning curve is steep & the result interesting only to a handful of specialists.

I'm not sure what good potentials would be, other than the obvious Wikipedia & OpenStreetMap, & they still need paid staff to build software & maintain data integrity, even if you took away hosting costs.

What is your ideal example?

@GavinChait the learning curve for p2p systems is not intrinsically any steeper than cloud systems, in fact there is a potential for it to be much simpler since much less needs to be concentrated in a single system. With generalized p2p the distinction between generalist and specialist data is actually much much easier - there doesn't need to be a specialist database or platform for each type of data.

neither wikipedia nor openstreetmap are p2p systems, and that is one of the reasons why they as platforms need to maintain paid staff to maintain data integrity.

there aren't any such systems, and there are good reasons for that having to do with the history of the platformatization of the web. the things that come closest are private bittorrent trackers which maintain excellent quality archives with next to zero budget in highly adversarial conditions.

here: https://jon-e.net/infrastructure/

I also built a whole tabular data restructuring system (at https://whyqd.com), & my next roadmap steps are to support linked data for categorisation, plus shareable schema & crosswalks.

What you raise is important, I'm just not clear on where the blockers are. Is it curation/interoperability, or distribution, or hosting? I've lead multiple national open data publication projects, & it's common for the realisation of how difficult curation for publication is to end up killing projects.

@GavinChait i'm definitely not saying p2p solves data formats or curation, but that it changes those questions compared to a cloud platform model which frequently bundles formats, curation, schema, interface, and more into a single vertical stack. like what does curation look like? who does it? when does it happen? what are the bounds of heterogeneity? and so on.

that piece has a whole section on my thoughts on barriers and they're also interspersed throughout the piece, so i'll skip that part except w.r.t. the example of stuff like whyqd (which looks cool!).

I am a huge fan of making lots of maps between formats, like a huge huge fan of that practice and it's cool to see what looks like a well done version of it. i tend to think about three things here: the first is re: the tooling that causes people w/ data to so often end up with data like this in the first place (though no matter what there is always slippage between formats). the second is re: why our schema authoring and manipulation tools are so poor that we end up so often with people shoving complex things into single tables. and the third is re: why 'cleaning for publication' is a discrete step where some internal format needs to be matched to an external one, rather than manipulating the form and relating it to other forms being a continuous part of the life of the data.

anyway that's all again at length in that piece so i won't repeat myself

@jonny In academia, there really isn't any excuse for not using formal data methods (although, if you've ever supervised dissertations, you'll know the blood-rush to dive direct into doing analysis without a data plan). It's the bulk of work-based public data - local & national government particularly (from an open data perspective) where drivers, motivations & skills just fail.

Focusing on getting things right in academia would be tremendous. I don't hold out much hope for the public sector.

@jonny That's my intention behind Whyqd as well, that you don't need to formally declare a schema, or ensure interoperability, as it can progress naturally. But, as we find on the OpenLocal project, when working with 350 different publishers - none of whom make any effort to adhere even to government regulated classification standards (like, types of tax) - it's a challenge.

Have you seen https://cedar.metadatacenter.org? It's a means of creating interoperable research forms.