Adam JacobActivityPub spec is a marvelous bit of tech writing. It’s also a pretty amazing example of HTTP’s enduring resilience as a protocol - tons of relatively simple things exposed as objects, with a well written spec for each. I’m glad I read it, because it really reframed what we’re actually doing here (basically posting json to our webserver, and getting copies of each others shit dropped off to shared inboxes across our federated server list). It’s an elegant design.
All that said, it’s going to dramatically fall down if it gets anywhere close to twitters scale - unless it recentralizes very quickly. Basically a small number of super-servers will have to appear, because the infrastructure to run the protocol will require it. Huge amounts of storage, duplication everywhere, it’s going to get crazy quickly. I don’t think that’s unique to activitypub, tho.
lisardggy
🐢@adamhjk I think a relatively scalable model that would still allow for smaller-scale governance is to have "relay" servers that function as centralized transports for many instances. They can deduplicate and cache, especially for popular accounts, and allow the leaf instances to focus on serving their users, not talking to every single server out there.
@lisardggY that feels real. It also feels like it would be tough to implement in the protocol. Mostly I’m worried about the storage duplication - there’s in flight messages, and cold storage. Most (all?) instances will eventually simply run out of space for cold storage.
@adamhjk Well, this current boost is exposing the limitations of the original architecture, so its exactly the right time to extend it. I completely agree with you that the p2p architecture is extremely wasteful in large scales, not only in storage and network connections. Not as a slight to the designers or implementers (as you said in the OP), but Mastodon simply wasn't designed for such a scale: https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/
@lisardggY totally. I hesitated to post the original, because I was worried they compliment would get lost in the critique. It’s an amazing spec.
Karl Matthias 🇺🇦@lisardggY @adamhjk messages aren’t *too* bad if they are time windowed, but media will definitely do it in unless shared better or aggressively pruned. Feels like keeping local copies of federated stuff cached for a time window and referring to server of origin for stuff not referenced in X amount of time could work for a much longer period of time. But that’s not how the protocol works now AFAICT.