One of the things I am really liking about Fastly, other than that it makes the instance incredibly fast from almost everywhere, is the observability. I was poking around and found this graph that shows the origin of traffic to the instance at a given point of time, with the size of the bubble representing how much traffic that pop is getting.

Note the two big circles. One over Germany, and one over Finland. Coincidentally, the location of Hetzner’s two main sites.

Much of that traffic is inter-instance communications. And the reason is that, I strongly suspect, much of the fediverse infrastructure exists in those two Hetzner datacenters.

NB: Hetzner is critical infrastructure for the fediverse.

@jerry Hetzner, OVH and Digitalocean basically

(Plus that shitty Wasabi S3 bucket infrastructure)

@jerry I was JUST pondering about the value of figuring out where other instances are and maybe putting a cluster down closer to them.
@jerry
That seems concerning... Similar to the concern there was about so many services being hosted in AWS and an availability region going down taking out many services. I worry that the goal of decentralized communication and social media is inadvertently circumvented by centralizing infrastructure to specific, lower-cost providers.
@seantpayne I think the issue is that Hetzner is not just a low cost provider, but indeed the lowest cost provider. And since nearly all mastodon instances operate on a shoestring budget, we all end up there. It’s not the best situation for sure.

@jerry
I agree. Your analytics does sort of bring concrete evidence of the problem that the low-cost infrastructure market is lacking competitive diversity for non-profits and low-budget online communities.

πŸ€” This gives me a business idea...

@seantpayne I don’t know why that model works in Germany and Finland and nowhere else…

@jerry @seantpayne

Last I checked they are still working on upgrading the bunny 🐰 to be S3-something-like-🐰-email?

#decentralized πŸ“₯πŸ“€πŸ“¨πŸ“¬πŸ’Œ--->πŸŒπŸ•ΈοΈπŸ•·οΈ?

@jerry @seantpayne I think there is decently good tax incentives here in Finland for data centers, and quite a bit of easy to access cooling. (A lot of closed down paper Mills with good energy infra + access to fresh water.)
@jerry to the best of my knowledge Hetzner is not VC backed. So they operated on sustainable growth and don’t have to flip from being (almost) free to huge profits $over_night.

@gtz42 @jerry This ^^.

Also just knowing which corners to cut and which not.

@jerry does that mean it would make more sense to some how route that traffic direct, instead of through fastly, since theoretically it is inside the same data center?
@ThePhag3 yes. If I had properly designed it way back when, I would have had people access the instance via, let’s say u.infosec.exchange, and put that behind Fastly, and then all the inter-instance traffic could happen to Infosec.exchange. The problem, of course, is that people would likely go straight to Infosec.exchange and not u.infosec.exchange, negating the benefits, like the ability to prevent a ddos on the instance.
@jerry ah 20/20 hindsight for massive growth, years after you initially stood it up. Thank you for all you do, and explaining how and why things are the way they are.
Erik Uden πŸ¦£πŸ‘ (@[email protected])

Hetzner invented the ActivityPub protocol to sell more VMs

MastodonDE
@jerry Probably just means that mastodon.social and the other oversized "default" instances are in Hetzner.
@jerry In short, Hetzner is the AWS of the Fediverse πŸ€ͺ .
@jerry I just wish there was even an ALMOST comparable US alternative that provides quality bare metal servers.
@jerry Uhh I’m a datapoint! The German!
@jerry Let's rather frame this as "Hetzner has seized an opportunity that others have missed, so far."

@jerry Yep. I'm in the UK but the server is in one of those data-centers.

Even the servers in my actual house route via the servers there which have more stable IP addresses than me.

If anyone is making coin from Fedi it's Hetzner.

@jerry hetzner is not just cheap. It's good bang for buck. 16 years a customer and only increasing value for money. That's industry atypical. I think OVH is comparable but have limited experience.
@poetaster Many have used OVH because they were cheap until their data center literally burned down.
@exec ah, vague memory of that. I didn't choose hetzner on price, but price and flexibility. But, I also had dedicated racks at other providers. It was never worth paying more.

@jerry Hmm, one tiny dot where Brisbane is…

Possibly two or three instances there, bne.social, this one, and maybe a third.

There's a few other tiny dots around the place too. Possibly small clusters of instances.

@stuartl some are instances and some are pockets of users
@jerry I see the Carolinas don't exist! 🀣