A lot of Mastodon instance admins are either getting their first hosting bill or getting a very increased bill from extreme growth and usage. Some may be experiencing bill shock.
Please make sure your paying/supporting them!
Twitter was free because you were the product. Mastodon doesn't work this way.
Many instances list a patreon or donation links on their about page.
The very wonderful @aurynn wrote about what it takes to host cloudisland.nz which is hosted in New Zealand on Catalyst (and not AWS/Azure/GCP)
Okay, I published a big post on what's gone on with Cloud Island this last month. https://www.patreon.com/posts/cloud-island-75379795
It's interesting to see how people organize things. TBH it looks overengeneered to me as well.
Ok, it's $1000 per month on the cloud. For that money you could go to Servers Australia and get a 48 core machine with 256GB of RAM and 2TB of Nvme storage and with 7TB of free bandwidth thrown in.
This is for a dedicated machine, not a VPS, so each processor would probably outperform a vCPU as well.
I think the cost to performance of cloud systems drops off sharply once you go beyond a small handful of small servers.
@Veticia @rastilin @aurynn @xssfox it's still a single machine. Things fail other than disks. If that machine fails, can you quickly fail over to another machine? Aurynn can easily stand up and fail over to replacement instances if the hardware she's on fails.
You're telling Aurynn that her service is over-engineered, without knowing the SLAs Aurynn is trying to work to. And I'll tell you now, they're SLAs that you cannot meet with a single physical machine.
@chopsstephens @rastilin @aurynn @xssfox
Oh, I don't have anything to criticise Audrynn for. Her setup looks pretty nice actually.
But I also don't necessarily see a single machine setup to be inherently worse. Aurynn's setup currently works on 8 machines, each with its own distinct role. That's 8 points of failure. If one of them dies of if connection breaks between them, all of them can stop working (especially if that hits the database one). It's nice if someone's dealing with redundancy for you, but you have to trust them to do it right.
As for my setup, I can just throw those disks to another machine (I still have a few laying around) and it'll most likely keep working without changing anything.
But I have to agree, sometimes it's just better to pay someone else to deal with all of that for you. (Unless when it's too expensive to justify the cost.) But since Aurynn instance is a paid one I guess she can figure something out. She does look like a smart one.
> objectively the opposite of how we should be building instances
Excuse me? Says who? Objective?
Aurynn wants high availability for her instance. Her users are willing to pay for that. I don't see how anyone else gets a say in whether that's the right way to do things or not.
To be clear, I don't have a problem with you all running on big chunks of physical hardware. We all get to choose, that's the joy.
@chopsstephens @nick
> Her users are willing to pay for that.
145 active users
Cloud Island is currently running across 8 virtual machines in the Catalyst Cloud:
2x webservers
1x Sidekiq server
1x Maintenance and Monitoring server
1x Redis server (used by Sidekiq for process handling)
1x Search server (used for you to do searching)
1x Postgres server (used to store everything)
1x Files server, used to serve media files. (Deprecated, to be removed)
Meanwhile:
https://blog.joinmastodon.org/2017/04/scaling-mastodon/
"What it takes to house 43,000 users"
2x baremetal C2M (8 cores,16GB RAM) servers:
- 1 running PostgreSQL (with pgBouncer on top) and Redis
- 1 running 4x Sidekiq processes between 10–25 threads each
6x baremetal C2S (4 cores, 8GB RAM) servers:
- 2 running Puma (8x workers, 2x threads each), Sidekiq (10 threads), streaming API
- 1 running Nginx load balancer, Puma (8x workers, 2x threads each, Sidekiq (2 threads), streaming API
- 2 running Sidekiq (20 threads)
- 1 running Minio for file storage with a 150GB volume
Also given that some people host 10k-users servers on a single VPS that tells me that you probably don't normally need that much hardware.
My instance mastodon.social has recently surpassed 43,000 users. I have closed registrations both to have more time to investigate the infrastructure and ensure a good experience for existing users, and to encourage more decentralization in the network (with a wonderful effect — the Mastodon fediverse now hosts over 161,000 people spread out over more than 500 independent instances!) But providing a smooth and swift service to 43,000 users takes some doing, and as some of the other instances are approaching large sizes themselves, it is a good time to share the tips & tricks I learned from doing it.
@Veticia @nick you've told me about two other instances, that organize their hardware in two different ways to meet different user requirements and availability requirements. This says nothing about Cloudisland and its choices.
I seriously don't understand what you're trying to prove at this point. Please stop. I'm bored now.
@nick @Veticia if you think that's the next Google you've got another thing coming. I'll tell you right now that's a relatively simple cloud infrastructure.
> How much are fedi users actually harmed if there’s downtime on their instance?
Once again: we all get to choose, that's the joy. You don't have to choose that, but if people do, they're not doing you any harm, and you should butt out.
@nick @chopsstephens
As a side note and a kind of an anecdote I was once (about 10 years ago) in a server room where some of the local, genuine facebook servers were collocated. They were just 3 pc boxes in old, mismatched desktop cases which looked like they were taken out from the '90.
Which proves to me, that even if you are a big internet company. you don't always need big, professional-looking datacenters to run some simple tasks.
Those were most likely cache servers. They have since been removed. The google cache servers that are there now look waaaay more serious. Way more blinky lights.
1.3.6.1.4.1.61513
rastilin
Veticia Misumena 🕷️🌺
Sam Stephens
nick