A lot of Mastodon instance admins are either getting their first hosting bill or getting a very increased bill from extreme growth and usage. Some may be experiencing bill shock.
Please make sure your paying/supporting them!
Twitter was free because you were the product. Mastodon doesn't work this way.
Many instances list a patreon or donation links on their about page.
The very wonderful @aurynn wrote about what it takes to host cloudisland.nz which is hosted in New Zealand on Catalyst (and not AWS/Azure/GCP)
Okay, I published a big post on what's gone on with Cloud Island this last month. https://www.patreon.com/posts/cloud-island-75379795
It's interesting to see how people organize things. TBH it looks overengeneered to me as well.
Ok, it's $1000 per month on the cloud. For that money you could go to Servers Australia and get a 48 core machine with 256GB of RAM and 2TB of Nvme storage and with 7TB of free bandwidth thrown in.
This is for a dedicated machine, not a VPS, so each processor would probably outperform a vCPU as well.
I think the cost to performance of cloud systems drops off sharply once you go beyond a small handful of small servers.
@rastilin Sure, it's cheaper.... Until a hard drive fails at 3 am or you unexpectedly need to scale up. And of course once you've scaled up you can't trivially scale back down again because you may be stuck with a minimum lease/commitment.
Not to mention all the administrative overhead of having to run a hypervisor for virtual machines and all that jazz yourself.
There certainly is a point at which dedicated hardware makes sense, but for a professional setup imo it comes much later.
@saluk @Veticia @rastilin @xssfox @aurynn
Sorry, it's not even for big businesses.
Because it's a myth that you do not need devops/admins to maintain your cloud infrastructure.
You do not need hardware engineers, nor move the hardware. Defining the infrastructure, maintaining the security, etc takes at least as much work in the cloud as in “real” data centre, or on premises.
@yacc143 @saluk @Veticia @rastilin @aurynn what a load of rubbish. We'd have to AT LEAST double our work force at $day_job to manage physical servers. To purchase infrastructure to scale acceptably when under increased load would be ridiculous.
Not to mention many things that are trivial and cheap wouldn't be feasible or viable in the cloud.
@yacc143 @Veticia @rastilin @xssfox @aurynn I'm not sure I see how its more effort, yes you still have to solve some of the same software problems... But you don't have to think about hardware. And actually when you are on the cloud you are kind of buying in to some software solutions that work well together. It gets you part of the way there.
But you're right, you can't just expect the cloud company to maintain it for you. You still need devops and security.
Just please don't tell me my company has wasted the last 2 years migrating from on prem lol.
@Veticia @rbairwell @rastilin @xssfox yes.
and I don't use Cloudflare because I don't want to support nazis.
@aurynn @rbairwell @rastilin @xssfox
You're talking about that time Cloudflare shielded neo-nazi website on the principle that they provide infrastructure and are not censors and therefore are on a moral obligation to not interfere.
Which they later retracted and kicked them out anyway.
Or are you talking about another incident?
@Veticia @rbairwell @rastilin @xssfox I am talking about the time they had to be pressured hard earlier this year to drop nazis, and kept refusing and kept refusing and kept refusing and are still unhappy that they had to drop the nazis
because they're nazis and want to protect their own
@Veticia @rbairwell @rastilin @xssfox and also giving them more power over the internet at large if I did so, so, still no.
As far as buying servers, I
- don't have >=$50k laying around to buy kit
- don't have relationships with any DCs to rack said kit
- don't have the time or interest in researching hardware monitoring
- don't have relationships to buy support contracts for said kit
- would lose a lot of flexibility
- have to deal with depreciation
- etc
@aurynn @rbairwell @rastilin @xssfox
I think the problems you describe are related to rather big scale servers. People run mastodons on raspberry pies. I don't think what we're talking about here is a $50k problem.
But let me go back to the international traffic. Is using VPN common in NZ? To pretend all traffic is local? How big of a difference in price are we talking about?
@Veticia @rbairwell @rastilin @xssfox so because Cloud Island is something people rely on, it’s not one server I’d have to buy, it’s multiple, including multiple disk servers to live in geographically disparate DCs to ensure recoverability in the event of catastrophe. I’d have to get support contracts so that I have easy access to spare parts, and multiple servers so that the site isn’t offline for weeks while I wait for parts to arrive and for the DCs smart hands team to install.
It adds up.
@Veticia @rbairwell @rastilin @xssfox and if I want my service to be reliable, I’m don’t feel that single servers in a single DC can provide it, whereas cloud lets me do a lot of things with a lot more capability than I would be able to otherwise, and I can abstract all of the depreciation and managing spares and maintaining DCs and everything away, and focus on making sure my users can rely on what I’m doing.
It is a tradeoff in cost, though, yeah.
@Veticia @rastilin @aurynn @xssfox it's still a single machine. Things fail other than disks. If that machine fails, can you quickly fail over to another machine? Aurynn can easily stand up and fail over to replacement instances if the hardware she's on fails.
You're telling Aurynn that her service is over-engineered, without knowing the SLAs Aurynn is trying to work to. And I'll tell you now, they're SLAs that you cannot meet with a single physical machine.
@chopsstephens @rastilin @aurynn @xssfox
Oh, I don't have anything to criticise Audrynn for. Her setup looks pretty nice actually.
But I also don't necessarily see a single machine setup to be inherently worse. Aurynn's setup currently works on 8 machines, each with its own distinct role. That's 8 points of failure. If one of them dies of if connection breaks between them, all of them can stop working (especially if that hits the database one). It's nice if someone's dealing with redundancy for you, but you have to trust them to do it right.
As for my setup, I can just throw those disks to another machine (I still have a few laying around) and it'll most likely keep working without changing anything.
But I have to agree, sometimes it's just better to pay someone else to deal with all of that for you. (Unless when it's too expensive to justify the cost.) But since Aurynn instance is a paid one I guess she can figure something out. She does look like a smart one.
> objectively the opposite of how we should be building instances
Excuse me? Says who? Objective?
Aurynn wants high availability for her instance. Her users are willing to pay for that. I don't see how anyone else gets a say in whether that's the right way to do things or not.
To be clear, I don't have a problem with you all running on big chunks of physical hardware. We all get to choose, that's the joy.
@chopsstephens @nick
> Her users are willing to pay for that.
145 active users
Cloud Island is currently running across 8 virtual machines in the Catalyst Cloud:
2x webservers
1x Sidekiq server
1x Maintenance and Monitoring server
1x Redis server (used by Sidekiq for process handling)
1x Search server (used for you to do searching)
1x Postgres server (used to store everything)
1x Files server, used to serve media files. (Deprecated, to be removed)
Meanwhile:
https://blog.joinmastodon.org/2017/04/scaling-mastodon/
"What it takes to house 43,000 users"
2x baremetal C2M (8 cores,16GB RAM) servers:
- 1 running PostgreSQL (with pgBouncer on top) and Redis
- 1 running 4x Sidekiq processes between 10–25 threads each
6x baremetal C2S (4 cores, 8GB RAM) servers:
- 2 running Puma (8x workers, 2x threads each), Sidekiq (10 threads), streaming API
- 1 running Nginx load balancer, Puma (8x workers, 2x threads each, Sidekiq (2 threads), streaming API
- 2 running Sidekiq (20 threads)
- 1 running Minio for file storage with a 150GB volume
Also given that some people host 10k-users servers on a single VPS that tells me that you probably don't normally need that much hardware.
My instance mastodon.social has recently surpassed 43,000 users. I have closed registrations both to have more time to investigate the infrastructure and ensure a good experience for existing users, and to encourage more decentralization in the network (with a wonderful effect — the Mastodon fediverse now hosts over 161,000 people spread out over more than 500 independent instances!) But providing a smooth and swift service to 43,000 users takes some doing, and as some of the other instances are approaching large sizes themselves, it is a good time to share the tips & tricks I learned from doing it.
@Veticia @nick you've told me about two other instances, that organize their hardware in two different ways to meet different user requirements and availability requirements. This says nothing about Cloudisland and its choices.
I seriously don't understand what you're trying to prove at this point. Please stop. I'm bored now.
@nick @Veticia if you think that's the next Google you've got another thing coming. I'll tell you right now that's a relatively simple cloud infrastructure.
> How much are fedi users actually harmed if there’s downtime on their instance?
Once again: we all get to choose, that's the joy. You don't have to choose that, but if people do, they're not doing you any harm, and you should butt out.
@nick @chopsstephens
As a side note and a kind of an anecdote I was once (about 10 years ago) in a server room where some of the local, genuine facebook servers were collocated. They were just 3 pc boxes in old, mismatched desktop cases which looked like they were taken out from the '90.
Which proves to me, that even if you are a big internet company. you don't always need big, professional-looking datacenters to run some simple tasks.
Those were most likely cache servers. They have since been removed. The google cache servers that are there now look waaaay more serious. Way more blinky lights.
1.3.6.1.4.1.61513
rastilin
lutoma
Veticia Misumena 🕷️🌺
saluk
Andreas K
Aurynn Shaw
Richard Bairwell
KlavsKlavsen
Sam Stephens
nick
Craig Askings