If any Mastodon instances are having trouble with their costs growing out of control, ping me and see if I can help lower your bill.

Apparently a recently closed instance was paying $300/month for hosting which is insane to read considering how small of an instance it was. When we had 40k monthly users I was paying about $120/month so an instance with less than 400 total users (not MAU) should be super cheap to run even with a few TB of storage and bandwidth.

It hurts my heart to read instances closing down over costs and finding out they’re paying more than instances 10x their size. 😢

https://newsie.social/@jeff/114513118997692099

Jeff (of the internet) (@[email protected])

Attached: 1 image Another Mastodon server gone. Airwaves.Social shut down & off the air for good. Reason: Costs! https://vmst.io/@User47/114508192446827453 Running a Mastodon server isn’t free & it’s often thankless. But it is essential. If you value independent, community-run spaces, now’s the time to chip in to your favorite server/instances Your server admins can’t keep the lights on with vibes alone. Want to Suport Newsie or Journa.host? DO IT! OpenCollective: https://opencollective.com/newsiesocial#category-CONTRIBUTE PayPal: https://www.paypal.com/donate/?hosted_button_id=BCV5VZ5HRMB3U

Newsie
@KuJoe @jeff My coats are pretty stable.
@vmstan one day I will learn how type on iOS again. 😅
@KuJoe I’m usually the last person to point out someone else’s typos because I’m just as awful about it, but the thought of my coats growing uncontrollably was too much. 🧥
@vmstan I feel like there’s a meme idea here but I’m too dumb to make it work.
@KuJoe @vmstan Just use sable coin to cover your coats.
(I'll show myself out now)
@KuJoe @vmstan iOS keyboard is the worst and SwiftKey has been constantly crashing for me for months now
@happystardiaz @vmstan I swear it used to be good, for the longest time it was the best typing experience I'd ever had on a phone and then something changed along the way. I thought I was just getting clumsy but it's been a common topic on reddit so at least I feel less crazy. 😄
@KuJoe I decided a while ago that the most I would pay to host https://toot.works was $20/mo. I kept new registrations open until I reached a point where I felt like it was the maximum number of users the instance could support, and then I closed new registrations. I see pleas for help with $2,000 monthly bills, and I think that would be just terrifying.
Toot Works

Toot Works is for folks who make things. It provides a collaborative opportunity to build lasting relationships within a supportive creator community.

Mastodon hosted on toot.works

@joe we turned off open registrations around the 50k mark because we saw things getting out of control fast, and it took us a few months to even understand what was happening under the hood. I’m about to move the instance over to my own hardware which will lower my bill significantly.

I’m assuming a lot of instance admins rely on other for the technical help or just throw money at the problem when things slow down, both of which are valid options if you can afford it.

It took me some time to figure out how the inner workings of Mastodon fit together and now I should use that knowledge to help save others money if I can.

@[email protected] @[email protected] I feel like this is par for the course for the official Mastodon project. Other options like GoToSocial and snac should take MUCH less to operate. Mastodon just feels bloated.
@blainsmith @KuJoe I use managed hosts for both https://toot.works (a Mastodon instance) and https://jws.social (a GoToSocial instance). The cheapest GTS plan is half the cost of the cheapest Mastodon plan. It is still very early days for GTS, though. When I first started, new accounts had to be created from the CLI, and it was only recently that they added the ability to invalidate an API key.
Toot Works

Toot Works is for folks who make things. It provides a collaborative opportunity to build lasting relationships within a supportive creator community.

Mastodon hosted on toot.works
Yeah, I can see that. My snac instance is all CLI at the moment. I hope alternatives get more user friendly to operate because the main Mastodon is a large commitment.

CC: @[email protected]

@blainsmith @joe the biggest difference between Mastodon and all of the other Fediverse platforms is that Mastodon caches all media for privacy and performance reasons which is usually a big cost for people (currently about $5-$7 per TB per month on the cheap end with a medium sized instance using about 2-3TB if they cache for 30 days).

It’s not ideal for single user instances or small groups unless privacy is the biggest concern (the server caches the media so your personal IP address is never logged on other servers).

Additionally keeping a cached copy of all media lets you load it faster and makes it available even if the instance goes offline.

Outside of the storage and bandwidth requirements, you can use an RPi to run Mastodon without a problem for smaller instances.

@blainsmith @joe @KuJoe aside from just not being written in a terribly efficient manner mastodon is written in ruby and ruby is SLOW, about the only language that executes slower is perl
@raptor85 @blainsmith @joe thankfully it doesn’t need to be fast since it scales well, just throw more Arm cores at it and your good to go. 🤣
@KuJoe @blainsmith @joe yeah, or they could have written it in a ...erm..."fast"...language like java (that was actually painful to say) and gotten upwards of a 10x speed increase off of language inefficiencies alone (ruby is actually SHOCKINGLY bad, like it's not a small difference), making some of these massive instances needing beefy servers likely able to run on raspberry pi level hardware. There's a reason mastodon's competitors are WAY cheaper to run.
@raptor85 I am very thankful my instance is smaller now, when it was processing >15 million Sidekiq jobs per day it really needed like 6-8 CPU cores to keep up sometimes. But now that I know it doesn’t care about core speed and just core count, I’m glad to know I can throw some Arm CPUs at it for a few bucks and be good for a long time.
@[email protected] @[email protected] @[email protected] cough Go cough Rust cough 😏

Caching all media is rather nuts though too. I get the appeal, but the cost seems to outweigh the benefits.

@blainsmith I agree, if I didn’t have Backblaze and Cloudflare my costs would easily be double what I pay now just for storage and bandwidth.

Even now I’m paying like $15/month and I barely use 2TB of bandwidth and only have 2.3TB of storage.

@blainsmith @raptor85 @KuJoe The media caching hasn't been a big deal for me. I'm using 1/5 of my allotment for media storage but 3/4 of my allotment for database storage.
@joe I wish I could reduce my DB size! I would love to use managed PostgreSQL hosting because I dislike DBA work, but managed DB services are too expensive right now if you need a lot of storage.

@blainsmith I was just about to ask why there isn't a project aiming to rewrite Mastodon in Rust, C, or whatever, and then I realised that snac is essentially doing that 🤔

@raptor85 @KuJoe @joe

@collectifission @blainsmith @KuJoe @joe I (distressingly) brought up java because it's by far the most common backend programming language still on larger web applications, so the community/library support/etc is already there and ready, and while it's way, way slower than C and somewhat slower than rust it would be the most "practical" choice. (there really aren't many C or rust web devs)

compared to ruby though they're all close in speed, ruby is almost 100x slower in general, ruby is SLOW

@raptor85 If the speed difference is THAT big, I'm wondering why Ruby would be used for anything beyond very small apps and prototyping?

@blainsmith @KuJoe @joe

There is also GoToSocial (Go) designed for small instances too, but being written in Go it can probably do very well with larger ones. I don't think the original Mastodon author could have predicted it getting as popular as it did so perhaps if one of the original design goals was "scale" he would have not used Ruby. Either way, the more important part is the specification, not the implementation. The more Mastodon API apps that come out the better. So far I've seen them for most languages. Just a matter of what the admins choose to run and support.

CC: @[email protected] @[email protected] @[email protected]

@collectifission @blainsmith @KuJoe @joe

I mean...web developers gonna web developer. I also regularly see gigabytes of javascript pulled in to do things HTML5 supports natively, lol

ruby is a particularly easy language for people who are mostly non-programmers, it's extremely fast to get things like web forms up and running so it's become extremely popular. At a small scale with hundreds of users it barely matters, probably didn't think it would get much bigger than that.

@raptor85 @collectifission @blainsmith @joe

Thankfully the Ruby piece for Mastodon is not demanding so you don’t need a ton of high performance compute to power them. You can just throw a bunch of super cheap ARM CPU cores at it and call it a day.

@raptor85 please stop sharing disinformation.

you’re literally on a thread that points out mastodon defaults to a privacy-preserving design that causes *bandwidth and storage* costs to shoot up - and that this can be tweaked.

@phillmv it's not disinformation, ruby is INCREDIBLY slow and vanilla mastodon is a very inefficient implementation, mastodon is severely bloated in storage, cpu, and memory utilization. sure the caching hurts as well, though none of the above thread talks about this, if you're seeing that in the above thread that's due to mastodon's broken threading that doesn't read the same across instances.
@KuJoe however we would not give a penny to Jeff
@helle I don’t know who Jeff is outside of that post I linked, I just saw the post and thought the message was worth sharing. 🙂
@KuJoe Let's be honest, those who say that they pay $300 for an instance are disingenuous... They want to earn some extra money, probably this is normal, but we all know that the costs are much lower in reality... Or is it not all?!
@pescetarian @KuJoe no, mastohost's add on fees can get expensive quick, especially for extra storage or bandwidth. You can host for less than 150 as even a large server but you have to self host, not managed.

@KuJoe @pescetarian @Crazypedia yeah.

For GotoSocial, the 3.90 EUR/month (later 5.90) VM turned out to be too small after all, so I now have one for 10 EUR/month (2 in the first year) with room for dozens more active users (which I’m unlikely to add to my own instance, but).

If I were to host for hundreds, I’d probably go the 30-40 EUR/month area.

@Crazypedia I agree I think the managed piece is the most expensive part. I wish I could volunteer to do the tech management for free for any instance that needed it, but who would trust a random person on the internet do that? 😂
@Crazypedia @pescetarian @KuJoe yeah, Mastohost's database pricing gets expensive quick, and their only supported methods of dealing with it are pay up or enable remote content cache deletion which breaks threads and deletes bookmarks/favorites/boosts. They don't support any of the less nuclear options. The downside of managed hosting.
@anticomposite That's interesting, I didn't realize that but it totally makes sense due to the costs of managed DB servers.
@anticomposite @Crazypedia @pescetarian @KuJoe That's not my experience, they were happy to run `tootctl statuses remove` for me to keep our DB a bit more controlled - I just had to write an email.

@pescetarian @KuJoe Yes & no. If you make bad choices or don't carefully monitor what's going on, you can end up paying far too much for hosting.

Recently I was looking at bare metal VPS hosting costs based on some real usage figures for a public Mastodon server. Monthly hosting in New Zealand was reportedly NZD 333 (€175), several times more expensive than the €35.95 monthly hosted in EU by Contabo, my current supplier, or €46.75 with the same company based in Sydney, Australia.

A year or so back I ran a *private* (I was the only user) Friendica instance on my server with between 1 and 5 posts a day and I noticed my bandwidth and database sizes were spiralling up. A bit of investigation showed a lot of traffic coming from apparently misconfigured sites. I ended up discontinuing the instance.

@juliaclement location is definitely a huge factor, I know people in NZ and Australia pay way more for bandwidth so hosting is expensive no matter what they do to keep prices down. Luckily most of the bandwidth is used by the media storage so if you can get a cheap enough CDN you can save a lot.

@KuJoe I'm painfully aware of that. My former boss used to describe NZ Internet as being at the wrong end of a damp string which is why I host half a world away from me in Germany.

I was surprised that Australian hosting was roughly 30% (€11) more a month than Germany but as you say, they also have to wear high bandwidth costs.

@juliaclement @KuJoe just anecdotally, the combined costs of Chinwag and The Blower are much less than that $300 per month figure (assuming USD) and we're going to be able to reduce that further very soon by moving to services that will bill us in local currency instead. Just avoiding the terrible conversion rate alone is worth it.
@mike location is a big factor, not only are you putting in the effort to find a reliable host you trust, but finding one that doesn’t have pricing that fluctuates every month so you can properly budget is a huge bonus.

@KuJoe we've been with Digital Ocean pretty much forever and I have no issues with the service there, but yeah I can't budget for the next few years with the USD going down the drain like this. 🙃

I'm going to lose a little bit of resiliency but save a lot of money bringing things to a smaller, local provider.

@mike @KuJoe

No, NZD = New Zealand Dollars. NZD 333 is roughly €175. €35.95 is roughly 20% of that which is why I host my existing (lower specification) websites in Germany. It would take unbelievable fluctuations to overcome that difference.

@juliaclement @KuJoe I was thinking mainly of the $300 figure for the recently closed instance in the OP, guessing it was USD there. I noted in the linked thread that masto.host was their provider and while that can be a good option to get off the ground I don't believe their prices scale very well for instances that grow.

That said, they do take a lot of the admin work off your hands, and I'm taking the exact opposite path and moving to dedicated servers in Australia where I'll need to do a bit more management myself but it's going to make the costs more predictable, as well as lower.

Ruud (@[email protected])

It's been a while since I posted something on our blog, I'll create a new post soon. I want to communicate about the finances and then start a fundraising campaign on all our instances, because currently the donations aren't covering the costs anymore. TL;DR last month we spent 1905 EUR and raised 1710 EUR. (The 2 previous months we did some fundraising and raised more than we spent so we still have reserves) mastodon.world lemmy.world sharkey.world friendica.world bookwyrm.world toot.community

Mastodon

@chebra both of those posts are depressing to read. Server admins shouldn't have to worry about finances, they should be focused on running their instances.

I'm sure the admins are doing the best they can with the information they have, I just hope they find some help before they post the dreaded "farewell" post when funding dries up. 🙁

@KuJoe true that. And also, especially when the costs get this high, it feels like donating is just sending more money to the big hosting corporations. So I definitely like the idea with helping to make a more efficient setup. And maybe another idea would be to make the software itself more efficient, so that more people can run it sustainably. #snac is a minimalistic mastodon server, but there are likely many improvements possible also in the main mastodon software. More instances is better.
@KuJoe I think our photog.social admin has recently said we've got about 3 months of hosting left...

@KuJoe I run a few instances, two of them host quite some people

The MAU has dropped the past time but the costs do not drop with this, their media is still stored and we do not pay more or less if we're within our 20TB limit a month, above that it's only a euro per TB so that's not the issue

Mastodon is a super great service but also a little expensive in how it works with media for example

Our monthly bills varry from betweek 1K up to 1400-1500 euros sometimes, that is 4-5x my home rent

@KuJoe However this does include:

3 Mastodon isntance, 1 PeerTube, 2 Pixelfed and some smaller services

But still, it's a lot of money.. without donations i would not be able to do this forever

@stux I'm guessing a lot of that cost is because of the location? I can't even comprehend what a deployment like that would look like. I'm trying to imagine what a $500/month deployment would look like. Do you happen to have an infrastructure diagram or anything so I can see what your deployment looks like? I really need to rewrite mine, this one is from 2023 where I had a 12 server deployment and it was costing me roughly $90/month.

https://docs.mindly.social/infrastructure-layout/

Infrastructure Layout - docs.Mindly.Social

This is the current infrastructure layout as of 01/16/2023.

@KuJoe Also I guess! I made sure it's all within Europe since im also based there but also legal reasons ofc

In the past years ive changed 2 or 3 times from S3 and server hosting to find the best for the most reasonable prices (you know how this can differ soo much)

Our setup is currently actually very simple, all instances but mstdn run on a single server and mstdn has 1 main db/web and 3 sidekiq servers

All media is remote in S3 since it's currently getting close to 50TB or perhaps more

@KuJoe I was for the fun checking masto.host for example, but their max pack would not be enough for 10 or 20% of our current setup 😆

@stux Yikes, that's a lot of storage! Are you doing colocation or renting hardware? 1400/month can buy you a lot of hardware and a 42U cabinet (at least in the US). 😉

I actually have like 6 older Dell R610s filled to the brim with SSDs and RAM that I was going to toss out, but they'd probably make a lovely k8s cluster for future Mastodon instances.

I really wish Mastodon would implement a way to split the media up so you could host cached media in cheaper object storage like S3 and the more important local media on the server's storage for faster performance and resiliency.

@KuJoe @stux I'm really curious to know where you host this to be that low in price

@Tacitus more smaller VMs is better than one or a few large VMs.

I mostly use Hetzner to keep the costs down and aside from my database server every service can run on their smallest VM with plenty of performance headroom to spare.

Most of the VMs were less than $5/month so my database server and object storage made up the majority of my costs. I had a lot of redundancy I didn't need like 2 frontend (nginx) servers, 2 database servers, 2 web servers, and 2 streaming servers so I could have easily cut my bill in half by removing half of my servers, but then I would experience downtime when I needed to do upgrades and reboots so it was a tradeoff.