Mastodawn

A chat server running on powerful hardware collapsing when handling more than 100 events per second isn't acceptable. Events scale up based on room activity from non-local users including spammers too. It's an issue for a server with 12 users too.

https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro/

Scaling to millions of users requires Synapse Pro

Nation-scale Matrix deployments will fail if built on the community version of Synapse. Huge deployments need a different architecture, which is what Synapse Pro delivers.

Element Blog

GrapheneOS Jan 18, 2025

Choosing to write the Matrix server software in Python in the first place was a huge mistake. It's now far harder to develop and maintain the software. It heavily contributes to it being buggy and fragile. It's the biggest factor in it being so incredibly slow and hard to scale.

Steven Jan 18, 2025

@GrapheneOS what would be a proper programming language to code in?

PS: I finally finished setting up my Pixel 9 XL with GrapheneOS.

Unfortunately I'm stuck to WhatsApp -_-

But wherever I could I disabled network or background use plus RethinkDNS with filter lists.

Kirrepic Jan 18, 2025

@Mixtape @GrapheneOS A functional language would be better suited for this use case IMO (as a python dev)

innerand Jan 18, 2025

@Mixtape
Elixir might be a good fit for a Matrix server.
@GrapheneOS

GrapheneOS Jan 18, 2025

@innerand @Mixtape It's such a complex and inefficient protocol that it needs all the performance and memory efficiency that it can get. It doesn't resemble a straightforward chat server implementation at all. The overhead largely comes from the decentralized nature or rooms with a decentralized consensus protocol between servers. Every invite, join, leave, kick, ban, power level change, room setting change, etc. is a state event which has to be kept around and is part of state resolution.

fmarl Jan 18, 2025

@Mixtape @GrapheneOS That's a good question. I would choose a language with a good ecosystem, static type system for maintenance and which can create applications with good performance and scales well. I probably would choose Java or Rust.

crepererum Jan 18, 2025

@Mixtape @GrapheneOS IMHO erlang, elixir, Go, Rust. JVM languages to some extend (they do scale better than Python at least).

~n Jan 18, 2025

@crepererum @Mixtape @GrapheneOS Erlang would have been an exceptionally good idea with regard to scaling it up for large organisations that need dependable infrastructure.

rook Jan 18, 2025

@Mixtape @GrapheneOS the matrix folk themselves have a “next gen” server project called dendrite that’s written in Go which seems like a reasonable choice… but it has been continually playing catch-up with synapse since it was started 7 years ago and still isn’t in a position to replace it. It just isn’t a priority to them, for whatever reason.

GrapheneOS Jan 18, 2025

@rook @Mixtape Go or Java would have been a reasonable language platform for the original implementation from a perspective perspective. Java has gotten much better in recent years and there are other Java ecosystem languages such as Kotlin which are less verbose. Both also have great library ecosystems which for this kind of software have better rather than worse options available than Python.

For a rewrite, neither of those are really a great choice for this when it's so performance critical.

GrapheneOS Jan 18, 2025

@rook @Mixtape Their choice of Rust for a partial reimplementation of the synapse code makes a lot of sense. Matrix is such a heavyweight protocol that it really calls for maximizing performance in every possible way. They're reusing the same architecture, design, algorithms, etc. from synapse so the fact that they got a 500x performance improvement out of it is truly ridiculous. It would perform far better than that if it was truly starting over and able to change the db, protocol, etc.

Domenico De Treias Jan 19, 2025

@Mixtape
I'm going OT about WhatsApp here:
https://livellosegreto.it/@De_Treias/113853826521774100

Domenico De Treias (@[email protected])

Hi @[email protected]. You sayd that you're stuck on WhatsApp: what issue are you facing? I'm planning to install and transfer WhatsApp in my pixel 7a + #GrapheneOS, what can I do to make it easy? I'd like to use it in a user profile I don't access usually: does it need a google profile logged-in? From here: https://mstdn.social/@Mixtape/113850969726694934

Livello Segreto

Elric Jan 18, 2025

@GrapheneOS I briefly considered starting a Java implementation some time ago. But it's already pretty complex to implement 😩

~n Jan 18, 2025

@elricofmelnibone @GrapheneOS The Problem with alternative implementations is that the spec will remain a moving target. It basically is „what the one reference implementation does“.

GrapheneOS Jan 18, 2025

@nblr @elricofmelnibone The protocol and implementation have major bugs which led to state resets causing the room to reset back to previous states dropping many of the users and bricking it. These issues have often being triggered by alternate server implementations doing things differently or not being complete. Of course, it's not reasonable to have a public-facing chat protocol where rooms can be bricked by sending certain state changes from other servers but that's Matrix.

~n Jan 18, 2025

@GrapheneOS @elricofmelnibone Eww. But, yeah… adding complexity is the enemy of robustness. It is a set of hard problems however. I‘m not envying them for the task.

GrapheneOS Jan 18, 2025

@nblr @elricofmelnibone Python is definitely not the only problem but they made this vastly harder for themselves by writing the reference implementation in Python. Getting 500x better performance from simply porting it 1:1 to another language able to be dropped into the existing implementation is genuinely ridiculous. That's not Rust being incredible but rather Python being astoundingly slow.500x better performance isn't a high performance chat server... the starting point was that bad.

GrapheneOS Jan 18, 2025

@nblr @elricofmelnibone They didn't get that performance by switching to a new architecture, algorithms and data structures. It's from directly porting it to another language. It goes to show how awful Python was as a language choice for this. If it was simply written in Java, porting it to Rust might have still ended up giving a 3x performance boost, sure, but they could have focused on others things long before it. They wouldn't have had to start making Dendrite and tons of other work.

Steven Roose Jan 18, 2025

@GrapheneOS If you want a chat server system that can handle more than 100 active users, Matrix isn't the answer. Get an #XMPP server and chat on #Jabber instead.

Johannes Brakensiek Jan 19, 2025

@stevenroose @GrapheneOS 100 events per second is not 100 users. When you read that post you see that 40 events per second refer to ~50K simultaneous users.

GrapheneOS Jan 20, 2025

@lazarus @stevenroose It heavily depends on the rooms and federating servers. It also isn't a constant rate based on number of users but rather heavily fluctuates based on who is actively using it, what they're doing and the rooms they're in.

Johannes Brakensiek Jan 20, 2025

@GrapheneOS @stevenroose Yes, that‘s true.

Steven Roose Feb 4, 2025

@lazarus @stevenroose @GrapheneOS I wasn't referring to the 100 events from the original post, sorry.

mirabilos Jan 19, 2025

@stevenroose @GrapheneOS or #IRC

Antranig Vartanian

@stevenroose @GrapheneOS on FreeBSD I just do `pkg install prosody` or `pkg install ejabberd` and just move on with my day. and I’ve been running a public jabber server for 5+ years now.

Protocols over Products. Communities over Companies. Matrix could’ve just been honest (like the sqlite folks)

Fazal Majid Jan 18, 2025

@GrapheneOS Instagram is largely written in Python and WhatsApp in Erlang, both relatively slow languages compared to C/C++/Rust/Zig/Go, and yet they manage to scale. At a previous company, we had a Python based server handling 4000 sustained requests/second with 9ms latency at the 99th percentile, so it's not just the language at fault.

GrapheneOS Jan 18, 2025

> Instagram is largely written in Python and WhatsApp in Erlang,

Those aren't decentralized chat protocols and had massive capital to throw at servers to delay efficiency to a later late. Instagram is not real time chat. WhatsApp has a low max size on groups even today despite all being on 1 server. Aside from that, evidence is needed that they're actually running in those languages to a substantial extent especially today. It's not verifiable in any way.

nicole mikołajczyk 🔜 linux app summit ➡️ piwo ➡️ gpn Jan 18, 2025

@fazalmajid @GrapheneOS calling Erlang *relatively slow* in the context of high-scale deployments like WhatsApp is unfair

GrapheneOS Jan 18, 2025

@mkljczk @fazalmajid Matrix is meant to be a decentralized chat protocol easily deployed by other people. They threw a massive amount of resources into hosting matrix.org which could have gone to development. They never had a large amount of capital available and put a massive amount of money into hosting matrix.org instead of developing a great chat platform. It's not comparable to burning massive amounts of VC funding on hosting to more quickly develop a product, then rewriting it with more.

GrapheneOS Jan 18, 2025

@mkljczk @fazalmajid A major part of why Matrix is so centralized around matrix.org is that the server software it's incredibly bloated, slow and doesn't scale well. Centralized software would not have to be designed in the same way and wouldn't have been in the same situation. Also, Matrix has more and more overhead as the decentralization increases. It scales up the load not only room activity and local users but also number of federating servers. If everyone had their own, it'd be a disaster.

groxx Jan 18, 2025

@GrapheneOS it absolutely boggles my mind that so many open source distributed systems keep choosing python or node.js as their reference implementation.

it's quite often lethal to the project IMO, as it leads to collapses under even minor growth, and the packaging/distribution stories are complete nightmares.

Archivarius Jan 19, 2025

@GrapheneOS if they ready to reimplement it using Rust, I will be happy to contribute

Also I will leave it here, Rust FTW
https://m.youtube.com/watch?v=3fWx5BOiUiY&pp=ygUKcnVzdCB2cyBnbw%3D%3D

Rust vs Zig vs Go Performance

YouTube

GrapheneOS Jan 20, 2025

@ujeenator They're reimplementing it in Rust as a closed source fork of the open source Synapse, which they say already has 500x better performance.

emon Jan 18, 2025

@GrapheneOS 😕

deutrino Jan 18, 2025

@GrapheneOS lol #Matrix

the totally awesome FOSS-as-in-freedom chat solution that's always 2 years away from working properly

GrapheneOS Jan 18, 2025

@deutrino Not FOSS anymore.

Tertle950 - Scrolling WMs da best Jan 18, 2025

@GrapheneOS Wait, since when?

GrapheneOS Jan 18, 2025

@tertle950 See the linked post above.

Oros Jan 18, 2025

@GrapheneOS @deutrino So I guess it's xmpp and simplex chat now ?

deutrino Jan 18, 2025

@GrapheneOS oh right, brilliant point that my non-caffeinated ass completely disregarded

slashtab Jan 18, 2025

@GrapheneOS So, they have better solution written in Rust called synapse pro but it'll be paid and available for commercial use, right? and the buggy, hard on your processor architecture will be available for public use. am I getting this right?

Currently French and German government uses this in some form, are there any more? Only reason they started using this because of the nature of the project, with closed down synapse pro, will it be same...uhh

slashtab Jan 18, 2025

>It’s worth reiterating that community Synapse continues to be Element’s primary focus, and new features, experimental implementations for Matrix Spec Changes, security work and general maintenance lands in community Synapse first to the benefit of the whole Matrix ecosystem - including general performance improvements.

I don't understand this. why do they wanna pour more energy in gutter?!

So the upshot is that the FOSS server code for Matrix sucks; and Element (who wrote most of that code) has launched a new commercial non-FOSS version of the server that doesn't suck.

Their blog is giving mixed messages though. In earlier posts, Element have argued that it is bad for a "country’s digital infrastructure [to be] operationally dependent on a consumer app from a private tech company". But now they say this kind of private tech is necessary to ensure that the code is properly funded.

GrapheneOS Jan 18, 2025

@karadoc The post acknowledges that the current server implementation is terrible but downplays it as only being an issue at a massive scale. If you look at the actual numbers involved and think about it, you can see it's clearly an issue with small deployments with even a dozen users when those users have joined rooms which are being raided by spammers. The events scale up based on room activity from other servers not only local users. They also scale up based on having more federating servers.

Joseph Nuthalapati

@GrapheneOS There is no good technical reason why the community version of Synapse should not include the worker implemented in Rust. A smaller, resource constrained server, such as a single board computer with 2 GB memory would benefit from a more efficient implementation.

Element seems to be adopting the open-core business model like GitLab. I don't blame them for this; they did mention funding problems recently, but the arguments and analogies in the article don't make sense.

GrapheneOS Jan 18, 2025

@njoseph This is a much less reasonable way of doing open core. Synapse is barely functional and gets bogged down by 1 person doing scripting to attack the overall public Matrix federation with spam. Their intention doesn't even need to be taking down servers in order for that to be the end result. Spam via automated account creation across servers ends up causing denial of service right now. Imagine if they started automatically creating servers at different subdomains or whole domains, etc.

bl4x1 Jan 19, 2025

@GrapheneOS @njoseph Do these issues also apply to small scale private instances that don't participate in federation? So something along the lines of "hosting your own discord" not "participating in a big network of federated identity providers"? Because the government instances in France and Germany are probably not participating in federation.

GrapheneOS Jan 19, 2025

@bl4x1 @njoseph It has awful performance and scalability with or without federation but federation does make it a bigger problem especially since the anti-abuse is so terrible. Federation also leads to lots of issues with decentralized state resolution resulting in bricked rooms. If they're not going to use federation, why would Matrix be chosen though? It makes huge sacrifices to support federation and is quite slow and awkward to use because of the resulting complexity, overhead, etc.

bl4x1 Jan 19, 2025

@GrapheneOS @njoseph So what are the alternatives if you want to have a full-fledged Teams or Discord experience including Messaging and Video Calls? Is there any reliable self-hosted FOSS Solution available that doesn't have the issues Matrix has? I don't know any.

ninkosan Jan 19, 2025

@[email protected] I’m not sure what they were thinking posting that, but it probably isn’t going to have whatever the desired effect was…

@GrapheneOS
Aren't there some good third-party(-ish) implementations of it e.g. Dendrite or Conduit)?

The idea of using the funding of an objectively better closed-source version to fund the worse open-source version is absolutely mind-boggling to me. I'm almost absolutely positive that the target audience for something like Synapse Pro would *never* move to Matrix.
This decision looks like an enterprise money cash-grab and not even a good one.

GrapheneOS Jan 21, 2025

@noodlez1232 Dendrite was being made by this same group of people and is essentially dead:

https://github.com/element-hq/dendrite

It has essentially no active development. It goes to show how many resources they're wasting.

Conduit is also close to dead and is hardly usable. Both Dendrite and Conduit have massive issues with bricking rooms. Conduit has a much more actively developed fork.

GitHub - element-hq/dendrite: Dendrite is a second-generation Matrix homeserver written in Go!

Dendrite is a second-generation Matrix homeserver written in Go! - GitHub - element-hq/dendrite: Dendrite is a second-generation Matrix homeserver written in Go!

GitHub

Andromxda 🇺🇦🇵🇸🇹🇼Jan 21, 2025

@GrapheneOS @noodlez1232 I assume you are talking about conduwuit? Why didn't you actually specify the name of the project? Is there something wrong with it?

Introduction - conduwuit 🏳️‍⚧️ 💜 🦴

conduwuit, which is a well-maintained fork of Conduit, is a simple, fast and reliable chat server for the Matrix protocol

Ellie 🏴🏳️‍⚧️Jan 22, 2025

@GrapheneOS letting all those community servers run on extremely inefficient code has to have an awful environmental impact too, no?

GrapheneOS Jan 22, 2025

@31113 There aren't that many of them. It is certainly wasting the time and money of people trying to use Matrix and experiencing awful performance/usability due to this. Spam attacks turn into denial of service even with only just manual account creation and messages.

Tim Makarios Jan 24, 2025

@GrapheneOS
Out of curiosity, why did you choose Discord, Telegram, and Matrix for your community chat rooms?

GrapheneOS Jan 24, 2025

@TMakarios There aren't a lot of platforms able to handle very large public rooms with thousands of active users. We also didn't choose Matrix or Telegram but rather communities began growing there and we made official rooms. We started on IRC, not either of those.

CyberFrog Jan 24, 2025

@GrapheneOS @TMakarios I've also spoken to several Graphene contributors about the Matrix rooms, previously more than one of them was bricked due to state resolution bugs that Matrix insists are/were fixed, it's not exactly ideal for a big support community lol

GrapheneOS Jan 24, 2025

@froge @TMakarios Matrix having tons of state resolution bugs isn't something we were aware of when we started on it. Hard to be aware of something like this when the project doesn't openly acknowledge it and hushes up discussion about it shaming people for talking about it. Our rooms haven't been bricked for a year due to avoiding actions which trigger it.

CyberFrog Jan 24, 2025

@GrapheneOS @TMakarios very very true, I am not impressed by their response, but I understand why Graphene maintains an official community there regardless