A chat server running on powerful hardware collapsing when handling more than 100 events per second isn't acceptable. Events scale up based on room activity from non-local users including spammers too. It's an issue for a server with 12 users too.
https://element.io/blog/scaling-to-millions-of-users-requires-synapse-pro/
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
Choosing to write the Matrix server software in Python in the first place was a huge mistake. It's now far harder to develop and maintain the software. It heavily contributes to it being buggy and fragile. It's the biggest factor in it being so incredibly slow and hard to scale.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS what would be a proper programming language to code in?
PS: I finally finished setting up my Pixel 9 XL with GrapheneOS.
Unfortunately I'm stuck to WhatsApp -_-
But wherever I could I disabled network or background use plus RethinkDNS with filter lists.
=> More informations about this toot | More toots from Mixtape@mstdn.social
@Mixtape
Elixir might be a good fit for a Matrix server.
@GrapheneOS
=> More informations about this toot | More toots from innerand@23.social
@innerand @Mixtape It's such a complex and inefficient protocol that it needs all the performance and memory efficiency that it can get. It doesn't resemble a straightforward chat server implementation at all. The overhead largely comes from the decentralized nature or rooms with a decentralized consensus protocol between servers. Every invite, join, leave, kick, ban, power level change, room setting change, etc. is a state event which has to be kept around and is part of state resolution.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@Mixtape @GrapheneOS IMHO erlang, elixir, Go, Rust. JVM languages to some extend (they do scale better than Python at least).
=> More informations about this toot | More toots from crepererum@mastodon.online
@crepererum @Mixtape @GrapheneOS Erlang would have been an exceptionally good idea with regard to scaling it up for large organisations that need dependable infrastructure.
=> More informations about this toot | More toots from nblr@chaos.social
@nblr @crepererum @Mixtape @GrapheneOS But a bad choice when it comes to finding contributors.
=> More informations about this toot | More toots from yahe@chaos.social
@yahe A great selector though as opposed to python unless you're just in for quantity.
=> More informations about this toot | More toots from nblr@chaos.social
@Mixtape @GrapheneOS the matrix folk themselves have a “next gen” server project called dendrite that’s written in Go which seems like a reasonable choice… but it has been continually playing catch-up with synapse since it was started 7 years ago and still isn’t in a position to replace it. It just isn’t a priority to them, for whatever reason.
=> More informations about this toot | More toots from rook@mastodon.online
@rook @Mixtape Go or Java would have been a reasonable language platform for the original implementation from a perspective perspective. Java has gotten much better in recent years and there are other Java ecosystem languages such as Kotlin which are less verbose. Both also have great library ecosystems which for this kind of software have better rather than worse options available than Python.
For a rewrite, neither of those are really a great choice for this when it's so performance critical.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@rook @Mixtape Their choice of Rust for a partial reimplementation of the synapse code makes a lot of sense. Matrix is such a heavyweight protocol that it really calls for maximizing performance in every possible way. They're reusing the same architecture, design, algorithms, etc. from synapse so the fact that they got a 500x performance improvement out of it is truly ridiculous. It would perform far better than that if it was truly starting over and able to change the db, protocol, etc.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS I briefly considered starting a Java implementation some time ago. But it's already pretty complex to implement 😩
=> More informations about this toot | More toots from elricofmelnibone@mastodon.social
@elricofmelnibone @GrapheneOS The Problem with alternative implementations is that the spec will remain a moving target. It basically is „what the one reference implementation does“.
=> More informations about this toot | More toots from nblr@chaos.social
@nblr @elricofmelnibone The protocol and implementation have major bugs which led to state resets causing the room to reset back to previous states dropping many of the users and bricking it. These issues have often being triggered by alternate server implementations doing things differently or not being complete. Of course, it's not reasonable to have a public-facing chat protocol where rooms can be bricked by sending certain state changes from other servers but that's Matrix.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS @elricofmelnibone Eww. But, yeah… adding complexity is the enemy of robustness. It is a set of hard problems however. I‘m not envying them for the task.
=> More informations about this toot | More toots from nblr@chaos.social
@nblr @elricofmelnibone Python is definitely not the only problem but they made this vastly harder for themselves by writing the reference implementation in Python. Getting 500x better performance from simply porting it 1:1 to another language able to be dropped into the existing implementation is genuinely ridiculous. That's not Rust being incredible but rather Python being astoundingly slow.500x better performance isn't a high performance chat server... the starting point was that bad.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@nblr @elricofmelnibone They didn't get that performance by switching to a new architecture, algorithms and data structures. It's from directly porting it to another language. It goes to show how awful Python was as a language choice for this. If it was simply written in Java, porting it to Rust might have still ended up giving a 3x performance boost, sure, but they could have focused on others things long before it. They wouldn't have had to start making Dendrite and tons of other work.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS If you want a chat server system that can handle more than 100 active users, Matrix isn't the answer. Get an #XMPP server and chat on #Jabber instead.
=> More informations about this toot | More toots from stevenroose@x0f.org
@stevenroose @GrapheneOS 100 events per second is not 100 users. When you read that post you see that 40 events per second refer to ~50K simultaneous users.
=> More informations about this toot | More toots from lazarus@fosstodon.org
@lazarus @stevenroose It heavily depends on the rooms and federating servers. It also isn't a constant rate based on number of users but rather heavily fluctuates based on who is actively using it, what they're doing and the rooms they're in.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS @stevenroose Yes, that‘s true.
=> More informations about this toot | More toots from lazarus@fosstodon.org
@stevenroose @GrapheneOS or #IRC
=> More informations about this toot | More toots from mirabilos@toot.mirbsd.org
@stevenroose @GrapheneOS on FreeBSD I just do pkg install prosody or pkg install ejabberd and just move on with my day. and I’ve been running a public jabber server for 5+ years now.
Protocols over Products. Communities over Companies. Matrix could’ve just been honest (like the sqlite folks)
=> More informations about this toot | More toots from antranigv@sigin.fo
@GrapheneOS Agreed, and not only that, it looks like they can't make up their mind on Dendrite. First it's a re-implementation, then the purpose is to be client-embeddable only (why, just do both?), I lost count on what the hell they are doing.
=> More informations about this toot | More toots from aquatica@mstdn.social
@GrapheneOS Instagram is largely written in Python and WhatsApp in Erlang, both relatively slow languages compared to C/C++/Rust/Zig/Go, and yet they manage to scale. At a previous company, we had a Python based server handling 4000 sustained requests/second with 9ms latency at the 99th percentile, so it's not just the language at fault.
=> More informations about this toot | More toots from fazalmajid@vivaldi.net
@fazalmajid
Instagram is largely written in Python and WhatsApp in Erlang,
Those aren't decentralized chat protocols and had massive capital to throw at servers to delay efficiency to a later late. Instagram is not real time chat. WhatsApp has a low max size on groups even today despite all being on 1 server. Aside from that, evidence is needed that they're actually running in those languages to a substantial extent especially today. It's not verifiable in any way.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@fazalmajid @GrapheneOS calling Erlang relatively slow in the context of high-scale deployments like WhatsApp is unfair
=> More informations about this toot | More toots from mkljczk@fediverse.pl
@mkljczk @fazalmajid Matrix is meant to be a decentralized chat protocol easily deployed by other people. They threw a massive amount of resources into hosting matrix.org which could have gone to development. They never had a large amount of capital available and put a massive amount of money into hosting matrix.org instead of developing a great chat platform. It's not comparable to burning massive amounts of VC funding on hosting to more quickly develop a product, then rewriting it with more.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@mkljczk @fazalmajid A major part of why Matrix is so centralized around matrix.org is that the server software it's incredibly bloated, slow and doesn't scale well. Centralized software would not have to be designed in the same way and wouldn't have been in the same situation. Also, Matrix has more and more overhead as the decentralization increases. It scales up the load not only room activity and local users but also number of federating servers. If everyone had their own, it'd be a disaster.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS it's not because of language.
It's because of its design.
=> More informations about this toot | More toots from andyy@fosstodon.org
@andyy No, and they got a 500x performance improvement porting the same architecture, database design, algorithms, data structures, etc. to Rust within the existing system. Sure, more is wrong with it than that but it's the biggest problem for that server software.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS but still designs of this shitty system is the problem.
You can rewrite it to another language but you cannot change the design of it.
Prosody can take a lot of active users and it's written in Lua...
The only choice is XMPP.
=> More informations about this toot | More toots from andyy@fosstodon.org
@andyy It's not decentralized in the same sense. Rooms are hosted on a specific server. Matrix isn't particularly decentralized in practice but it was intended to be.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS @andyy
An XMPP room is hosted on a specific server, but its occupants don't all have to have accounts on that server.
If the room is going to be moderated, authority over it has to be centralized anyway, in the person or people who choose the moderators. In that context, how much more decentralization do you want than what XMPP offers?
I mean, a peep-to-peer system would be nice, but is even harder to design well than a federated one.
=> More informations about this toot | More toots from TMakarios@theres.life
@TMakarios @andyy With Matrix, our chat rooms continue working and can continue to be moderated if the grapheneos.org server goes down. It's nice that when we do things like PostgreSQL database version migrations it avoids room downtime. It similarly reduces incentive to DoS, although Matrix makes that too easy. It's not at all worth the major downsides which come with it being more decentralized: state consensus issues leading to state resets essentially bricking rooms, awful performance, etc.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS it absolutely boggles my mind that so many open source distributed systems keep choosing python or node.js as their reference implementation.
it's quite often lethal to the project IMO, as it leads to collapses under even minor growth, and the packaging/distribution stories are complete nightmares.
=> More informations about this toot | More toots from groxx@hachyderm.io
@GrapheneOS slightly simplistic view… it’s very possible to write python that doesn’t suck (though it wouldn’t be my first choice for this; Rust or something Erlang-ish probably). But from an outside perspective it sure sounds like they are intentionally handicapping the design of the open source product.
=> More informations about this toot | More toots from ianthetechie@fosstodon.org
@ianthetechie They got 500x performance with a partial in-place rewrite in Rust using the same architecture, database structure / connections, algorithms, data structures, etc. It's an in-place partial rewrite and all that's really changing is the language. They put huge effort into trying to optimize the Python and make it work that way. It is not capable of providing reasonable performance for this. It's also clearly far less maintainable at this level of complexity and scale.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS if they ready to reimplement it using Rust, I will be happy to contribute
Also I will leave it here, Rust FTW
https://m.youtube.com/watch?v=3fWx5BOiUiY&pp=ygUKcnVzdCB2cyBnbw%3D%3D
=> More informations about this toot | More toots from ujeenator@mastodon.social
@ujeenator They're reimplementing it in Rust as a closed source fork of the open source Synapse, which they say already has 500x better performance.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
🤖 Tracking strings detected and removed!
🔗 Clean URL(s):
https://m.youtube.com/watch?v=3fWx5BOiUiY
❌ Removed parts:
&pp=ygUKcnVzdCB2cyBnbw%3D%3D
=> More informations about this toot | More toots from c558c7cc69bbda3c271782b736babc64acd2da258b14f356dbca966cb0b7b89e@mostr.pub
@GrapheneOS do you think there is a clear culprit, like bloated data structures, too much context switching, GIL concurrency issues etc.? Or is it likely a heavy mix of everything and the overall language design is fundamentally unfit for such tasks?
=> More informations about this toot | More toots from via4@mastodon.social
@via4 It's unfit for this purpose due to the massive overhead for anything written in Python code, lack of proper threading due to the GIL, etc. The main way people write higher performance Python is delegating nearly all the work to C extensions, which still has high overhead in most cases. The data structures do have high memory and performance overhead but so does everything else in the language. It's also not good at async. It's not suited to writing a server with reasonable performance.
=> More informations about this toot | More toots from GrapheneOS@grapheneos.social
@GrapheneOS Thank you. I'm sorry, I did not notice the attached blogpost at first, they layout the main bottlenecks quite clearly there. Indeed, not all of them seem to be pure Python's fault. But the decision to introduce whole new code base and commitment to maintain it in parallel seems a bit odd. Not sure how it'll play out in the long run given the already tight budget
=> More informations about this toot | More toots from via4@mastodon.social This content has been proxied by September (3851b).Proxy Information
text/gemini