TheDude

June 12 - Server Capacity Update

https://sh.itjust.works/post/49787

June 12 - Server Capacity Update - sh.itjust.works

I’m aware that some of you have been getting some errors loading this instance. This was a configuration that I needed to be adjusted which has since been. Do be patient if we run into other issues, I’ll be continuously working in the back-end with others to better your overall experience. Still a lot of capacity available!

Jun 12, 2023 at 10:15amWeb
Show threadTheDudeJun 12, 2023
if anyone starts getting any weird errors please do let me know but everything still seems to be running smoothly from my side.
Show threadcsm10495Jun 13, 2023
I tried to create a new community a couple times and it hung a bit ago. I'll try again tomorrow.
Show threadTheDudeJun 13, 2023
That's not normal. Let me know if it happens again
Show threadcsm10495Jun 14, 2023
Yeah still hanging for me (even on a different computer) :/ In looking at the dev tools, it doesn't seem to do a POST or anything. The only thing it does is load the svg for the 'thinking' icon on the button.
Show threadPCChipsM922UJun 13, 2023
Yeah, it hangs sometimes. Log out, close the tab, clean cache, then log in again, should work 👍.
Show threadBarbarianJun 12, 2023
Great to hear, and thanks for the update!
Show threadmrmanagerJun 12, 2023
Users should spread out across instances - that is the entire point of being distributed and federated.
Show threadCracks_InTheWallsJun 12, 2023

Absolutely. At the same time, when you have a bunch of users migrating from a centralized platform who don't quite get the fediverse stuff yet, they're likely going to go to well-populated instances.

Hopefully as things progress messaging around this point will penetrate with new folks (like me!)

Show threadShadowAetherJun 12, 2023
Thank you!
Show threadEcholotJun 12, 2023
Would love to hear some more details on the misconfiguration if you want to share @[email protected]
Show threadTheDudeJun 12, 2023
just updated the thread post to include more information for your curious beautiful mind
Show threadtaladarJun 12, 2023
Not sure if you already did or the default config does it but if you only have one backend in an nginx reverse proxy it might also make sense to configure the max_fails and possibly the fail_timeout options so nginx won't consider your backend down for a few seconds every time it receives a TCP error connecting to that (single) backend. max_fails=0 in the named upstream section (see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream ) is what you want here.
Show threadMstraaJun 12, 2023
Thank you for this great server!
Show threadstarroxJun 12, 2023
Thanks dude. The performance here is really excellent in my opinion. Had a couple of page errors but nothing that couldn't be fixed by reloading.
Show threadJordJun 12, 2023
Thanks for addressing it so quickly!
Show threadfalkerie71Jun 12, 2023
Thanks for the update! Any plans for a separate channel outside of sh.itjust.works like Discord or Mastodon, just in case to give people a heads up if the server goes down or is in maintenance?
Show threadPolDeltaJun 12, 2023
I like the idea of a status account on Mastodon. That’s probably a little more accessible than Discord.
Show threadborariJun 12, 2023
Matrix would also be a viable alternative to Discord.
Show threadcontentedness Jun 12, 2023
Nice work dude
Show threadagreenbhmJun 12, 2023
@TheDude does Lemmy support a distributed configuration with multiple database and app servers, or are you limited to a single instance of everything?
Show threadTheDudeJun 12, 2023
The official supported deployments are single instance based, that being said things are already broken up into separate docker containers so it should be pretty easy to do. Would need to do some testing before hand. If this instance continues growing this way I'll need to look into scaling horizontally instead of vertically
Show threadagreenbhmJun 12, 2023
Thanks for the info! Keep up the great work man.
Show threadhttpjamesJun 12, 2023
Software engineer here - check out CockroachDB, a distributed SQL database compatible with PostgreSQL clients.
PostgreSQL Compatibility

A summary of CockroachDB's compatibility with PostgreSQL

CockroachDB Docs
Show threadcanJun 12, 2023

If this instance continues growing this way I'll need to look into scaling horizontally instead of vertically

Could you elaborate what this means?

Show threadisildunJun 12, 2023
Better to have more servers as opposed to one ultra powerful server because the ultra powerful server is expensive. Also, you pay for all the power even during times you don't need it whereas you can take down or add more weak servers as necessary.
Show threadcanJun 13, 2023
Ah, that makes sense. Thank you for sharing.
Show threadscarrexxJun 12, 2023
Thanks for the fix @[email protected].
Show threadProvenscrollJun 12, 2023
Glad to have an update on this, Thanks!
Show threadmanifexJun 12, 2023
Thanks @TheDude! Scaling systems like this is always a challenge and you really get to learn the performance quirks of the code. Thanks for all your work.
Show threadAllHailTheSheepJun 12, 2023
thanks!
Show threadThe Real Geno SmithJun 12, 2023

And through it all, the Dude abides.

Thanks for hosting, thanks for the expertise, thanks for the response time!

Show threadlmlowerJun 13, 2023
I joined for the server URL, and definitely stayed once I saw his username. Long live the Dude!
Show threaddzaffairesJun 12, 2023
Thanks for the great server! I've read your sticky post saying the storage was the most unexpected ressource going up. Can you share what kind of storage space your talking about? I could in the future be interested in spinning up an instance, and I haven't seen any documentation or info on that.
Show threadcanJun 12, 2023
Are things scaling well then? Is it still alright to recommend new people come here or would you rather hold off for a bit now?
Show threadTheDudeJun 12, 2023
I figured out the storage issue last night, the instance is only at about 20% utilization so we should be good to take on a good amount more. We probably will need to do some more tweaking as we grow but for now its looking pretty good!
Show threadBarbarianJun 12, 2023
Wow. 20% usage at 2.4k users is pretty damn amazing.
Show threadTheDudeJun 12, 2023
its not so much about how many registered users as it is about how many online users. The online user count is what adds fuel to the fire!
Show threadflambonksciousJun 13, 2023
Oh wow so it's really the transactions per minute, not so much the baseline set of data?
Show threadPCChipsM922UJun 13, 2023
Yeah, basically the problem is number of calls (requests) per minute. If it's not too many, everything works fine, if it overloads, you get 500 errors 🤷.
Show threadcanJun 14, 2023
Including value to federated communities, yes?
Show threadPCChipsM922UJun 14, 2023
I'm not sure about that, but most probably yes, since if you post on federated communities, your username pops up as [email protected] and the post is saved on your instance (from what I know).
Show threadVinSJun 12, 2023
In Montreal, do we need to send you some hard drives?
Show threadimaqtpieJun 12, 2023
Good to know. Keep up the good work, you're the man.
Show threadPCChipsM922UJun 13, 2023
Think long term. Don't take too many new users or you'll end up with loads of new content (mainly images). Storage is not expensive nowadays, but you can't load new disks to the arrays indefinitely.
Show threadcanJun 14, 2023
Can't we just link to images hosted elsewhere? Like a certain site I once knew
Show threadPCChipsM922UJun 14, 2023
We can, but then you might loose history. This is long known problem on forums. Trying to cut on disk space by linking to image hosts. Sometimes the images survive, most of the time they don't. If that doesn't bother most of the user base, fine, but certain communities would like to keep a history of the media, since sometimes they share valuabe information, like maybe schematics or art.
Show threadcanJun 14, 2023
I guess it's a balance. I remember the imageshack purge but I also don't want to overload the instance.
Show threadPCChipsM922UJun 14, 2023

Photobucket was next after that, not to mention countless others. Imgur has been pretty consistent over the years, but after 2 purges and hundreds of lost images (schematics in most cases) on my forum, I'm scared to death of trusting another image hosting company.

Basically, we still allow externally hosted images on the forum, but only for temp things, like buy/sell, stuff like that. Everything else is attached on the forum.

Show threadcanJun 14, 2023
Imgur is deleting all images that were uploaded without being signed into an account.
Show threadPCChipsM922UJun 15, 2023
Didn't know that, thanks for the info.
Show threadcanJun 15, 2023
And all nsfw too iirc
Show threadHoochIsCrazyJun 12, 2023
What a dude. Thanks for your work
Show threadthisJun 12, 2023
congradulations, you have now beat reddit itself for uptime, lmfao.
Show threadplayer1Jun 12, 2023
Funny enough I’d say Shit Doesn’t Just Work - it takes your hard work. Thanks for helping fill the Reddit void @[email protected]
Show threadlarktrebligJun 12, 2023
Thanks for the update, i really like the transparency
Show thread_haha_oh_wow_Jun 12, 2023
Thank you for hosting this and keeping us informed!
Show threadImFresh3xJun 12, 2023
Thanks. Glad I’m here.
Show threadPootJun 12, 2023
Really glad to have a home here. Thank you for your work!!!!
Show threadWhooping_SealJun 12, 2023

Just out of curiosity, would it help if when posting images we use services such as imgur (or alternatives)? I’m assuming if there is storage issues those type of posts are the biggest culprit.

Thank you for hosting this server @TheDude :)

Show threadTheDudeJun 13, 2023
Yes 100%. Using external image hosting service would reduce how much storage is being consumed