​​

So, an update on things.
#Calckey

1. I'm very sorry for all of the degraded performance lately on calckey.social. As you may or may not know, we've been testing changes with our codebase, database deployment, ims deployment, and a whole lot more. After a lot of testing and changes, I thought that we were good enough to start opening signups again. Turns out... we're not. As I continue to work on things and start team up with some very smart people who've worked in deployment and scaling systems, we're going to do our absolute best to make sure that we can handle things on all ends, and hopefully get extra hands on deck for day-to-day operations, so it's not all on my shoulders. I was sadly unable to attend to a lot of things yesterday due to being out of town for the holiday, and being in a no-service area for the majority of the day, so I didn't know that there were issues until it was too late. Thank you for your patience as we go through growing pains. A full write-up on what went wrong, the steps we took, and how it's fixed (once it's fixed!) will be made after all this is done. For real.

2. I did NOT expect this amount of support, interest, and coverage! Hitting 13k from 11k in a little over 2 days across all servers is still boggling my mind. And it's not just calckey.social -- so many other servers are getting new users, and even ~30 new servers cropped up! Thank you all, from the bottom of my heart. I'll make sure that I do my hardest to make sure you all have a great experience.

3. As promised, you all deserve a peak into what we've been working on in secret for months. More details to come tonight/tomorrow. Stay tuned.
πŸ˜‰

Sincerely,
Kainoa
@kainoa super excited for tonight’s sneak peak!
@kainoa Thanks for the update. It's such a great product and we all want it to succeed. As you say, lots of interest in #Calckey. Perhaps if some people migrate to alternative servers that will lighten the load until the mothership is running on full power.

Most importantly,
take care of you..

@calckey
@kainoa Thanks for all your hard work. Yes, Calckey was difficult to use recently, but we known how crazy it is to manage such a project with such a small team. We can be patient when things don't go as expected.
@kainoa Your flagship instance got so much interest and traffic it was overwhelmed. Think about that - A M A Z I N G!

Excited to see the platform expand, congratulations to you and your whole behind the scenes team!

When you get a chance to breathe, take a beat to enjoy the moment - before the next wave hits.
🌊
@kainoa what would be magnificent is that we document all this so that scaling issues can easily be addressed for other growing servers as well... i mean, i know your writup will likely have that, but i am thinking a special documentation section addressing scaling best practices kind of thing.
@kainoa ​​ am very interested in that writeup
@kainoa why are you continuing to try to expand the flagship instance instead of encouraging people to go to smaller instances instead
@april ...I'm not? We closed signups on calckey.social, ALL of the posts talking about joins have linked to the calckey server list, and I've been consistently going through submissions for said list. Even the list itself talks about joining smaller servers.
@kainoa so y were you reopening signups
@april please read the above post. We reopened signups 3 days ago under the impression that all issues were resolved, and through increased traffic, it was made clear they were not. After that was noticed, signups were immediately closed. Is there something I'm missing?
@kainoa yes. why did you (try to) reopen signups 3 days ago
@april please read my comment to you and my above post. After a lot of work over the last couple weeks, testing many different fixes to both the code and our deployment infrastructure, we THOUGHT that we would be able to handle signups, so we opened them. From that, it turns out there was more work to be done. We then closed signups. Please, tell me if there's something else in that you don't understand.
@kainoa my question is why do you want to open signups on calckey.social in the first place
@april tell me, why SHOULDN'T be open if all is well (or at least perceived to be well) technically? Are you saying that servers made by project creators shouldn't exist?
@kainoa i'm not saying that servers made by project creators shouldn't exist
i'm saying that servers that are really big shouldn't exist
@april calckey.social has a little over 2,000 users. Mastodon.social has 1.3 million.
@kainoa maybe our definitions of big are a little different
@april if you think a server with 2,354 users is big, I don't think we'll be able to see eye to eye on this.
@kainoa agree to disagree
@april fair enough. Not to make this go on for too long, but I'd like to give my personal observations, having been on the Fediverse for years. This is how I'd personally categorize server size:

Tiny: 1-500
Small: 500-2000
Medium: 2000-10000
Large: 10000-50000
Very large: 50000-200000
Extremely large: 200000+

I've been on servers with each of these sizes, from the smallest private self hosted up to m.s.

@april @kainoa

Why would someone want users on a platform intended for people to use?

@kainoa You guys deserve all of this and much more! Calckey is what made me fall in love with the fediverse, and is IMHO one of the only platforms worth using currently
@kainoa Big ups to Kainoa for bringing Calckey into our digital universe! Seriously, the fediverse wouldn't be the same without it. We're chatting, sharing, and connecting across so many awesome features. It's a whole new level of interaction, and we wouldn't be here without the incredible work you've put into the Calckey project.

In the meantime, I really appreciate you using the official server for development! It gives us confidence that the version we deploy will be smooth and stable - it's like a safety net that lets us, the server maintainers, upgrade the instance without worrying too much about the stability.
@kainoa I enjoy being on Calckey.social, except for the downtimes. Am hoping for improvements, so am willing to stay on. Thank you for all your work.
@kainoa Really appreciate your honesty and openness. Things happen in life and it is how we respond that matters most. You and the team are significant contributors in this new era for the Fediverse. I am proud of you and excited for Calckey's future. I am looking forward to the announcement
@kainoa thanks for the heads-up...
@[email protected] Thanks for the update. Not gonna lieβ€”the downtime is starting to bug me. Not in a β€œI’m fucking out of here if it goes down one more time” way or anywhere near, but more of a β€œreally? Again? [sigh]” sort of way.

So again, thanks for the update. I’m looking forward to reading the report and to a more stable future. Cheers!
@kainoa amazing work. appreciate you very much.
@kainoa The hiccups are fine! The site isn't down for very long, and most of us know (I hope!) that this is the move-fast, break-stuff place to help the other instances get stable. You and @atomicpoet are doing a great job in telling us what is going on.

After the slower performance earlier this week compared to today, it feels like things are moving on the right track. Even slow, it still worked, which beats out "scary message and blank page panic".
πŸ’œ

Can't wait for the good news/sneak peek! But I'll tune in tomorrow, it's bedtime here.
πŸ›ŒπŸ˜΄ Get rest too if you need it.