Weirder Admin

@[email protected]
319 Followers
7 Following
200 Posts
Official sock puppet account of the weirder.earth admin team.
Community Patreonhttps://www.patreon.com/distelfliege
Mod Guidelineshttp://weirderearth.de/moderator_guidelines.html
Instance Ruleshttps://github.com/weirderearth/weirder-rules/blob/main/rules-and-etiquette.md
Show threadWeirder AdminDec 18, 2022

On December 14th, shortly before the fediblock post about Anna from witches[.]live, we sent the following:
---
While we understand there might be a lot going on and are not pushing for an immediate resolution, we would like to check in to make sure our prior message didn't get lost. Please let us know if you've received our prior DM on Dec. 10.
---
They replied promptly to let us know that they had and to apologize for being busy. As of today, they have had a full week to stop being busy and respond. As of today, they have not responded.

Presuming there are no changes to their moderation, Weirder Earth will, per our usual policy when dealing with instances that have many cross-follows, be suspending kolektiva.social seven days from the time of this post, on December 25th.

Show threadWeirder AdminDec 18, 2022

Later on the 10th, we replied with this, focusing specifically on Eris from Disqordia in the interest of being clear.
---
This is a frustrating response to receive, but you said you had difficulty navigating the threads. We'd like to highlight a single exchange documented in them. For context, @packbat is Black.

The first screenshot is of https://akko.disqordia.space/notice/AMOna3GxzSqiJN6t9M and one of its replies. Both posts were made by Eris on August 11th.

The second is from an August 13th thread by the Packbats, https://weirder.earth/@packbat/108815931937152540 - the second of the five links we sent you about Eris's anti-ubuntu.buzz campaign.

The third requires a content warning for suicide, and comes from the akko.disqordia.space thread above. Puf shared this screenshot in https://rage.love/@puf/108829477530153878 on August 15th.

We consider it clear that Eris directed abusive language at the Packbats, a Black plural system.

Let us know if you want additional documentation of misconduct.

e̷r̴i̸s̶ :verifiedpurple: (@[email protected])

ah yes Disqordia, the instance that runs on Ubuntu, isn't Ubuntu enough for ubuntu.buzz lol

Weirder AdminDec 18, 2022

Given https://ubiqueros.com/notes/98t4vk2u1q regarding anti-Blackness from Kolektiva, the Weirder Earth mods would like to summarize for #fediblock our recent DMs with Kolektiva's mods on an unrelated issue.

On December 2nd, we warned them about two instances with a history of anti-Black harassment, akko.disqordia.space and social.xenofem.me. They asked for more information, and we shared five threads on Eris's campaign against ubuntu.buzz, a Black-centric instance that got harassed off fedi three months ago:

1. https://rage.love/@puf/108806514088155480
2. https://weirder.earth/@packbat/108815931937152540
3. https://rage.love/@puf/108829477530153878 (cw: suicide mention)
4. https://weirder.earth/@packbat/108992791415798214
5. https://weirder.earth/@packbat/109430760505891265

and linked to the followees list of vrtra, a xenofem mod - https://social.xenofem.me/vrtra?tab=followees - showing accounts from kiwifarms[.]cc, bae[.]st, iddqd[.]social, and freespeechextremist[.]com. (Attached is a screenshot from the 3rd.)

On the 10th, subMedia informed us, on behalf of Kolektiva, that they would take no action.

Ro (@Are0h)

I had my doubt about kolektiva[DOT]social, but as I just discovered Anna from witchesDOTlive has found a home there, that's an immediate #fediblock for me Anna is well known to be on of the primary antagonists of PV back in the day, enabling all sorts of racist and anti-black against my team and me, and whenever I would pushback against their harassment, they would call me transphobic, which is a very common methodology racists use when they want to harass Black and Brown people, but don't want to be viewed as bigots. Anna is one of the worst actors in the fedi, and anyone that gives that person a home cannot be trusted. (📎2)

Ubiqueros: A PV Joint
Show threadWeirder AdminNov 9, 2022
Instance update:
- We've finally cleared all five queues. Your timelines should look correct and you should be getting your emails as normal - let us know if that's not happening.
- That said, a lot more jobs have failed and needed retries than usual these past few days. It's clear that we're not dialed in, and you should expect further hiccups as we test out changes.
Show threadWeirder AdminNov 9, 2022

Instance update:
- Fun fact: Our config said mailer. The queue is called mailers.
- Also: our database was half tied up in orphaned jobs and excessive Puma threads.

Those issues are now fixed, and, with help from @Taniwha, our Sidekiq config is better set up to clear backlogs. The server is being fully utilized right now and should have lots of excess capacity when the default queue clears, and what we know now about how the system behaves should make future incidents easier.

Show threadWeirder AdminNov 7, 2022

Update number ... three?

Good news:
- The server hardware is good. When we get the configuration dialed in, we should be handling our needs, no problem.
- The mailers, pull, push, and scheduler queues are all clear.
- @secretpeej has volunteered to help out (thanks muchly!).

Bad news:
- The default queue, the one responsible for actually putting messages in your timelines, is still backed up.

For now, we're monitoring things and discussing hypotheses. Further bulletins as events warrant.

Show threadWeirder AdminNov 7, 2022
- tl;dr on what queues means: the thing what means that your notifications timeline is two hours behind and your home timeline is nine hours behind.
- We have about 168k events left to process, down from 194k two hours ago, and have been getting through 490/min on average over the past fifteen minutes.
- We plan to decrease shared_buffers from 2GB to 1GB once things stabilize, and then take care of other smaller issues we noticed.
Show threadWeirder AdminNov 7, 2022
Status update on the instance:
- As you probably noticed, the settings change from yesterday did not clear the Sidekiq queues.
- The three of us have continued to do research and system configuration changes, most notably:
= Increasing shared_buffers RAM for the postgres database to handle repeated requests better; and
= Splitting Sidekiq into multiple processes, each assigned to different queues.
- Right now, it looks like the queues are actively emptying.
- Updates as events warrant.
Show threadWeirder AdminNov 6, 2022

Packbats here - update on the server situation.

tl;dr:
- Around the same time as @LouInABox's scheduled maintenance, the Twitter migration started overloading the server.
- Between Lou and @noracodes (thank you, Nora!), we identified the problem and how to address it.
- The instance is currently in emergency clear-the-twelve-hour-backlog mode, so some posts or notifications might get lost overnight.
- Once it's had time to clear, it'll be put in stable mode and we shouldn't have that problem.

Weirder AdminNov 6, 2022
Heads-up: we are continuing to have increasing issues with overloaded Sidekiq queues - expect major delays on emails, notifications, and timelines until we can resolve these issues.