Lemmy.ml & Lemmy.world not federating, no issues with Lemmy.world not federating with Mastodon

https://lemmy.world/post/2424133

Lemmy.ml & Lemmy.world not federating, no issues with Lemmy.world not federating with Mastodon - Lemmy.world

Cross post from lemmy.ml [http://lemmy.ml] - https://lemmy.ml/post/2640822 [https://lemmy.ml/post/2640822] Been experimenting and reviewing why our community based on lemmy.world is super active across Lemmy & Mastodon is dead on lemmy.ml [http://lemmy.ml] - not sure if it’s a lemmy.world issue, lemmy.ml [http://lemmy.ml] issue, or lemmy in general issue so crossposting. See the screenshots. Basically, people posting to [email protected] [/c/[email protected]] from lemmy.world or other instances all seem to work well, even going to Mastodon with no issues. But lemmy.ml [http://lemmy.ml] doesn’t seem to be seeing many of the post and is slow to update things like pinned posts. Nor does lemmy.ml [http://lemmy.ml] posts to the community seem to be going to Mastodon. Challenging to build a community on Lemmy when one of the big instances can’t see many of the posts making an active community of over 1500 users seem dead. lemmy.ml [http://lemmy.ml] view: https://lemmy.ml/c/[email protected] [/c/[email protected]] [https://lemmy.ml/c/[email protected]] lemmy.world view: https://lemmy.world/c/alternativenation [https://lemmy.world/c/alternativenation] mstdn.social view: https://mstdn.social/@[email protected] [https://mstdn.social/@[email protected]]

@[email protected] contacted me about an issue with federation, seems we blocked an IP too many in our attempt to block malicious requests… I’ve removed that block and federation works again. Sorry about the inconvenience.
How did you identify which IPs to block? I’ve had several members discuss.online acknowledge failures to sync with lemmy.world. A big reason why I’m posting this is to test that it works.
This is another test.

FYI, these aren’t actually making it to lemmy.world:

From what I can tell, it looks like discuss.online hasn’t federated properly for at least a week, probably longer.

We’re aware. I’m making them test the messaging. We seem to be sending the message okay. Not sure why they’re not receiving it. Ruud is looking into it on his end.

We agree that it started after the 0.18.3 upgrade.

I just now came across a post saying the same thing, I was actually on my way to send it to you when I saw you responded:
lemmy.world/post/2650904

Looks like it’s going around, as you said. Thanks for putting in the work to make this nice for everyone.

PSA for Lemmy instance admins: in backend v0.18.3 there is a bug that causes your instance to stop federating properly and to stop sending out outgoing messages - Lemmy.world

This new version introduced a system so that your instance stops sending out content to other instances that are supposedly dead / offline. Unfortunately for some reason there’s false positives. When I checked comparing the results from a curl request vs the information in our Lemmy database I found over 350+ false positives. In the DB there is a table called “instance” which has a column called “updated”. If the date on that column is older than 3 days, your server will stop sending any content to those instances. For some reason I had entries that were dated as last being alive in July, while actually they were always up. If an entry is incorrect, you can fix it by manually using an update statement and adding today’s date. If your instance is not too large you can safely update all entries to today’s date and check if everything works as expected from then on any new content created on your instances. The dead instances won’t have an impact unless your instance is larger and generates more content, thus it might be easier to simply update all entries and have Lemmy believe they’re all alive if you start noticing wonky behavior and don’t want to check one by one. If you don’t know how to access the database run this command where domaincom is your instance domain without the dot. 1. docker exec -it domaincom_postgres_1 busybox /bin/sh 2. psql -U (The default user is ‘lemmy’) You could technically do this is one single step, but it’s good to know the command to get shell access to the container itself if you didn’t know how to. This should give you access to a postgres CLI interface. Use \c to connect, \dt to list tables and \d+ tablename to list table definition. You can also run SQL queries from there. Try with this query: SELECT * from instance to list all instances and their updated date. You can use other SQL queries to get better results or correct false positives. Just be careful with what you execute since there’s no undo.

Thanks a lot! I love all the help from the community. Looks like it’s starting to sync again now.
If I comment on this will it sync the whole chain back to lemmy.world? let’s see!