About updating Lemmy & explanation of recent federation issues with lemmy.ml - Lemmy.World

Hello World, as many of you know, several newer Lemmy versions have been released since the once we are currently using. As this is a rather long post, the TLDR is that we’re currently planning for late January/early February to update Lemmy.World to a newer Lemmy release. We’re currently running Lemmy 0.19.3 with a couple patches on top to address some security or functionality issues. As new Lemmy versions have been released, we’ve been keeping an eye on other instances’ experiences with the newer versions, as well as tracking certain issues on GitHub, which might impact stability or moderation experience. We updated to Lemmy 0.19.3 back in March this year. At that point, 0.19.3 had been released for a little over a month already and at that point all the major issues that troubled the earlier 0.19 releases had been addressed. Several months later, in June, Lemmy 0.19.4 was released with several new features. This was a rather big release, as a lot of changes had happened since the last release. Only 12 days later 0.19.5 was released, which fixed a few important issues with the 0.19.4 release. Unfortunately, Lemmy 0.19.5 also introduced some changes that were, and to some part are still not fully addressed. Prior to Lemmy 0.19.4, regular users may see contents of removed or deleted comments in some situations, primarily when using third party apps. Ideally, this would have been fixed by restricting access to contents of removed comments to community moderators in the communities they moderate, as well as admins on each instance. Deleted comments will be overwritten in the database after some delay, but they might still be visible prior to that. This is especially a problem when moderators want to review previously removed comments to either potentially restore them or to understand context in a thread with multiple removed comments. Lemmy modlog does not always record individual modlog entries for bulk-removed items, such as banning a user while also removing their content would only log their ban but not the individual posts or comments that were removed. We were considering writing a patch to restore this functionality for moderators in their communities, but this is unfortunately a rather complex task, which also explains why this isn’t a core Lemmy feature yet. While admins can currently filter modlog for actions by a specific moderator, this functionality was lost somewhere in 0.19.4. While this isn’t something our admin team is using very frequently, it is still an important feature to have available for us for the times we need it. This also included a few security changes for ActivityPub handling, which resulted in breaking the ability to find e.g. Mastodon posts in Lemmy communities by entering the post URL in the search. It also caused issues with changes to communities by remote moderators. The 0.19.4 release also broke marking posts as read in Sync for Lemmy. Although this isn’t really something we consider a blocker, it’s still worth mentioning, as there are still a lot of Sync for Lemmy users out there that haven’t noticed this issue yet if they’re only active on Lemmy.World. Over the last 2 weeks we’ve had nearly 5k active Sync for Lemmy users . This is unfortunately something that will break during the upgrade, as the API has changed in upstream Lemmy. There are also additional issues with viewing comments on posts in local communities that appear to be related to the 0.19.4/0.19.5 release, appear to be a lot more serious. There have been various reports of posts showing with zero comments in Sync, while viewing them in a browser or another client will show various comments. It’s not entirely clear to us right now what the full impact is and to what extent it can be mitigated by user actions, such as subscribing to communities. If anyone wants to research what is needed to restore compatibility and potentially even propose a patch for compatibility with both the updated and the previous API version we’ll consider applying it as a custom patch on top of the regular Lemmy release. If there won’t be a Sync update in time for our update and we won’t have a viable workaround available, you may want to check out [email protected] [/c/[email protected]] [https://lemmy.world/c/lemmyapps] to find potential alternatives. There were also several instances reporting performance issues after their upgrades, although they seemed to mostly have been only for a relatively short time after the upgrades and not persistent. Lemmy 0.19.6 ended up getting released in November and introduced quite a few bug fixes and changes again, including filtering the modlog by moderator. Due to a bug breaking some DB queries, 0.19.7 was released just 7 days later to address that. Among the issues fixed in this release were being able to resolve Mastodon URLs in the search again and remote moderators being able to update communities again. 0.19.6 also changed the way post thumbnails generated, which resulted thumbnails missing on various posts. A month later, now we’re in December, 0.19.8 was released. One of the issues addressed by 0.19.8 was Lemmy returning content of removed comments again for admins. For community moderators this functionality is not yet restored due to the complexity of having to check mod status in every community present in the comment listing. At this point it seems that most of the issues have been addressed, although there seem to still be some remaining issues relating to thumbnails not reliably being created in some cases. We’ll keep an eye on any updates on that topic to see if it might be worth waiting a little longer for another fix or possibly deploying an additional patch even if it may not be part of an official Lemmy release yet at the time. While we were backporting some security/stability related changes, including a fix for a bug that can break federation in some circumstances when a community is removed, we accidentally reverted this patch while applying another backport, which resulted in our federation with lemmy.ml [http://lemmy.ml] breaking back in November. This issue was already addressed upstream a while back, so other instances running more recent Lemmy versions were not affected by this. Among the new features released in the Lemmy versions we have missed out on so far, here are a couple highlights: - Users will be able to see and delete their uploads on their profile. This will include all uploads since we updated to 0.19.3, which is the Lemmy version that started tracking which user uploaded media. - Several improvements to federation code, which improve compatibility with wordpress, discourse, nodebb. - Fixing signed fetch for federation, enabling federation with instances that require linked instances to authenticate themselves when fetching remote resources. Not having this is something we’ve seen cause issues with a small number of mastodon instances that require this. - Site bans will automatically issue community bans, which means they’re more reliable to federate. - Deleted and removed posts and comments will no longer show up in search results. - Bot replies and mentions will no longer be included in notification counts when a user has blocked all bots. - Saved posts and comments will now be returned in the reverse order of saving them rather than the reverse order of them being created. - The image proxying feature has evolved to a more mature state. This feature intends to improve user privacy by reducing requests to third party websites when browsing Lemmy. We do not currently plan on enabling it with the update, but we will evaluate it later on. - Local only communities. We don’t currently see a good use for these, as they will prevent federation of such communities. This cuts off users on all other instances, so we don’t recommend using them unless you really want that. - Parallel sending of federated activities to other instances. This can be especially useful for instances on the other side of the world, where latency introduces serious bottlenecks when only sending one activity at a time. A few instances have already been using intermediate software to batch activities together, which is not standard ActivityPub behavior, but it allows them to eliminate most of the delays introduced by latency. This mostly affects instances in Australia and New Zealand, but we’ve also seen federation delays with instances in US from time to time. This will likely not be enabled immediately after the upgrade, but we’re planning to enable this shortly after. edit: added information about sync not showing comments on posts in local communities