Mastodawn

Nutomic Sep 22, 2023

Lemmy Development Update 2023-09-22

Lemmy Development Update 2023-09-22 - Lemmy

Some years ago we used to post weekly development updates to let the community know what we are working on. For some reason we stopped posting these updates, but now we want to continue giving you information every two weeks about the recent development progress. This should allow average users to keep up with development, without reading Github comments or knowing how to program. We’ve been working towards a v0.19.0 release of Lemmy, which will include several breaking API changes. Once this is ready, we’ll post the these changes in dev spaces, and give app developers several weeks to support the new changes. This week @nutomic finished implementing the block instance feature for users [https://github.com/LemmyNet/lemmy/pull/3869]. It allows users to block entire instances, so that all communities from those instances will be hidden on the frontpage. Posts or comments from users of blocked instances in other communities are unaffected. He also reworked the 2-Factor-Authentication implementation [https://github.com/LemmyNet/lemmy/pull/3959], with a two-step process to enable 2FA which prevents locking yourself out. Additionally he is reworking the API authentication [https://github.com/LemmyNet/lemmy/pull/3946] to be more ergonomic by using headers and cookies. Finally he is adding a feature for users to import/export community follows, bocklists and profile settings [https://github.com/LemmyNet/lemmy/pull/3976]. @dessalines is currently implementing a redesign of the join-lemmy.org website [https://github.com/LemmyNet/joinlemmy-site/pull/243]. He is also keeping the lemmy-js-client updated with the latest backend changes 1 [https://github.com/LemmyNet/lemmy-js-client/pull/184] 2 [https://github.com/LemmyNet/lemmy-js-client/pull/185] 3 [https://github.com/LemmyNet/lemmy-js-client/pull/181]. @phiresky optimized the way pagination is implemented [https://github.com/LemmyNet/lemmy/pull/3872]. He is also fixing problems with federation workers [https://github.com/LemmyNet/lemmy/pull/3960] which are causing test failures and performance problems in the development branch. These problems were introduced during a complex rewrite of the federation queue [https://github.com/LemmyNet/lemmy/pull/3605] which was recently finished, and is thought to allow Lemmy federation to scale to the size of Reddit. @SleeplessOne1917 is implementing remote follow functionality [https://github.com/LemmyNet/lemmy-ui/pull/1875], which makes it easy to follow communities from your home instance while browsing other instances. He is also fixing problems with the way deleted and removed comments are handled [https://github.com/LemmyNet/lemmy/pull/3965]. @codyro and @ticoombs have been making improvements to lemmy-ansible [https://github.com/LemmyNet/lemmy-ansible], including externalizing the pict-rs configuration, adding support for AlmaLinux/RHEL, cleaning up the configuration, as well as versioning the deploys. These will make deploying and installing Lemmy much easier. ## Support development @dessalines and @nutomic are working full-time on Lemmy to integrate community contributions, fix bugs, optimize performance and much more. This work is funded exclusively through donations. If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development [https://join-lemmy.org/donate]. Recurring donations are ideal because they allow for long-term planning. But also one-time donations of any amount help us. - Liberapay [https://liberapay.com/Lemmy] (preferred option) - Open Collective [https://opencollective.com/lemmy] - Patreon [https://www.patreon.com/dessalines] - Cryptocurrency [https://join-lemmy.org/donate] (scroll to bottom of page)

Show thread

fmstrat Sep 22, 2023

We’ve been working towards a v0.19.0 release of Lemmy, which will include several breaking API changes. Once this is ready, we’ll post the these changes in dev spaces, and give app developers several weeks to support the new changes.

Thank you for the update and the heads up.

How will this be announced, and what specifically does several weeks mean? Since Lemmy goes beyond Mobile Apps to all kinds of systems including moderation tools, auto-purgers, bots, CSAM, auto-subscribers, searchers, etc, breaking changes to the API can have far-reaching impacts.

Could something be set up specifically for breaking-change announcements where participants could be alerted? Even just a Breaking Changes issue that could be followed would work nicely.

Thank you again.

Show thread

Nutomic Sep 23, 2023

When we are ready to publish the first release candidate, we will make a post that lists all the breaking changes. You can follow [email protected] via rss reader to get notified about it. We will also share it in different Matrix chats, and I’m sure it will get upvoted to the frontpage as well.

Announcements - Lemmy

Official announcements from the Lemmy project. Subscribe to this community or add it to your RSS reader in order to be notified about new releases and important updates. You can also find major news on join-lemmy.org [https://join-lemmy.org/news]

Show thread

Xusontha Sep 28, 2023

Do you have any sort of timeline for when 0.19 or the release candidates will become available? I only wonder because I’m eager to check out some of the new features that have been mentioned here and on Github

Show thread

Nutomic Sep 29, 2023

See here: lemmy.ml/post/5711722

Lemmy 0.19 Breaking Changes - Lemmy

We are getting closer to the next major release. This version will have many breaking changes, so we are listing them here for app and client developers to adjust their projects. As we prepare for the release of Lemmy 0.19.0, we’d like to provide any app or client developers ample time to upgrade their apps, as well as discover any problems, before we do the release. This will be at least 4 weeks from now (but likely longer). Server admins can also upgrade to the latest release candidates for testing. Be aware that they are still unstable and shouldn’t be used in production. As with any upgrade it is important to have working backups in place. It should be possible for clients to support both Lemmy 0.18 and 0.19 without major workarounds. If backwards compatibility is causing you trouble, comment below and we will help to find a solution. To test, you can point your app to the following test instance running a release candidate of 0.19.0: https://voyager.lemmy.ml [https://voyager.lemmy.ml] A diff of API changes is here: lemmy-js-client API changes from 0.18.3 -> 0.19.0-rc’s [https://github.com/LemmyNet/lemmy-js-client/compare/0.18.3...0.19.0-rc.13] Note for developers not using typescript or rust: If you’d like to auto-generate an API client for your language, you can try out @MV-GH’s lemmy_openapi_spec [https://github.com/MV-GH/lemmy_openapi_spec], or (if in kotlin), use Jerboa’s script here [https://github.com/dessalines/jerboa/blob/main/copy_generated_types_from_lemmy_js_client.sh]. ## Major Changes ### Authentication Previous Lemmy versions used to take authentication as query/post parameters. This is insecure and unnecessarily complicated. With 0.19, the jwt token can be passed either as cookie with name auth, or as header in the form Authorization: Bearer . A major advantage is that this allows us to send proper cache-control headers, with responses to unauthenticated users being cacheable. It also prevents token leaks in web server logs. The login and registration endpoints attempt to set the cookie automatically. If that is supported on your platform, you don’t have to worry about the authentication token at all. In order for your client to be compatible with both Lemmy 0.18 and 0.19, you should send auth in both ways. Meaning with each API call, send the old auth query/post parameter, as well as the new header or cookie. A few PRs detailing these changes: - https://github.com/LemmyNet/lemmy/pull/3725 [https://github.com/LemmyNet/lemmy/pull/3725] - https://github.com/LemmyNet/lemmy/pull/3926 [https://github.com/LemmyNet/lemmy/pull/3926] - https://github.com/LemmyNet/lemmy/pull/3946 [https://github.com/LemmyNet/lemmy/pull/3946] - https://github.com/LemmyNet/lemmy/pull/3982 [https://github.com/LemmyNet/lemmy/pull/3982] ### Users can block instances Users can now block instances, so that their communities are hidden from listings. This is done via POST /api/v3/site/block with parameters int instance_id, bool block. https://github.com/LemmyNet/lemmy/pull/3869 [https://github.com/LemmyNet/lemmy/pull/3869] ### New sort options A new scaled sort option has been added. This sort is identical to the Hot sort, but also takes into account the number of each community’s active monthly users, and so helps to boost posts from less active communities to the top. https://github.com/LemmyNet/lemmy/pull/3907 [https://github.com/LemmyNet/lemmy/pull/3907] ### 2FA / TOTP Rework Two-Factor-Authentication is now enabled in a two-step process to avoid locking yourself out. Now a secret needs to be generated first with POST /api/v3/user/totp/generate (no parameters). The generated token needs to be added by the user to an authenticator app. Once this is completed, 2FA can be enabled with POST /api/v3/user/totp/update. This takes a string parameter totp_token (generated by authenticator app), and boolean enabled. 2FA can be disabled again with the same update endpoint. Additionally, the 2FA algorithm has been changed to SHA1 for better compatibility. The update disables 2FA for all accounts, so that users who are locked out can use their accounts again, and to ease the transition to the SHA1 algorithm. https://github.com/LemmyNet/lemmy/pull/3959 [https://github.com/LemmyNet/lemmy/pull/3959] ### Timestamps now include timezone Previous Lemmy versions used timestamps without any timezone internally. This caused problems when federating with other software that uses timezones. Going forward, all timestamps in the API are switching from timestamps without time zone (2023-09-27T12:29:59.113132) to ISO8601 timestamps (e.g. 2023-10-29T15:10:51.557399+01:00 or Z suffix). In order to be compatible with both 0.18 and 0.19, parse the timestamp as ISO8601 and add a Z suffix if it fails (for older versions). https://github.com/LemmyNet/lemmy/pull/3496 [https://github.com/LemmyNet/lemmy/pull/3496] ### Cursor based pagination 0.19 adds support for cursor based pagination on the /api/v3/post/list endpoint. This is more efficient for the database. Instead of a query parameter ?page=3, listing responses now include a field "next_page": "Pa46c" which needs to be passed as ?page_cursor=Pa46c. The existing pagination method is still supported for backwards compatibility, but will be removed in the next version. https://github.com/LemmyNet/lemmy/pull/3872 [https://github.com/LemmyNet/lemmy/pull/3872] ### New endpoints for export/import of user settings data Users can now export their profile settings data (including subscriptions and blocklists) via GET /api/v3/user/export. The returned JSON data should not be parsed by clients, but directly downloaded as a file. Backups can be imported via POST /api/v3/user/import. https://github.com/LemmyNet/lemmy/pull/3976 [https://github.com/LemmyNet/lemmy/pull/3976] ### Make remove content optional during account deletion When a user deletes their own account using POST /api/v3/user/delete_account, there is a new parameter called delete_content. If it is true, all posts, comments and other content created by the user are deleted (this is the previous default behaviour). If it is false, only the user profile will be marked as deleted. https://github.com/LemmyNet/lemmy/pull/3817 [https://github.com/LemmyNet/lemmy/pull/3817] ### Outgoing Federation Queue The federation queue has been rewritten to be much more performant and reliable. This is irrelevant for client developers, but admins should look out for potential federation problems. If you run multiple Lemmy backends for horizontal scaling, be sure to read the updated documentation [https://join-lemmy.org/docs/administration/horizontal_scaling.html] and set the new configuration parameters. The Troubleshooting [https://join-lemmy.org/docs/administration/troubleshooting.html] section has information about how to find out the state of the federation queues. https://github.com/LemmyNet/lemmy/pull/3605 [https://github.com/LemmyNet/lemmy/pull/3605]

Show thread

fmstrat

For the truly breaking changes like API auth and TOTP, is there a reason you don’t roll the deprecation like most software?

I.E. 0.19 supports both methods, and 0.20 deprecates the old one? This way developers aren’t caught off guard if they’re not following (which will get worse as time goes on), and allows development using official releases vs RCs.

For instance, if I want to update my app now, I have to release it with an RC library. If there was a version between deprecation, I could update at any point during the official 0.19 lifespan.

Show thread

Nutomic Oct 13, 2023

In case of pagination both old and new variants are supported in 0.19 (see my reply to fmstrat above). TOTP is currently broken so it wouldnt make sense to keep supporting the old version. In case of auth it would be possible to keep backwards compat, but keep in mind that we are only two fulltime devs with tons of other things to work on. If we spend a lot of time on this, it means less time for other important tasks. Besides you can support both 0.19 auth and 0.20 auth at the same time by sending auth as param and header/cookie.

Show thread

fmstrat Oct 13, 2023

Thanks for the response, I actually put another comment in after I started diving in and saw that the pagination/auth were overlapping, which was great news, it just didn’t come across clearly to me in the write-up for some reason. Thank you for structuring things this way.