I'm using Crystal's stdlib as example of real-world implementations of structures and algorithms. The advantage of a self-hosted language that has a slim syntax :-D

#crystallang

The biggest change in release v3.4.0 of Ktistec is cursor-based pagination for all web-navigable collections (timeline, notifications, etc.). Offset-based pagination will be removed completely in the next release.

Offset-based (e.g. page/size) pagination works well on collections that don't change. But, what does "the second page" contain in a dynamic timeline? Support for cursor-based pagination is required by the Mastodon-compatible API, but has been a desirable feature for quite a while.

While updating queries to paginate by cursor, I also made performance improvements to the queries themselves, as mentioned elsewhere. Scrapers and bots have already adapted—sort of. I now see odd hybrid requests in the log like /tags/xyz?page=7&min_id=123. Overall CPU usage under normal load is now sitting at 0-1%.

Here is the full changelog for the release:

Added

• Cursor-based pagination for web-navigable collections. (fixes #122)
• Mastodon-compatible API: /api/v1/timelines/tag/:hashtag endpoint.

Fixed

• Negative replies count when viewing a post that is also a reply.
• Order cached actors' posts by published rather than id.

Changed

• Report 401 and 403 as distinct errors in Ktistec::Network.get.

Removed

• Unused paginated query methods.

Enjoy!

#ktistec #crystallang #activitypub #fediverse

Release v3.3.9 of Ktistec continues the security hardening work from recent releases, with further progress on the Mastodon-compatible API.

Of note: all network connections now go through a new Ktistec::Network module. This allows Ktistec to limit the size of HTTP bodies it reads, on both inbound and outbound requests, and ensures it only opens connections to valid remote IP addresses.

Here's the full changelog:

Added

• New Mastodon-compatible APIs.

Fixed

• Close DNS rebinding window for outbound HTTP requests.
• Limit the size of HTTP bodies the server reads.
• Sanitize RSS feed output to prevent CDATA breakout.
• Destroy all sessions and access tokens on account termination.

Changed

• Ensure all GET and POST requests utilize Ktistec::Network.
• Process local recipients in-process in inbox/outbox activity processors.

As always, it's worth upgrading for the security fixes!

#ktistec #crystallang #activitypub #fediverse

Crystal's interpreter is a pretty nice one for a compiled LLVM language, but I kinda wanted it to be a bit more robust and feel less like it wasn't an integral part of the whole thing. Like it is in Common Lisp.

Few million tokens later, I have a prototype of something like that. A JIT interpreter that is trying to reach feature parity with the default AOT compiler.

Probably will never be good enough for a PR since I don't understand half of it, and I'm not interested in submitting "slop" like that. But I'll probably keep the little demons maintaining the branch for my own amusement.

Also it should eventually (not quite yet) do hot reloading, which should enable dynamic modules written in Crystal. Not without caveats obviously, because Crystal has carved some pretty hard architectural decisions to get the kind of performance and type discipline they have from a Ruby basis, and I'm not changing those.

Branch is easily found if you care that much, but because of the slop aspect, I'm not directly advertising the link.

#crystal #crystallang

This release continues my focus on security instead of new features. As I wrote earlier this week, I rebuilt the template framework Ktistec uses with type safety as a central principle. What does that mean?

Imagine that you have an instance of a String that holds federated data. Where can you safely render that in a browser, and what operations (sanitization, escaping, etc.) do you need to do first?

The only way to answer that is to look carefully at the lineage of that data: where it came from, how it was stored, how it was transformed, and where it's rendered. A name holds text; an href or src attribute holds a URL. If you want to render a name inside an HTML element you should HTML escape it. You should escape href and src, too, but the escaping rules for URLs are slightly different from the HTML rules. It's easy to make mistakes.

Ktistec uses four "safe" types to express the contracts:

SafeHTML: A String wrapper marking HTML markup safe to emit raw into HTML data slots (text content, between tags).

SafeAttrValue: A String wrapper marking a value safe to emit raw inside a double-quoted HTML attribute (attr="..."), other than URL or event-handler slots.

SafeURI: A String wrapper marking a URL safe to emit raw into a URL attribute slot (href, src, action, etc.).

SafeJSON: A String wrapper marking JSON output safe to emit raw into the body of a <script type="application/json"> block.

Using the wrong type at a call site is either a compile-time error, or it triggers automatic sanitization of the underlying string value.

Here's the full changelog:

Added

• String safety framework with typed "safe" strings.
• New Slang template engine with compile-time safety checks.
• Vendored WebFinger and HostMeta client shards.

Fixed

• Prevent delivery to unknown IRIs.
• Narrow Like/Dislike addressing to the liked object's author.

I have at least one more cleanup pass to do, and then I'll turn my attention back to the Mastodon-compatible API and a few features I've been looking forward to—like scheduled posts.

#ktistec #crystallang #activitypub #fediverse

One of the nice benefits of working on an open source project is that you can scratch an itch for as long as you feel like scratching. A game I like to play while scratching is called "what invalid states can I make unrepresentable" using the type system.

ktistec uses a template language for views and partials. In its original form, it allowed a programmer to use = to escape an untrusted value or == to render it unescaped. You might want to escape an actor's name property because a name should never contain HTML but you might want to sanitize an object's content property and then render it without escaping because the body of a post can contain HTML.

The problem is you have to remember the rules and never make a mistake. If you accidentally type == actor.name you've just created a potential cross-site scripting (XSS) vulnerability!

ktistec's template language now makes it much more difficult to screw up.

There's now only = syntax and by default it escapes everything. The only way to get it to emit a string without escaping is to wrap it in SafeHTML. The sanitize helper sanitizes HTML and then wraps it for you. Other common helpers (e.g. path construction helpers) do the same.

Importantly, if you interpolate a safe value into a string, it is demoted back to a string and will be escaped unless it is explicitly wrapped again.

Unescaped HTML is still possible to construct, but it's now much harder to do so accidentally. You can't just concatenate some strings together—forgetting that one comes from an untrusted source—and render that as unescaped HTML.

#ktistec #crystallang

Release v3.3.7 of Ktistec fixes several bugs and introduces two enhancements.

Security is a focus in this release. Every gap in input sanitization or escaping is a potential vulnerability, and I've been systematically closing them. I am also carefully, and maybe conservatively, restricting things like supported URL schemes and uploaded file types.

The two enhancements improve compatibility with Mastodon-compatible clients. Mastodon's OAuth tokens don't expire, and Mastodon clients don't know how to handle tokens that do. Sliding expiration ensures that tokens in active use stay alive, while unused tokens eventually expire.

Here's the full changelog:

Added

• Sliding token expiration for OAuth2 access tokens.
• Mastodon-compatible API: /api/v1/accounts/update_credentials endpoint.

Fixed

• Prevent pinning of (and auto-unpin) private objects.
• Don't save a quote if the quoted actor cannot be dereferenced.
• Fix rendering of federated actor profile attachment values.
• Remove href attributes with unsafe schemes from sanitized HTML.
• Escape interpolated values in view helpers and the actor icon streaming refresh.
• Restrict upload extensions and serve uploads with X-Content-Type-Options: nosniff.
• Escape publicKey and scrub Tag.href.
• Sanitizer no longer permits single-quote attribute injection.
• Ensure bearer-token sessions cannot reach the web UI.
• Require client authentication on the OAuth token endpoint.

I'm working on performance improvements for the next release. A rewrite of the Slang template library looks like it will cut both build time and executable size by around 10%!

📡 Stay tuned!

#ktistec #crystallang #activitypub #fediverse