PSA: You CAN'T use bunny.net as DDoS protection for Mastodon/Fediverse servers.

Their reverse proxy implementation quietly breaks Mastodon request signing, and support just got back to me and said it's intentional and they won't fix it.

https://github.com/mastodon/mastodon/issues/38119

I'm disappointed.

@lina Field names are case-insensitive, and RFC9110 allows a proxy to transform the message as long as its semantic meaning is unchanged.

So to me what they're doing is legal and reasonable.

Seems to me that Mastodon should be normalizing (including ordering, for example) before signing and before validating.

@ktims

It's not modifying the field name. It's modifying the field value (adding a space). The name being lowercased is fine (that's just an HTTP/2 thing AIUI).

The ActivityPub spec is here:

https://swicg.github.io/activitypub-http-signature/

And it references:

https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures-12

And that explicitly calls out proxies, but does not call out any specific headers that are problematic.

Unfortunately, since the servers that do the signing are the remote servers, this is not going to be fixed across the ecosystem any time soon. So regardless of who is "wrong" here, the status quo is Bunny is not usable for this purpose.

Ordering is already normalized and only a subset of headers are signed. The problem here is a header value is being modified.

ActivityPub and HTTP Signatures

@ktims If we are being spec lawyers, the spec says:

https://www.rfc-editor.org/rfc/rfc9110.html#section-7.7-7

And that does not provide for any leeway around non-semantic changes, it just says a proxy SHOULD NOT modify that header (again, the value, not the name, is the problem), so Bunny is breaking an HTTP SHOULD NOT.

RFC 9110: HTTP Semantics

The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230.

@lina You're totally right, my bad. I misread the report and thought it was calling out their normalization to lowercase names. Modifying the value is not acceptable.

I was trying to figure out if I was hitting this, because I'm pretty sure Caddy also normalizes to lowercase header names.

@ktims Yeah sorry, edited the comment to explicitly mention that. It's confusing.
@lina that signature in the screenshot really seals it πŸ₯²