The #python library #aiohttp uses #yarl for URLs internally, and yarl normalizes URLs by default. It silently decodes some %-encoded characters in the query string that do not strictly need to be encoded.
Sounds harmless, but it isn't. Changing the URL breaks any protocol that signs important aspects of a HTTP request for security.
Took me a while to find this bug. I usually expect an HTTP client library to not silently manipulate the URLs before sending a request. Smarter is not always better
Yes, this is documented behavior, but I rarely read the documentation for HTTP client libraries. I know how HTTP works, I know how those libraries should work, and silently altering URLs without any need is something I really did not expect from a low level HTTP lib.
Imagine #curl would silently change the order of headers or query parameters or remove an explicitly defined content-length header because it decides that chunked transfer is better for you. Stuff would break.
@glyph Does not help much if the HTTP client lib in question (aiohttp.client) wraps URLs in `yarl.URL` internally, which does the normalization by default. A design flaw if you ask me. I can only side-step the issue by wrapping all URLs with `yarl.URL(url, encoded=True)` myself to avoid this strange default behavior :/
Still bookmarking that, good URL libraries are hard to come by.
defnull
Glyph