Mastodawn

zeitwurm Nov 17

s0 Is Not A Sysadmin

This is a cry for help.

Has anybody on Fedi put OpenStack Swift behind an nginx reverse proxy successfully before?

All the docs seem to suggest it should “just work” but I get 401s on preflight checks to upload media through the reverse proxy, and it works fine direct. I’m forwarding the X-Real-IP etc and swift is recognising that.

My next step is to just futz around with the proxy forward headers in the hope that something will help, but I can’t find any docs to actually address this situation. It’s weird.

Also this is a legacy system running an old version from 2014, so that might contribute. The goal is to have the up-to-date nginx RP doing SSL termination so as to change as little as possible of the config in this legacy mess.

s0 Is Not A Sysadmin Nov 19

@julia @yottatsa @poundquerydotinfo @mr_daemon
Update:
I fixed the fucking thing this morning after two hours of fucking with nginx and swift.
The problem was that the poorly documented (zero examples) swift proxy server config directive cors_allow_origin, inexplicably, wanted the full base URL including scheme (“https://webserver.domain.blah”), not any of the values that were previously in there or tested, all of which were bare domains or hostnames (“webserver”, “webserver.domain.blah”, “mediaserver”, “mediaserver.domain.blah”).
Crucially, there was no log level that even hinted towards this; 401 vs 201 response was my only clue.

Setting strict_cors_mode to false did nothing at all.

Now I just hope this stupid clown era is over and there aren’t any more undocumented microservices in this pile of dev slop.

Also I found a bunch of private files including keys from other clients of this company, because apparently when they deployed on prem, they just copy and pasted their cloud server’s resource folder and edited settings in place.

I love this, so I Nov 19

@s0 *screams at last paragraph*

@s0 TIL, thank you for sharing! never used swift as a web-facing storage, so didn't expect this at all

@s0 (Assuming I'm understanding the issue you're having correctly):

nginx limits the size of requests by default to levels that block most uploads, so in the nginx config, in the server {} section and possibly location {} sections within (ie more than once), you'll need to add a line:

client_max_body_size 50M;

(Or higher if you foresee yourself uploading files larger than 50M)

s0 Is Not A Sysadmin Nov 17

@poundquerydotinfo I don’t think that’s the issue, as it’s the HTTP OPTIONS preflight request that’s returning 401 — this is done before attempting to submit a request with actual body data. Following that, the upload itself is also chunked into smaller PUTs that shouldn’t cause an issue.

mr_daemon Nov 17

@s0 I did it previously when my provider (ovh) did not have S3 compatible and just old crusty Swift (probably also from the same era tbh) available in the region where I ran my static content server.

Looking back at my then config in git history, the only special things were that I _had_ to disable ssl verification, and set the "Connection" and "Authorization" headers to an empty string ("").

Past that it was just proxy_pass to full endpoint, with proxy_intercept_errors off, and hiding some response headers (Access-Control-Allow-* stuff).

I also added the Access-Control-Allow-Origin header set to "*" in responses, but i don't think that matters here specifically

s0 Is Not A Sysadmin Nov 17

@mr_daemon yeah I’ve definitely already fucked with the access control allow headers. But yeah they’re a later stage problem

When you say changing the Connection and Authorization headers, were you running to swift with an internal stub SSL? My internal swift server is HTTP only currently but I could try enabling HTTPS on it and reverse proxying re-wrap into that ?

mr_daemon Nov 17

@s0 I did use ssl on the swift endpoint, but I don't imagine straight http would be an issue, as long as you also do `proxy_set_header X-Forwarded-Proto https`

@s0 i did (back in 2015). do you have full openstack setup (keystone + swith) or standalone one? is it s3 or switf API?

if it is swift api and full openstack setup, could it be that `X-Auth-Token:` is not passed because of `proxy_pass_request_headers off` or something along those lines (like token wasn't received at all)?

s0 Is Not A Sysadmin Nov 17

@yottatsa the successful ones don’t seem to have a token header at all? They have an X-Trans-Id (lol) and a temp URL and temp url sig. maybe I need to look at an even more pre-pre-flight request that gets a token for upload or something?

s0 Is Not A Sysadmin Nov 17

@yottatsa oh yeah it’s Swift API, hence the specific undocumented hell. I have no idea about keystone. Maybe? Clients get a JWT for the backend from an LDAP auth flow, I don’t know exactly how the rails backend issues the token for swift upload.

@s0 do you have access to the swift configs? would really like to take a look at `pipeline` (chain of middlewares) https://github.com/openstack/swift/blob/mitaka-eol/etc/proxy-server.conf-sample#L82 as this will explain the configuration.

my dogscience theory is: keystone (authentication/token generation/service discovery) solicites the swift endpoint which doesn't match the new swift-proxy endpoint behind, so the token is not passed by whatever client..

swift/etc/proxy-server.conf-sample at mitaka-eol · openstack/swift

OpenStack Storage (Swift). Mirror of code maintained at opendev.org. - openstack/swift

GitHub

s0 Is Not A Sysadmin Nov 17

@yottatsa yeah ok, that’s a great tip to follow up. I’ll collect some more info

@s0 also, do you proxy the URL without modification, or is there any URL rewriting/filtering in nginx (non `/` location, trailing url after `proxy_pass`, etc)?

s0 Is Not A Sysadmin Nov 17

@yottatsa nope, IRL preserved exactly

@s0 I had a problem with preflight requests for the S3-“””compatible””” mode of Swift a while ago and I tracked it down to a bug in Swift, but then it turned out our provider uses Ceph for the S3 API, so it was another bug, in Ceph. One of the bugs was that the middleware in charge of verifying the signature for pre-signed URLs was trying to verify it for the preflight request, which it shouldn’t, and will always fail because the request headers are not all the same.

@s0 I think this bug was the one in Swift, but I need to double check once I’m at the computer. I remember it was a bug affecting older versions of Swift, so it tracks.

@s0 I found the commit in our Slack. It’s specifically for S3 mode, though, so it might not be the issue you’re experiencing.
https://github.com/openstack/swift/commit/460dcf7562b7fa6d3244c28097ddd77287782274

s3api: Allow CORS preflights for pre-signed URLs · openstack/swift@460dcf7

Looks like browsers *do* send the query string in the OPTIONS request. Change-Id: Id10e6e32890f1c9a09c91990e5a6ee729bf4d973 Related-Change: I985143bf03125a05792e79bc5e5f83722d6431b3

GitHub