chuang
Jonty WareingUser: you charge me when people make unauthorised requests to an S3 bucket?
AWS: yes of course
User: but
AWS: working as intended
User: but
AWS: thank you for your money
Matthew Flint
SpaceLifeFormYou should not be charged for inbound traffic that you DROP.
Alexander The 1st@SpaceLifeForm @jonty I'm half expecting a charge from AWS when you're behind on payments and AWS drops the API call itself with "S3 Bucket requests exceeded existing budget." messages.
That's basically the banking overdraft method, after all.
Peter Amstutz@jonty
I am trying to wrap my head around this. If a bad actor wanted to harass a person or organization that is running a service using S3, and can figure out bucket names, they could trivially make the target run up a bill of tens of thousands of dollars and there's absolutely nothing that can be done about it except begging AWS for mercy?
I am trying to wrap my head around this. If a bad actor wanted to harass a person or organization that is running a service using S3, and can figure out bucket names, they could trivially make the target run up a bill of tens of thousands of dollars and there's absolutely nothing that can be done about it except begging AWS for mercy?
Erik Ableson@tetron @jonty @clementd Follow-up tip. In regular usage any of these suppliers will be 5-10x less expensive than doing S3 from Amazon anyway. Unless your entire infrastructure is based on AWS there’s no good reason to ever use Amazon’s S3 service (and even then it’s debatable)
Additional option: roll your own with something like Minio, Scality, …
biggestsonicfan@jonty There are tools to "empty" all AWS services, but I had reoccurring monthly charges billed to me so I had to close my AWS account. I didn't realize at the time that permanently bans you from using AWS ever again.
LunaphiedIf you want to use the same email address for a different AWS account, we recommend updating it before closure. See Update the AWS account name, email address, or password for the root user for instructions on updating your email address.
What a joke of a company. Same statement for hardware 2FA tokens. Banned forever so change it first. That one makes even less sense!
They don't seem to actually want to ban the person just their account identifiers???
Also lol they have caps on closing member accounts of organizations how does that make any sense
@Lunaphied @jonty I barely checked the emails AWS sent me because 90% of them were useless to me. What makes them think I am going to constantly monitor a non-primary email address?! A gmail specific workaround is putting periods in your username or putting "+aws" after. So it's literally the same email, but not according to their database...
Fell@jonty Just get a regular old server guys. If one isn't enough, get two or three. It's not that hard. Cloud is a scam.
petur 😶🇺🇦🇵🇸🇹🇼@jonty LOL how do they still have customers??
Luka Rubinjoni@jonty Even bigger news: the backups ending up in the unfortunately named S3 bucket.
Momo@rubinjoni
Whoever I just linked that story to, I add "and read the entire article! The headline is NOT the best part of it!"
@jonty
Whoever I just linked that story to, I add "and read the entire article! The headline is NOT the best part of it!"
@jonty
Szaweł Trecz
Evan K@jonty The really messed up thing here is, if you use static web hosting from an S3 bucket, your bucket name HAS to match the domain through which it's served
🆃🅸🅼 🏴☠️@evanskaufman but then it’s a public bucket which falls into a different category of financial DDoS risk.
Dizzy@jonty Wow... I need to check this on Azure.
Gabriele Svelto [moved]@jonty this is the kind of stuff that makes you reevaluate keeping a server closet with a bunch of physical machines in it.
Daniel@jonty "As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket."
Toasterson (Till Wegmüller)@jonty Would the backup data be more valuable than the AWS price? Asking for a friend.
Jonas Köritz@jonty there is no real way to prevent this? Bucket names aren't exactly a secret...
This is ridiculous, at least unauthorized and 404ed requests should be exempt from billing.
This is ridiculous, at least unauthorized and 404ed requests should be exempt from billing.
IBBoard@jonty On the one hand, this is bad. People wouldn't expect those bills.
On the other hand, is this really materially different to a VPS or anything else with "included amount then overage charges" for bandwidth?
If someone keeps hitting your server with large request payloads then you're still going to get charged bandwidth whether you 403 it or not.
VPS firewall would limit it, but not prevent it.
Main difference is that this can be millions of 0 byte requests.
@jonty @ibboard most/all hosting we've seen doesn't charge ingress bandwidth. So you can at least theoretically block outbound transfers once you've used up your bandwidth (I say theoretically because there's no tools for strongly enforcing this so you're still gambling if you get hit by lots of request traffic).
The reason this is different is because AWS (and many others but AWS in particular is terrible about this) charges per-request too, and that is charged regardless of validity. The cost is tiny subtractions of a cent but because they're effectively free inbound, you just send a lot and boom. Massive bill.
@Lunaphied @jonty I think most hosts work on the assumption that a web server is receiving small requests and returning much larger payloads. But if you're accepting a constant stream of gigs of data then they're not going to be happy. Some companies explicitly list this as "your bandwidth is the higher of ingress or egress". Others will keep quiet until you become a problem.
Bandwidth still costs per request. It's just not recorded that way.
@ibboard @jonty yeah I think ingress is a little different though because they're more likely to consider sudden high traffic ingress as a DDOS and protect you if they offer that service or just cut you off.
Outbound bandwidth is expected though so it's just like "hope you don't fuck up your configs!"
@Lunaphied @jonty If I run an archive-only server where I upload terabytes of data per week but hardly download anything, would that be blocked as a DDOS? Because I'm well within the outbound bandwidth limits, but you won't get far claiming that your account should be fine because you're not downloading much.
AWS is just in the "accepts both inbound-heavy and outbound-heavy workloads and charges appropriately" camp. But here it's unexpected.
@Lunaphied @jonty But my point is, is it really that different a scenario?
Someone makes lots of requests into your service on any platform where the billing is by volume. You have a high volume, you get billed a lot, whether the service at the end of the bill-by-volume bit accepted the request or not.
I have no way of preventing people sending lots of data to my physical server with bandwidth limits. Just as there's no way to stop requests to S3 buckets.
@Lunaphied @jonty Is it RIGHT that AWS charges for denied requests as well as authorised ones? Generally not.
I expect they can tell the difference here and could choose not to charge (whereas a network provider can count bandwidth without working out if I accept it decline the traffic).
It's only reasonable if it came from your own account and was denied. And even then, I can't see a use case where you would intentionally leave something making millions of rejected S3 requests.
@jonty @ibboard yeah but this is flat out stupid. If their systems don't have anti-abuse code to handle enough failed requests that it's costing thousands then their systems are poorly designed. Absolutely they would not accept such a thing hitting their own usages if it were incurring real and actual costs like that.
AWS is a scam of a provider. They literally structure their entire service not just around making money (as is typical of a normal business) but especially around creating thousands of hidden fees which have only gotten worse and worse with time.
John Gordon
dbread*innocent configuration oversight*
"If all those misconfigured systems were attempting to back up their data into my S3 bucket, why not just let them do so? I opened my bucket for public writes and collected over 10GB of data within less than 30 seconds. Of course, I can’t disclose whose data it was. But it left me amazed at how an innocent configuration oversight could lead to a dangerous data leak!"
This is so infuriating. Not only that devs use some magic packets with default configuration and produce data loss, but also that the managers require the devs to produce results asap. That's how such mess happens. #devlive #agile #rapid #prototype
The Jack of Sandwich@jonty what surprises me us that you can give an S3 bucket such a simple unqualified name that it could match up with some default configuration somewhere.
It would be like finding out that "example com" was available to buy.
I would have assumed s3 buckets would at least need to have the account number attached to it. If I name a bucket "my-bucket", I'd assume anyone else could also have their own "my-bucket" in their own account
@jack_of_sandwich @jonty the global namespacing with user specified keys is the weirdest part of S3 compatible storage as a product in general IMO. I feel like it really exposes the "we made this to run Amazon services on" design
Codimp@jonty It's also an excellent example of why using names created on purpose for documentation is important. Never use real names in your documentation or default configuration!
Use names created for documentation.
And not only names:
- Domain names (RFC 6761)
- IP addresses (RFC 5735)
- etc
You write a documentation or example configuration? Don't use names, domains or IP addresses which are not created for this purpose!
Use names created for documentation.
And not only names:
- Domain names (RFC 6761)
- IP addresses (RFC 5735)
- etc
You write a documentation or example configuration? Don't use names, domains or IP addresses which are not created for this purpose!
RedstoneLP2@jonty
innocently googles "how to bankrupt company i don't like using AWS"
innocently googles "how to bankrupt company i don't like using AWS"
Angela Scholder@jonty I already much disliked AWS, but it clearly is even worse.....
Steve Loughran@jonty this is both insane , and utterly predictable.
Makes me wonder if unauthentic/403 requests made to a specific path will trigger throttling. Going to have to knock up a test for that.
Makes me wonder if unauthentic/403 requests made to a specific path will trigger throttling. Going to have to knock up a test for that.
Och Menno
Space Hobo@jonty So do any right-wing sites use S3 for storage?
Paul McO'Smith III@jonty oh, you've given everyone the best idea for a DDoS... i can bankrupt you bouncing up and down on your S3 with a bunch of zombie machines? oh, goodness...
kabelsalat@jonty Next thing you know, they'll add "calling us out on social media" as a billable item. Rip your wallet.
oparisy@jonty "Adding a random suffix to your bucket names can enhance security": I've seen that a lot, I thought this was to avoid name collisions, I'll know better now