abadideaJune 2023: a Google data center in France floods and they call it a “water intrusion event”
March 2026: an Amazon data center in the Middle East is literally struck by a fucking ballistic missile in a hot war and they call it “impacted by objects”
Sven@0xabad1dea if it was "literally struck by a ballistic missile" they'd not be talking about bringing it back up in a few hours...
@HeNeArXn presumably the missile was not aimed directly at the data center but at something nearby. If your neighbor is exploded off the face of the earth by a missile and your power generator is taken out by the kinetic impact, you have been struck by a missile, you’re just luckily alive enough to complain about it
JdeBPThe power generator was not taken out by any sort of impact, here, though. It was taken out by the fire department.
https://mastodonapp.uk/@JdeBP/116155667315973528
@wild1145
#UnitedArabEmirates #AWS #Amazon #CloudComputing #DataCentres #HansGruber
Ryan Wild@JdeBP @0xabad1dea @HeNeArXn Looks like the two UAE AZ's were due to drone strikes. AWS have updated their status at 4:19pm pst yesterday. It looks like it's the Bahrain regions az that was colatoral damage of drone strikes.
I don't think that there's actually a contradiction here between what they said earlier and later. The root cause is the missile strike, whereas the proximal cause was the fire department shutting off the power in response to the fire that that caused.
It's an interesting thing for a data centre customer to consider: In the event of a fire, gas leak, explosion, or some such, is the fire department going to, as S.O.P., shut off, without any grace period, *both* primary and backup power?
Because my reading of what #Amazon first wrote is that that is what the #UnitedArabEmirates fire department's response was: shut off all power, mains and backup generators, immediately.
Hence #HansGruber. (-:
It's not unreasonable of the fire department, but it's also salutary to remember that there are going to be cases, to DR plan for, where it is actually procedure to have an unclean shutdown of all machines at once.
@0xabad1dea @HeNeArXn @vmstan
#AWS #CloudComputing #DataCentres
@JdeBP Ah yes, I'd mis-read one of the earlier replies in the thread. I think part of why this is so noticeable is the AWS region is clearly a fairly small region, it's not super common now but in the newer regions especially having 3 Data centres mapping to 3 AZ's is not uncommon, but it does mean these sorts of things can have a much bigger impact.
This is the first time at least that I've found in AWS history that a region is significantly impaired due to effectively the loss of two geographically isolated AZ's (I don't remember from my time working there what the rules were around distance but they won't be exactly next door to one another!).
It'll be really interesting to see if everything can be recovered, given it's physical damage to the actual DC facilities there's likely to be a lot more work involved making the sites safe and getting it operational, it takes years to build these sites and they aren't exactly simple...