Mastodawn

abadidea Mar 2

I am sick and very out of it, as you might be able to tell by how I initially characterized a screenshot with a timestamp in March written right on it as February

DJGummikuh Mar 2

@0xabad1dea people killed by this war are probably taking a "leave of absence based on health issues"...

RolloTreadway Mar 2

@0xabad1dea I hope you feel better soon.

Greem (Graeme. Not Graham!)Mar 2

@0xabad1dea Sending good vibes your way. I hope you feel better soon.

Alex Mar 2

@0xabad1dea Sooo.. you had a calendar synchronization anomaly due to internal firmware instability? xD

abadidea Mar 2

if you want to @ me about how akshually it was probably a DRONE not a MISSILE, then no matter how good a grade you got in ballistic impact analysis, you're getting an F in The Point

StoneBear

@0xabad1dea How did the mansplainer die? He fell down a well, actually.....

Greem (Graeme. Not Graham!)Mar 2

@0xabad1dea At least OVH had the decency to call the Strasbourg fire a fire, not a thermal event.

sigi714 Mar 2

@greem @0xabad1dea Its called Cloud upload ;)

CounterPillow Mar 4

@greem @0xabad1dea
Once saw someone make fun of a Hetzner support notice for plainly stating that an issue was fixed by re-plugging a partially unplugged power cable, and honestly props to Hetzner for not reaching for euphemisms there.

Jernej Simončič �Mar 2

@0xabad1dea "… creating sparks and fire."

Claude Flammang Mar 2

@jernej__s @0xabad1dea
Sounds more like an angry GOD to me.

Sven Mar 2

@0xabad1dea if it was "literally struck by a ballistic missile" they'd not be talking about bringing it back up in a few hours...

NKT Mar 2

@HeNeArXn @0xabad1dea of course they would!

eestileib (she/hers)Mar 2

@Dss @HeNeArXn @0xabad1dea

Ballistic Missile just means it finishes its burn relatively early (contrast to Cruise Missile eg).

There are many sizes and payloads, they aren't all nuclear.

NKT Mar 3

@eestileib @HeNeArXn @0xabad1dea if it was a nuke, it wouldn't be back up any time soon. But it also wouldn't be from Iran. Israel or the US have nukes though. Neither are part of the worldwide nuclear monitoring program, unlike, er, Iran...

eestileib (she/hers)Mar 3

@Dss @HeNeArXn @0xabad1dea

If it had been a nuke we would all either know, or be dead.

abadidea Mar 2

@HeNeArXn presumably the missile was not aimed directly at the data center but at something nearby. If your neighbor is exploded off the face of the earth by a missile and your power generator is taken out by the kinetic impact, you have been struck by a missile, you’re just luckily alive enough to complain about it

Sven Mar 2

@0xabad1dea or its fragments from an intercepted missile or drone, or from something fired at an incoming missile, or bits from a plane, or ... and they might not even know yet what exactly it was. Hence, "objects".

https://mastodonapp.uk/@JdeBP/116155667315973528

JdeBP Mar 2

@0xabad1dea @HeNeArXn

The power generator was not taken out by any sort of impact, here, though. It was taken out by the fire department.

@wild1145
#UnitedArabEmirates #AWS #Amazon #CloudComputing #DataCentres #HansGruber

Ryan Wild Mar 3

@JdeBP @0xabad1dea @HeNeArXn Looks like the two UAE AZ's were due to drone strikes. AWS have updated their status at 4:19pm pst yesterday. It looks like it's the Bahrain regions az that was colatoral damage of drone strikes.

@0xabad1dea @HeNeArXn @vmstan
#AWS #CloudComputing #DataCentres

JdeBP Mar 3

@wild1145

I don't think that there's actually a contradiction here between what they said earlier and later. The root cause is the missile strike, whereas the proximal cause was the fire department shutting off the power in response to the fire that that caused.

It's an interesting thing for a data centre customer to consider: In the event of a fire, gas leak, explosion, or some such, is the fire department going to, as S.O.P., shut off, without any grace period, *both* primary and backup power?

Because my reading of what #Amazon first wrote is that that is what the #UnitedArabEmirates fire department's response was: shut off all power, mains and backup generators, immediately.

Hence #HansGruber. (-:

It's not unreasonable of the fire department, but it's also salutary to remember that there are going to be cases, to DR plan for, where it is actually procedure to have an unclean shutdown of all machines at once.

@0xabad1dea @HeNeArXn @vmstan

Ryan Wild Mar 3

@JdeBP Ah yes, I'd mis-read one of the earlier replies in the thread. I think part of why this is so noticeable is the AWS region is clearly a fairly small region, it's not super common now but in the newer regions especially having 3 Data centres mapping to 3 AZ's is not uncommon, but it does mean these sorts of things can have a much bigger impact.

This is the first time at least that I've found in AWS history that a region is significantly impaired due to effectively the loss of two geographically isolated AZ's (I don't remember from my time working there what the rules were around distance but they won't be exactly next door to one another!).

It'll be really interesting to see if everything can be recovered, given it's physical damage to the actual DC facilities there's likely to be a lot more work involved making the sites safe and getting it operational, it takes years to build these sites and they aren't exactly simple...

Hugo Mills Mar 2

@0xabad1dea "Impacted by Objects" is either my next band name, or doctors notes from the colorectal unit in the hospital.

velzard of jiraidon!!Mar 2

@darkling nobody to say it can't be both

- ice blue (he/him)

Тр3тий Сергеевич Mar 2

@0xabad1dea And I thought my government was acting cringe by calling gas leak explosion "a pop".

Stoneface Vimes Mar 2

@0xabad1dea so I guess if one burns down it'd be "influenced by a temperature event"

Andy Fletcher Mar 2

@0xabad1dea

Maybe they can knock out a few other data centres whilst they are about it. I'm open to suggestions.

v Mar 2

@0xabad1dea theres a kind of psych major who literally nerds out about this kind of thing. like "if you say x it creates y impression because z psychology reason" and i respect that but like sometimes....sometimes

cesium Mar 2

@0xabad1dea NARRATIVE FLOODGATES ACTIVATED

Viacheslav A Mar 2

@0xabad1dea This is a characteristic trait of authoritarian systems, which corporations inherently are. In Russian propagandists media they used to call explosion a "clap." No kidding, "a gas clap in residential building" has become a meme

Finn 🔜 #EF30 Mar 2

@0xabad1dea irl dos attack

Impeach||GTFO

@0xabad1dea tryin to get the insurance to pay.

Martin Hamilton 🔜 EMF Mar 2

@0xabad1dea When you think about it, a flooded data centre is really just a...

cloud migration

@m @0xabad1dea

Needed a grumpy-cat "NO" emoji.

CrazyDogLadySezPEACE Mar 2

@m @0xabad1dea badum *kish*! 🥁

💉💉💉Kumara📎Republic↙️↙️↙️Mar 2

Sounds like they're trying to avoid share prices tanking when these events happen.

74ミ [angelspit]

@0xabad1dea linkedin recruiter speak

@0xabad1dea I see a zone-name like "mec1" and I assumed it was in mecca. Looks like it was in UAE, though?

AWS Data Center in UAE Hit by Fire After Objects Strike Facility Amid Regional Tensions - EconoTimes

Amazon Web Services (AWS) confirmed that one of its data centers in the United Arab Emirates experienced a temporary power shutdown after objects struck the facility, causing sparks and a fire. The incident occurred at...

EconoTimes

Nicolás Alvarez Mar 2

@0xabad1dea @ferricoxide I think it means "Middle East central"

xssfox (crossy)Mar 2

@0xabad1dea maybe should have corrected that bucket policy

Rairii

@0xabad1dea ok now i'm imagining the opening strike of a war being on us-east-1

Colin Beveridge Mar 2

@Rairii We have always been at war with us-east-1.

Mark Lynch Mar 2

@icecolbeveridge @Rairii to be fair, if Iran did manage to hit us-east-1, would anyone believe the ensuing outage was actually war-related, as opposed to business as usual for AWS? 😜

Karl Mar 2

@0xabad1dea there's a word for that in every language, and English chose "cowardice".

Banda Bassotti Mar 2

All that effort busting unions, and a single strike still took out their data centre.

Angua

@0xabad1dea

Data Centre Owners: dysfunction due to anatomical misalignment.

Guillotine Jones, Flâneur Mar 2

@0xabad1dea
Well, objects (large and explosive) DID strike the Amazon data center.
What's yer point?
(Here I'm being sarcastic.)

The Witch of Crow Briar Mar 2

@0xabad1dea somebody accidentally deployed a J2EE application

starkraving666 Mar 2

@0xabad1dea no politics at Amazon!

Simon Zerafa Mar 2

@0xabad1dea

That's certainly a different issues from "a duck in a secure data center" 😣🤦‍♂️

Arne Dehli Halvorsen Mar 2

@0xabad1dea Any events of other kinds, like "rapid unscheduled disassembly", "kilo-Kelvin event", "accelerated oxidization event", "negative health alteration", "unhealthy substance ingestion", "head/body estrangement", "local high speed lead overdose", "descent into terrain" or anything else not necessarily in an advantageous direction?

Luna Lactea Mar 2

@0xabad1dea Did these "objects" also "undergo rapid, premeditated disassembly"?

Florian 'floe' Echtler Mar 2

@0xabad1dea Anyone remember the icbmto: URL format? 😉

https://en.wikipedia.org/wiki/ICBM_address

ICBM address - Wikipedia

@0xabad1dea

Lulz, two datacenters:

Mar 02 6:22 AM PST We continue to work towards recovery of the two impaired Availability Zones (mec1-az2 and mec1-az3) in the ME-CENTRAL-1 Region. We are expecting recovery to take at least a day, as it requires repair of facilities, cooling and power systems, coordination with local authorities, and careful assessment to ensure the safety of our operators. EC2, Amazon DynamoDB and other AWS Services continue to experience significant error rates and elevated latencies.

We recommend customers enact their disaster recovery plans and recover from remote backups into alternate AWS Regions, ideally in Europe. Further, we strongly advise customers to update their applications to ingest S3 data to an alternate AWS Region. We will provide an update by 11:00 AM PST on March 2, or sooner if we have additional information to share.

@0xabad1dea

A friend who has in-laws living in Dubai said:

some of the school's' remote learning apps don't work because local companies used local infrastructure.