so

if github had an outage

and was offline for 48 hours

what would catch fire?

if there was a bgp fuckup, and github was misrouted to anthropic, how many cicd pipelines would catch fire because claude is built in now?
if us-east-1 went offline for a week, what happens?
FBI warns Iran aspired to attack California with drones in retaliation for war: Alert

The warning came in a bulletin that was reviewed by ABC News.

ABC News

how many companies are you aware of that had "oh fuck someone actually blew up the datacenter" in their threat model or DR recovery efforts?

how many are like "whatevs, our shit is in the cloud and all the contract language has us well protected against lawsuits if shit goes down"?

how many do you think realize that attitude towards technology solves for "the lawsuits after the fact" but does absolutely zero for business continuity?

@Viss I tried to drill into my last group that Disaster Recovery mean you have a USB drive with an OS iso and brand new servers with nothing on them in a location that's no longer your lab.

And the local wiki on another drive with any security certificates and the build/release system needs to be built from scratch with just that.

@rsanders @Viss oh dang, yeah, pretty hard to restore things without the RootCA... Or the intermediate CAs...

@Viss

The best #ReplyGuy keeps saying, this is why they want to put them in space.    🌙 

@Viss Back in the day, I worked somewhere that had a data center approach for almost all the services that could deal with one of the sites abruptly vanishing if it came to it. And one of the biggest problems we had was with software vendors that couldn’t seem to conceive of this and struggled to implement measures we needed to support that.

The company wrote most of its own software, but boy, did we have to have some arguments about the stuff we didn’t write. Had one vendor - who we did not select for purchase after this behaviour! - very snottily tell us ‘you don’t need that’ when we asked about a feature that supported some data redundancy we DID very much need.

@Viss Of course, then came acquisitions and “the cloud” and a seeping away of people who knew and cared how to do stuff right. I expect what’s left of that environment today, if anything, is very fragile.
@Viss $lastco did have this in the dr plans.
Because the asphalt plant across the street had an accident years before that damaged property. So yeah DR site is out of country and plans have been tested with low trust that cloud services are available.
@Viss as far as I know, only my own, since two years ago I started our internal IR tabletop with this:

@Viss my favourite dr scenario included that the primary dc was under a flight path. and *all* of IT was within yards of that dc.
the COO could not wrap his head around why my planning included non-IT-folks levels of documentation in case, you know, plane+dc interaction.

still they went cloudy since then so it's all safe from harm and outages now /s

@Viss there are also the chaps with access to a US military base who stole four drones from a warehouse. Caught on camera. They’ve disappeared. FBI says “nothing to worry about”.

I’m normally in the “false flag is a conspiracy theory” camp but with these psychopaths in charge these days, at this point anything short of the U.S. launching an ICBM at Sacramento (because Newsom hurt his feelings) has a modicum of plausibility.

@Viss

Has Amazon invested in counter drone tech or point defence? 🤔🤷‍♂️

@Viss

I wonder if there will be in the not to distant future.

@Viss A week more than usual?
@Viss I mean for our dev crew it would be a PITA because of GitHub until all relevant repos would be transferred to our own Source Code Manager, but that is what we usually want anyway 😁 with digital sovereignty and all. But same goes for west-1 Ireland or Frankfurt, and then in Europe things prolly wouldn't be as fine

@Viss

nothing i care about? :)

@paul_ipv6 sure but does like, any automated shit in your house care? what about your phone carrier? what about your isp? your power company?
@Viss @paul_ipv6 If something depends upon github being available. then you don't own the IP. Microsoft does.
@rsanders @paul_ipv6 granted, but that wasnt the question i asked

@Viss

home, none. phone carrier, not much if my basic internet is working. ISP? possible though so far they seem to be doing their revision control mostly in house still. my power company only recently started offering electronic statements, so i'm probably safe there for the moment. ;)

work/customer related shit? probably a much uglier picture.

@Viss

i've had similar discussions more times than i'd care to about server OS maintenance. the number of under 40 yr olds who don't understand why trying to repro a system related issue when you just spin new virtual servers from the screaming newest externally maintained updated multiple times daily repos might not be a good way to see if it's your code or the OS that is screwing things up.

i have sometimes won the argument of at least doing everything from local repos and not updating the repo without regression testing but not nearly as often as i'd like or turned out to be the correct answer.

automation that depends on someone else's stuff working, not being unreachable, and not being impossible to revert to earlier versions is bound to be a problem at some point.

@Viss

hell, the idea of regression testing at all and not just wiping and leaping forward on new versions seems to be a dying concept.

@[email protected] @Viss Am I the only one wondering why if you rely upon external repositories for critical systems that you are not self-hosting your self-hosted and offline-tested mirrors?

I mean, seriously!?‽ For all of the supply-chain attacks in the past decade, how can you justify not changing your habits?!‽

@robert

self hosting and offline would certainly be my personal recommendation but not all companies or clients are willing to go that route.

@Viss
Why, you got a plan? I'm in.
@FritzAdalis of course i have a plan
@Viss so, so much would catch fire.
@Viss
Out-ogg-eye
@chocobo13 HAHAHA YES
AWW TOGG AYE!
@chocobo13 muhnoggay luggoggay caboggay awwtogay levroggay!