Just received an email on how to recover machine bricked by #CrowdStrike.

Our IT department (large Japanese corporation) is literally telling users to retreive BitLocker keys, launch CMD from recovery menu and completely nuke CrowdStrike kernel modules from C:\Win\Sys32\drv\CS.

This shit is absolutely hilarious, wish I had bought popcorn
@elly I'm so glad I use #Linux exclusively because that shitshow would've made me quit #tech entirely...

@kkarhan @elly If everyone would use Linux, there would be crowdstrike-like corporate malware as well, with our systems failing the same way.

We, Linuxers, were only saved by the fact that we're exotic.

@Zugschlus @elly I sincerely disagree because on #Linux there is neither need nor demand for #Scareware like #CrowdStrike and even then noone should allow such garbage in their systems as a #rootkit!

Matthew Garrett (@[email protected])

"Linux would have prevented this!" literally true because my former colleague KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen, so everyone please say thank you to him

Nondeterministic Computer
@kkarhan @Zugschlus @elly
"CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed"
https://mastodon.radio/@[email protected]nsky.name/112820180849984453
Hacker News 50 (@[email protected])

CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed Link: https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/ Discussion: https://news.ycombinator.com/item?id=41018029 #linux

Mastodon

@WB2EEE @Zugschlus @elly Which only applied to a small subset of machines and almost all #Linux #sysadmins know to properly test stuff or at the very least have proper #backups, use #snapshots to test stuff and be able to #backroll stuff if not quickly setup stuff easily.

  • Again I blame both #CrowdStrike and #Microsoft because the first one made a #Rootkit and pushed it without proper #QA #testing and the latter one signed that shit and build an OS that doesn't easily allow for reproduceable and automated system installations!

Further proving my point in that the amount of issues would not have happened on Linux!

Matthew Garrett (@[email protected])

@[email protected] my understanding is that that affected people still using the old kernel driver (eg, if your os is too old to have the new ebpf hotness)

Nondeterministic Computer
@kkarhan @WB2EEE @Zugschlus @elly I'm a Linux fanboy but what you're saying are half-truths mixed with non sequiturs to make it seem like this issue could never happen on Linux. What happened to Windows/CrowdStrike can happen to _any_ system. Full stop. The only reason it didn't affect every Linux system is because the ecosystem isn't homogeneous. If they had all been on the same version that crashed, then **every single one** would have crashed. Good admin practice isn't exclusive to Linux.

@Cappyjax @WB2EEE @Zugschlus @elly

I disagree with your assessment to some extent, as on #Linux there's not much of a tolerace or even acceptance of #rootkits like #CorwdStrike for all the right reasons!

  • #Scareware like that has no legitimate reason to exist and the only place I've seen that is on #Windows because people distrusting #WindowsDefender and thus #Microsoft are unwilling to to what is evidently the correct solution and yeet Windows for good.

Cuz it's not about #Fanboyism, but a fact that most Windows - #Sysadmins (or rather decisionmakers for that matter) seem to be high on #Copium and refuse to take actual #consequences but accept the #Enshittification as a fact of life when it's not...

  • Or am I the only one who remembers the rightful backclash when #ads were shoved into #Ubuntu whilst on #Windows11 and #Windows10 people just seem to shrug their shoulders?

It's just that this entire shitshow would not have happened on #Linux because here we don't have an entire ecosystem of #ValueRemoving "#AntiVirus" bs being shoved on Desktops and Servers in the form of #Binaryblob #Kernelhacks!

@kkarhan @Cappyjax @WB2EEE @elly If your company's policy tells you to install that stuff, then you install that stuff or are out of a job. In sad reality, auditors expect some kind of "endpoint protection" to give you the compliance certificate that the company needs, , and most companies decide to buy that instead of implementing it youself.

And it is also in your "best" interest to accept that as a system administrator. If the bought software fscks up, people shrug it away and continue (including continuing to use said software). If your home-built solution fscks up, you're at least in for some very uncomfortable appointments in your own C-suite, if not immediately out of a job.

That's sad reality, and I regret writing that. But.

@Zugschlus @Cappyjax @WB2EEE @elly well, I'd rather not take or stay in a job than commit what I call "Professional #Malpractice"!

  • I know this makes me an outlier, but the fact that I did my job so well that everything I deployed runs like clockwork to this day amd that I'm not short of offers tells me that being a honest #sysadmin is the way to go morally instead of being a #bootlicker!

Again: We have this entire shitshow because we allow #TechIlliterates and other dipshits to make up regulations on the spot.

  • Also yes, there are means to harden #Linux on Sesktops amd Servers beyond the already existing #CommonCriteria and #CIS2 as well as beyond #PCIDSS compliance and good Distros will even offer a warranty and assurance for that directly - something #Microsoft just won't do for #Windows no matter the amount of money one shoves down their throat!

The fact that we even allow that #Govware and #Scareware [to even exist, espechally] in #CriticalInfrastructure when in both cases their #EULA explicitly bans that use-case is a testiment for the false priorities of regulators and their rules.

  • So yeah, if a concrete-headed #TechIlliterate wants that they can have it - but not from or with me!

And then they all whine about why noone wants to work for them... What a shitshow.

Tell you what, I'd rather welcome such meetings, because the last time some CEO did that (with an absurd office mandate forcing a colleague into a 500km [one-way!] commute twice a week) they basically mobbed out the two best colleagues I had and subsequently imploded the Linux Infrastructure team.

  • Last time I checked that company hadn't filled the vacancies and once Recruiters hear the story, they tend to fire said company as a client.

#Sarcasm #venting #CrowdStrike

Marc Haber (@[email protected])

@[email protected] @[email protected] @[email protected] @[email protected] If your company's policy tells you to install that stuff, then you install that stuff or are out of a job. In sad reality, auditors expect some kind of "endpoint protection" to give you the compliance certificate that the company needs, , and most companies decide to buy that instead of implementing it youself. And it is also in your "best" interest to accept that as a system administrator. If the bought software fscks up, people shrug it away and continue (including continuing to use said software). If your home-built solution fscks up, you're at least in for some very uncomfortable appointments in your own C-suite, if not immediately out of a job. That's sad reality, and I regret writing that. But.

Zug.Network
@kkarhan @Cappyjax @WB2EEE @elly You're very lucky that you seem to be working for a company that does not need to subject itself to a compliance audit and I wish you that it stays that way for as long as possible.

@Zugschlus @Cappyjax @WB2EEE @elly You are mistaken to some extent:

I just evidently exceed any requirement there is and I've yet to find a case where this was impossible.

  • Regardless if telco, healthcare or finance.

Cuz notwithstanding any NDAs I can assure you that me using #Linux is not the problem, but if there is one, it's any #Windows machine in their networks.

  • But those ain't my problem because I don't get paid to give a fuck about those!

@kkarhan @Cappyjax @WB2EEE @elly

The OS is not the problem. It's the corporate policies. And commercial snake oil, eh, corporate malware, eh, security software suites.

All that compliance stuff drives the fun out of our work.

Just to make sure that you don't see me as a Windows fanboi: I am an active Debian Developer and have been for two decades.

@Zugschlus @Cappyjax @WB2EEE @elly I disagree to some extend as Windows is in fact part of the Problem cuz on linux this would not have happened...

  • OFC allowing #TechIlliterates in decisionmaker positions (regardless if regultors or corporations) is at best wrong and IMHO should he as illegal as hiring a freshly convicted (wire-)fraudster as CFO of a bank.

The entire #Scareware / #DigitalSnakeoil industry should be outlawed as the fraudulent business model it is and security should be procured from the maintainers of the OS only...

Matthew Garrett (@[email protected])

"Linux would have prevented this!" literally true because my former colleague KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen, so everyone please say thank you to him

Nondeterministic Computer

@kkarhan @Cappyjax @WB2EEE @elly

A lot of IMHO and SHOULD. Reality, sadly, is different.

@Zugschlus @Cappyjax @WB2EEE @elly I'm doing my part towards that goal by being intolerant to intolerable bs...

Even if that may seem not very wise...