Hey fellow #Linux users, despite the #CrowdStrike problem only affecting #Windows, this is not a windows problem.

This is an "automatic update that got forced onto everyone with insufficient testing while also having way too many permissions" problem.

If you think big corps wouldn't run something similar on Linux, I have a an NFT of a bridge to sell you.

Boot fails with "vmlinuz has invalid signature" or "bad shim signature, you need to load the kernel first" · Issue #543 · fedora-silverblue/issue-tracker

Current workaround See #543 (comment) Original issue text Describe the bug Trying to rebase an existing SB39 to SB40 fails to boot showing vmlinuz-6.8.1-300.fc40.x86.x64 has invalid signature. you ...

GitHub
@ljrk @ainmosni that is microsoft/uefi/bios issue - not fedora
@xbezdick @ainmosni Putting the blame for a defective update to fedora resulting in a failed boot... on Microsoft is surely a take one can make.
@ljrk @ainmosni Microsoft UEFI CA was revoked (so partial blame there), firmware on machines didn't get the new cert chain and Fedora is finally trying to get rid of the old CA signature. Or I understood it wrong?
@xbezdick @ainmosni As @ArcaneAlchemist said: The breakage was due to the bootloader not receiving updates, even if the rest of the OS depended on this. Yes, in this case the update was necessary in the first place due to external reasons but that's completely irrelevant to the fuckup of not updating the Bootloader.
@xbezdick @ljrk @ainmosni that was a Fedora silverbluenissir. And I’ll stop here I will not be nice.

@ainmosni

I've already collected one anecdotal response.

https://mastodonapp.uk/@JdeBP/112813114562289051

JdeBP (@[email protected])

All of the Linux people being smug don't know that #CrowdStrike's Falcon Sensor has a Linux version, that integrates into the kernel, and has "channel files" too. They have escaped through accident, not by dint of any inherent superiority. In another universe where Linux systems were instead deployed in the many businesses/public services/governments with CloudStrike as the common anti-malware choice, today would have been Linux panic day. https://crowdstrike.com/press-releases/crowdstrike-falcon-expands-linux-protection-with-enhanced-prevention-capabilities/ #NeverWindows11 #MO821132

Mastodon App UK
@JdeBP Yeah not surprising at all. This kind of stuff is all too common in certain companies.
@ainmosni Not a Linux problem, but absolutely a failure of the proprietary and software as a service models. If this were an open, in-tree Linux driver, it would likely go through a much more robust review and test cycle as well as not be simultaneously pushed to every Linux system in existence using it. It would be filtered through multiple distros' testing and release processes which would make discovery of the bug much more likely.
@CalcProgrammer1
If this was part of the OS, sure, but this is third party stuff that the corps installed themselves. Again, they would do exactly the same on Linux, and as replies to my post show, CrowdStrike is common on Linux as well.
@ainmosni I agree, the root issue here is giving corporations permission to install proprietary code into your kernel that you can't review or fix should it go wrong.

@CalcProgrammer1 @ainmosni the computers belong to the corps/orgs. Why should you have to give them permission?

The root cause feels like a lack of change management by the corps/orgs.

@puck @ainmosni In this case it's corporations giving other corporations said access, still, allowing third parties free access to your kernel isn't a good idea no matter who you are.

@CalcProgrammer1 @ainmosni totally agree.

But unfortunately for these kinds of tools to work, they need that level if access. What shouldn't be allowed is pushing out automatic updates.

(Personal gear is different, but I also hate to think the fall out of non-technical home users having to recover)

@ainmosni I don't think it's totally a "Modern Microsoft problem"

But I think there's a case that at least partially the fault of late 90s/early 2000s Microsoft, who were more than happy enough to ignore security until it became too big a problem to ignore, creating the environment for a swathe of slightly dodgy AV vendors to grow into?

@ainmosni Cant get hit by dodgy updates if you keep forgetting to update 🧠
@ainmosni I spent 40 years in the IT industry including 10 as an industry analyst. I'm well versed in modern platforms and CI/CD. Yet, this suprises me. Given how widespread this is, it seems that cursory testing on the part of CrowdStrike, it's customers, and Microsoft should have caught this.
Much of tbe blame is the fire and forget mentality of cloud services. Customers of these services need to test before rolling out anything new and not rely on the vendors for testing

@DJMRP

I am both amazed and not-surprised at all that this happened.

At this scale, they should have had a pretty advanced multi-tier, multi-tier rollout system.

But instead it seems they committed the YOLO-est of all YOLO deployments.

@ainmosni @DJMRP

You want a hot one unrelated to Crowdstrike?

Dayforce Integration with AD gets turned on, cool, this makes SSO work. Dayforce Integration starts overwriting AD user accounts with their email address so xyjones became x.jones, and nobody can login to anything for 30 minutes.

All because someone thought they were in test and not prod.

@DJMRP @ainmosni Don't think that is possible with any EDR product. One of the selling points is near-realtime protection against emerging threats and there are dozens of updates every day.

Not saying this is great, just pointing out that "test every update" is not feasible for 99.9% of orgs.

@tribut when your Bank of America or a major airline it is. Let's say it's arisk you have to take, why not roll ot out to a small number of servers first and limit the blast area? Why not engineer a way to recover more quickly? The answer, IMHO, is money. I would like to see something in the financial filings of these companies that says "there is an ongoing risknof computet disruption due to our choosing not to test updates from our IT vendors."
@ainmosni take off; they're too busy hammering us with useless tenable scans and misinterpreting the results ;o)
@ainmosni Only if the NFT is limited edition!
@ainmosni this is not an "corpo" thing, the big problem here is that, this shouldn't happen, this means that all the inside infrastructure is not working, and their shadow security thing is not doing well, a company having kernel level access to any computer with your software is not a good security measure, and not having a comprobation pipeline only makes it worst.

if you want to see how linux and companies can work together by doing an open solution. that has all the comprobationes needed to make it secure. just see SUSE, a company making profit by making secure systems, fast, and all the things without this kind of security problems, i already mentioned this case:
https://securelist.com/trng-2023/ on my toots, where kaspesky got hacked getting root access to all the company memebers iphone's. something has to change, neither microsoft making a better software or companies reliying on better companies to mantain their infrastructure software. but this is not a "anti-corpo pro linux thingy". this is a serious problem in how microsoft handle their software.
Operation Triangulation

We discovered a previously unknown mobile APT campaign targeting iOS devices. We are calling this campaign "Operation Triangulation"

@ainmosni

Problem itself is certainly not the fault of Windows, but I think such a great stoppage in the critical operation is.

On Linux rollback using Btrfs snapshots is something common. You can uninstall faulty program, rollback to older version and pin it to prevent updating without even booting the system from the disk. There is also a world of immutable distros, which propably would just boot to previous (pre-update) slot and show notification.

Meanwhile entering Windows recovery mode for a temporary fix looks like that: https://101010.pl/@didek/112812731137102264

Dawid Rejowski (@[email protected])

Attached: 1 image Now is a great opportunity to remind everyone of Microsoft's official instructions for entering Windows recovery mode. Or restarting a broken washing mashine. #Microsoft #Crowdstrike

101010.pl

@didek And yet, if you give an incompetent corp root/kernel access, all those things won't help depending on the error.

Sure, this exact problem would be easy to fix, but what if it caused data loss? The OS is not worth much if all the user data is gone.

I really prefer linux over windows, but this problem wasn't caused by windows.

@ainmosni

I see we both agree. Again, I also think the problem is not Windows fault at all.

Just wanted to point out some of Windows related issues showing their relevance alongside all of this.

@didek @ainmosni Windows installations are on average more susceptible to people selling bad software. This due to both causation and correlation.
@ainmosni For data loss, the solution is easy: "backup before update", which always should be done especially with seriously needed devices.
@ainmosni truth. I already got screwed on ubuntu 22 lts by updated nvidia drivers that would crash randomly, freezing my display and making it impossible to recover cleanly.
@ainmosni I mean if anyone uses linux as their main, they know how kernel updates can break the entire OSes ability to do certain things. This is an automatic, forced update, to something that has core control of so much stuff inside of the operating system of a lot of critical infrastructure as a security measure. Sure you can leave your headless Debian server not updated for years and it'll be just fine, but with this software, you dont have that choice, on windows or linux or macos or anything. It just does it. And you have no say. It could have easily affected linux computers like this instead of windows. I mean yeah you could have disabled automatic updates, but you'd fail any security audit that came your way.
@RedCyberPandaz Not if you're using an immutable distro, you can always boot with the last working image.
@joe9nf even in the niche of linux, immutable distros are a niche of a niche. You're just splitting already split hairs at this point.
@RedCyberPandaz
1. Niche or not, it works, it is more secure and more reliable compared to the common OS model we have now.
2. It's the future of desktop, server and mobile OS.
Android and iOS are already immutable, Fedora, nixOS, OpenSUSE, Ubuntu and others already started it for the Linux desktop and server, Apple is already making MacOS immutable gradually, Linux mobile OSs like PostmarketOS are planning to become immutable too.

@ainmosni the way tooany permissions problem you mentioned - is a Microsoft problem

It's the way Windows designed it to work, to promote Defender and make it unnecessary more difficult for commercial products, and mostly impossible for open source - to tackle the functionality of Defender on Windows

Perspective of shared responsibility, of course - just to be clear, the permissions problem is not really a choice of CrowdStrike

@stoff

Note that I'm saying this as a big fan of linux with decades of experience.

Run anything on linux as root, and it has too many permissions. This is not a windows problem.

Sure, there's SELinux, but people find that "too difficult", so many turn it off.

I'm all for smack talking windows, but in this case, the problem is CrowdStrike.

@ainmosni @stoff Yes, but people making Linux decisions generally have a higher degree of technical knowledge than people making Windows decisions. Many more Windows decisions are made by clueless managers. That happens with Linux as well, but to a lesser degree.

@ainmosni

You're completely right.

The problem is with the model of automatic and forced updates, where users have no process of "acceptance testing" (and no practical mechanism to roll back if a version fails the test).

Unfortunately, I see it creeping into the world of Linux users, too.

For example, Ubuntu seemed pretty good about 10 years ago when I started using it, but it migrated to a model of forced updates (also switching to the snap package manager, which also brings with it some other problems). (I ran the same version of Ubuntu for 10 years before switching to LMDE6 this year.)

@johnlogic @ainmosni Switched from Ubuntu to Mint, and now using antX, since all my computers are old. Yeah, if I want an update, I will use the terminal.

@ainmosni I've had this exact thing happen with CrowdStrike Falcon on a couple of Linux servers last year, where a buggy CS update caused their kernel module to randomly corrupt kernel memory, resulting in occasional crashes.

So yes, not an OS problem, but a problem with this particular class of security software that has its tendrils all along the spine of an operating system.

@ainmosni it partly is a Windows problem. The kernel shouldn't crap itself at boot when it encounters a corrupt unsigned binary excuse for a third party sys file.
@ainmosni Ubuntu and their now cultish desire to snap the whole OS. i have to use snap if i want to use LXC/LXD and it gives me a lot of pause using it extensively because i can't control the auto-update of snaps in the background. everytime i think i can, they push something onto the system that undoes my settings and they start auto-updating without my control. i hate it.
@blogdiva yeah, Ubuntu's NIH syndrome made me give up on the distro a decent time ago.
@ainmosni
When people say this is not likely to happen in Linux, they don't just mean "it's not possible to put faulty code in the Linux kernel as an update" or "it's not possible to force automatic updates in Linux", but they mean instead "if it happened, it won't happen in all the Linux world like Windows", so it would be almost unnoticeable, why? (continued in the next comment)
@ainmosni
Combine all the following to understand the point:
1. It is the failure of the proprietary and software as a service models. If this were an open, in-tree Linux driver, it would likely go through a much more robust review and test cycle as well as not be simultaneously pushed to every Linux system in existence using it. It would be filtered through multiple distros' testing and release processes which would make discovery of the bug much more likely (mentioned by @CalcProgrammer1 ).

@ainmosni
2. The filter mechanism of Linux distros failed and we got the fault update (that no one knows of)? Backup before an update and the update failed, we can still roll back with BTRFS or roll back to the last bootable image when using an immutable distro (Mentioned by @didek ).

(continued in the next comment)

@ainmosni
Windows does not have all this, all it has is, a non-snapshot and non-immutable OS model that's relying on forced automatic updates that are sent directly to all Windows machines around the globe without the different distros review and testing mechanism that usually their updates goes through.

So It's not just a technical issue, it's also a philosophical issue in Windows.

@ainmosni My experience is the opposite. In any org that I was allowed a company-owned and issued Linux system, we were exempt from the requirement to have the enterprise remote install/control BS. (Only two orgs allowed this.)

That said, it's certainly _possible_ to have similar procedures and process around a Linux install base -- and also similarly ill-advised.

@ainmosni
....except, the fix would be scalable and automated, from a compartmentalised part of the system, and more importantly: immediately reversible. Potato's, tomatoes.

@ainmosni crowdstrike stated it was only rolled out to Windows, not apple or linux.

So could easily have been all 3....

@ainmosni

The bad actors could try, but it really is a Windows design problem.

It should not automagically install new drivers.

No other OS does this.

#Linux #CrowdStrike #Windows

@SpaceLifeForm @ainmosni CrowdStrike for Linux does the same thing and even caused similar problems before.
Crowdstrike did this to our production linux fleet back on April 19th, and I've ... | Hacker News

@emberquill @ainmosni

Those links do not point to evidence of this.

@SpaceLifeForm @ainmosni Clearly you have no idea what is actually happening. It's not a #windows issue, it's a #crowdstike issue. Crowdstrike is basically a rootkit level cybersecurity tool #enterprise orgs use to manage their computers. No personal windows install will automatically do this.

@chasehainey @ainmosni

It really is a Windows admin issue, because the org using Windows decided to use Crowdstrike on Windows.

Yes, Crowdstrike messed up, but the org using Windows messed up in the first place by trusting an outside vendor that they gave root (system) to.

Do not give root to outsiders.

#Linux #CrowdStrike #Windows

@SpaceLifeForm @chasehainey @ainmosni but we _do_ give root to strangers all the time! The linux devs, the distro packagers, systemd devs, gnu people, the list goes on and on. We do give them root because we trust them. The problem is when that trust is broken and what are the consequences. Don't forget the Debian SSH issue 15ya+, log4j like 4ya or even xz/ssh issue this year.

@mdione @chasehainey @ainmosni

Except, I do not have to give outsiders root in order for me to compile from source and install the binaries.

#FLOSS

@SpaceLifeForm @chasehainey @ainmosni I mean how do you not? Every time you install an application (depending on how you install it anyway), you're giving root to an outsider. This applies equally to Linux as well as Windows.
@SpaceLifeForm @ainmosni Well now you’re confirming you don’t understand what’s happening. First you blame the OS (ie #microsoft) and now you’re blaming the orgs using #windows (ie the victims) and we’re giving a pass to … checks notes… #crowdstike. Guess I mute and move on.