Kevin BeaumontCrowdstrike published a faulty update. Causes Windows to bluescreen. Driver is C-00000291*.sys. Will cause worldwide outages. Thread follows, I suspect. 🧵
I am obtaining a copy of the driver to see if malicious or bad coding, if anybody else checking let me know.
If anybody is wondering the impact of the Crowdstrike thing - it’s really bad. Machines don’t boot.
The recovery is boot in safe mode, log in as local admin and delete things - which isn’t automateable. Basically Crowdstrike will be in very hot water.
You know it was coming...
Crowdstrike's BSOP theme tune
Favour to IT folks fixing - could you please copy the C-00000291*.sys file to somewhere and upload it to Virustotal, and reply with the Virustotal link or file hash? It's still unclear if the update was malicious or just a bug.
I've obtained copies of the .sys driver files Crowdstrike customers have. They're garbage. Each customer appears to have a different one.
They trigger an issue that causes Windows to blue screen.
I am unsure how these got pushed to customers. I think Crowdstrike might have a problem.
For any orgs in recovery mode, I'd suspend auto updates of CS for now.
If anybody is wondering, the update was delivered via channel file updates in Crowdstrike.
BBC tracker (they mix up an earlier Microsoft outage, what they're actually tracking is the Crowdstrike issue) https://www.bbc.co.uk/news/live/cnk4jdwp49et
The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.
This is going to turn out to be the biggest 'cyber' incident ever in terms of impact, just a spoiler, as recovery is so difficult.
CrowdStrike's shares are down 20% in pre-market.
I'm seeing people posting scripts for automated recovery.. Scripts don't work if the machine won't boot (it causes instant BSOD) -- you still need to manually boot the system in safe mode, get through BitLocker recovery (needs per system key), then execute anything.
Crowdstrike are huge, at a global scale that's going to take.. some time.
Crowdstrike statement: https://www.bbc.co.uk/news/live/cnk4jdwp49et?post=asset%3A0c379e1f-48df-493c-a11a-f6b1e3d1eb63#post
Basically 'it's not a security incident... we just bricked a million systems'
For anybody wondering why Microsoft keep ending up in the frame, they had an Azure outage and- this may be news to some people- a lot of Microsoft support staff are actually external vendors, eg TCS, Mindtree, Accenture etc.
Some of those vendors use Crowdstrike, and so those support staff have no systems.
But MS isn’t the outage cause today.
Steve Hill 🏴🇪🇺@GossiTheDog Something none of the news seem to be reporting: are the Crowdstrike and Azure failures in any way related, or are they just independent shit-shows that coincidentally happened at the same time?
skry@steve @GossiTheDog The Register says no: https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
Edit: Wikipedia says there were two Azure incidents, one unrelated on the 18th, one right after the Crowdstrike update. https://en.m.wikipedia.org/wiki/2024_CrowdStrike_incident#Outage
random thoughtsHad an outage on our Azure MFA but hard to say if it's related or not, who knows what's running inside the service.
@hittitezombie @steve @GossiTheDog Microsoft told BBC they fixed the “underlying issue” but I don’t know if MS meant for the Azure problem or for Crowdstrike triggering the BSOD.
https://www.bbc.com/news/live/cnk4jdwp49et?post=asset%3Ad69abf29-6c37-4b11-8335-f73fe5b03f3d#post
Robert J. Berger@steve @GossiTheDog I’ve read (but don’t have the reference now) that they were independent. It was a single region at Azure which would tend to mean it was independent as well.
Andrew 
@GossiTheDog Also to the general public, a BSOD is a Windows issue. They're not able to look at the specific driver and identify it as a Crowdstrike service.
your auntifa liza 🇵🇷 🦛 🦦@piepants @GossiTheDog 🛎️ 🛎️ 🛎️ THIS!
System Adminihater@GossiTheDog I dont know how to use this platform but you seem to. here is a semi automatic way that I solved this on 1000 machines in 30 minutes.
Copy your custom drivered WinPE image (or a bare one from the ADK) to your system.
Mount it with wimlib.
Edit startnet.cmd and add
del C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys
exit
unmount image
put image in your PXE loader OR make it a usb bootable in Rufus
Save an assload of time.
Suwitcha1@systemadminihater @GossiTheDog How about system with bitlocker?
gigantos@systemadminihater @GossiTheDog most Corp machines are encrypted, so this likely won't work
@gigantos @GossiTheDog You can unlock BL in WinPE but cool whataboutism
JaxxAI@GossiTheDog they kind of are in the frame.... their kernel shouldn't shit its pants when a driver goes bad or doesn't load properly.
@JaxxAI @GossiTheDog yes it should. Unless they changed it to a model where each driver has a dedicated address space and can't alter memory for other parts of the kernel. You should not catch "corrupted memory" exceptions and ignore them, as you have no idea if said corrupted memory is important to the rest of the system.
The same applies to most widely used kernels such as Linux.
Having drivers running outside the kernel is a great idea though, but hard due to performance. #Redox makes a pretty decent attempt
@gigantos @GossiTheDog OR just don't load it in the first place if its bad. Linux can taint bad drivers and not crash though, giving you the opportunity to sort the issue.
@JaxxAI @GossiTheDog so can Windows, and it will use the last known good configuration. But unfortunately there are exceptions.
And in many cases, you can't know who caused an issue. The altered memory may not be attributed to the faulting driver at all.
@gigantos @GossiTheDog I'm under the impression the sys driver file is unsigned garbage. Under no circumstances should the kernel be loading it. At reboot, it should not load this file and let the user into Windows where an error pops up about driver not loaded. The admins then automate a fix rollout and all good. End users just need to reboot when BSOD occurs. My opinion is the kernel should not BSOD due to a garbage sys file like this at boot, that's a security hole in its own right.
@JaxxAI @GossiTheDog I don't disagree with your wish.
I just try to point out this is no better elsewhere. On windows a driver must be signed to be loaded, either by a catalogue file or embedded in the file itself.
I'm out traveling, so can't verify myself, but in this case, I suspect it's a data file named as a driver, and that another signed driver loads it. Nothing would stop this from happening on other operating systems.
ocdtrekkie@gigantos @JaxxAI @GossiTheDog I mean, when you're building a security product, you definitely want the system to just *load without it* when it's tampered with, right?
There's an argument to be made that your security product crashing the system when it isn't able to function is a feature, not a bug.
@JaxxAI @ocdtrekkie @GossiTheDog that works, when it works. But not for all cases. In this case I suspect the data file causing the crash is not part of what is snapshot'ed.
@ocdtrekkie @gigantos @GossiTheDog thinking about it... Why not? Safe mode procedure does this anyway. The key is to have a mechanism on your network or in the software that fixes itself once 'in' windows. I still think the kernel needs to be more resilient.
MOVED TO @@GossiTheDog It's funny, I think the Guardian liveblog has completely conflated the two
João Tiago Rebelo (NAFO J-121)@GossiTheDog #Alt4You #AltText
Crowdstrike says global IT issues caused by 'defect in 'content update
Here's the full statement from George Kurtz, the CEO of Crowdstrike:
"Crowdstrike is actively working with customers impacted by a defect found in a single content update for Windows hosts
"Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
"The issue has been identified, isolated and a fix has been deployed
"We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website,
"We further recommend organisations ensure they're communicating with Crowdstrike representatives through official channels.
'Our team is fully mobilised to ensure the security and stability of Crowdstrike customers."
Crowdstrike says global IT issues caused by 'defect in 'content update
Here's the full statement from George Kurtz, the CEO of Crowdstrike:
"Crowdstrike is actively working with customers impacted by a defect found in a single content update for Windows hosts
"Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
"The issue has been identified, isolated and a fix has been deployed
"We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website,
"We further recommend organisations ensure they're communicating with Crowdstrike representatives through official channels.
'Our team is fully mobilised to ensure the security and stability of Crowdstrike customers."
Zeb Palmer (Syringa Admin)@GossiTheDog visit our (login required) portal for details....
Chris, and a spooky owl@GossiTheDog I'd wager its quite a lot more than a million…
James Gilbert@cawhitworth @GossiTheDog Crowdstrike lives up to their name by striking down crowds of computers.
Kate Lance@GossiTheDog
I love how 'bricked' has become the new 'fucked'.
I love how 'bricked' has become the new 'fucked'.
Landwomble@GossiTheDog You'd think they'd update their website - it's all happy sales there. Perhaps their CMS is down, lol
Jason 🍸🫧
sotolf@GossiTheDog It's also kind of fun that they especially say "found in a single content update" like if it matters if it's only in one update :p
@GossiTheDog
A million systems?
29k customers, and assuming 5k boxes per customer == 145 million boxes.
Al@GossiTheDog Those bricks are very safe from hackers now
Kenneth@alslater
Plenty of door stops.
David Sullivan@GossiTheDog Unbootable systems *are* secure right?
@GossiTheDog and on Azure VM you have to take the disk offline, attach the unmanaged disk, jump through hoops to allow offline repair... my oh my.
Luís Silva@jt_rebelo what a nightmare! @GossiTheDog
Rivetgeek (He/Him)@jt_rebelo @GossiTheDog EC2 fix: Detach the root volume, attach it to another instance as just a volume, delete the .sys file, and then detach and reattach to the original instance. It's slow-going work, but it works.
Alex@GossiTheDog I'm a Linux guy. When using LAPS will they even have local admin access, if the LDAP is also down?
chrislawrence-EF⬡⬡42.pf@a_lex_ander @GossiTheDog LAPS password is still a local password with no requirement on network etc, it's just a managed local password that's system unique and automatically cycled on a schedule.
So if, IF, your AD is still up and you're a user with the LAPS authorisations you can retrieve what those managed passwords are.
Expertenkommision CyberunfallLDAP to be recovered first, otherwise bitlocker keys are unavailable as well.
Clients are useless until AD/LDAP available
Andrew Bartlett@a_lex_ander @GossiTheDog Depends if the AD DC runs Windows or Samba ;-)
DJGummikuh@GossiTheDog is it already clarified why affected .sys files from different customers have different hashes?
jonas@GossiTheDog
Surely purely academical at this point, but shouldn't a partially scripted solution be technically possible using a PXE boot scenario? Just curious.
Surely purely academical at this point, but shouldn't a partially scripted solution be technically possible using a PXE boot scenario? Just curious.
Jim@GossiTheDog there are ways around the bitlocker issue https://x.com/LetheForgot/status/1814203140842868797?t=Q-F-FApK033_gSIvIg6CZA&s=19
LetheTheForgotten (@LetheForgot) on X
@SwiftOnSecurity What we did was use the advanced restart options to launch the command prompt, skip the bitlocker key ask which then brought us to drive X and ran "bcdedit /set {default} safeboot minimal"which let us boot into safemode and delete the sys file causing the bsod.
Christopher@toyotabedzrock @GossiTheDog I'm trying to figure this out. How would it be possible to delete files on disk after skipping Bitlocker recovery? Surely files in \Windows\System32\drivers\CrowdStrike would be encrypted?
@toyotabedzrock @GossiTheDog on second thought, maybe it is decrypting the drive transparently, and they are just skipping the prompt, though I still don't see how, if this works, it isn't a big security issue. Weird.
@cbyrd01 @GossiTheDog it's just telling windows you don't want any drivers loaded but you still have to log in. So it is not really a security hole.
@toyotabedzrock @GossiTheDog thanks, that makes sense. It still assumes TPM-only mode, or that the user is there to enter the PIN I suppose.
@cbyrd01 @GossiTheDog yes it's still a manual process that needs someone with the admin password
@cbyrd01 @GossiTheDog It's not my solution but my guess is that the boot settings are not encrypted given they need to be read on startup. The instructions are telling windows to boot normally but not load anything besides the minimum needed for a command prompt. You obviously need to later reverse this change. So delete the file then follow this instruction. https://x.com/LetheForgot/status/1814205827974144488?t=l1CLI7uJ7WjBLlIUzm0IRg&s=19
Quixoticgeek@GossiTheDog they claim 24000 customers. If each customer averages 1000 machines. That's 24 million machines needing manual intervention.
Even more fun for those attached to the ceiling of an airport...
This is gonna take days of not weeks to fully fix...
Jeffrey@GossiTheDog is the market even going to be able to open?
Stirling C / @stirlo.be@jtig @GossiTheDog we’ll see soon enough 😬