Mastodawn

After basically the whole #Microsoft #Azure cloud was hacked (see list of related sources on https://karl-voit.at/cloud/ ), the first follow-up incidents went public caused by missing containment actions:

60,000 emails were stolen from 10 #USA #StateDepartment accounts
https://www.reuters.com/world/us/chinese-hackers-stole-60000-emails-us-state-department-microsoft-hack-senate-2023-09-27/

If you didn't understand until now: basically EVERYTHING at Microsoft got hacked and Microsoft can't (or won't) get rid of the intruders. Everything authenticated by Microsoft is tainted. Even #Windows auth.

You Can't Control Your Data in the Cloud

Now that I have migrated some of my hosts to #NixOS, I do have a bad feeling because of #Microsoft and most probably GitHub being hacked.

As mentioned on https://www.karl-voit.at/2023/09/12/nix/ the deep #GitHub dependency turns out to be a real downer for this OS.

#security #integrity

I Started With Nix, NixOS, Home Manager and Flakes

I Started With Nix, NixOS, Home Manager and Flakes

public voit - Web-page of Karl Voit

David Clubb Sep 29, 2023

@publicvoit I hear so many positive things about this on various podcasts (you probably know the ones), but I once tried it and couldn't even get to a useable desktop environment. That was a while back but I will stick with other OSs for now; and maybe if I go immutable I will try #Fedora first

Stephane L. Rolland-Brabant ⁂Sep 29, 2023

@davidoclubb @publicvoit "couldn't even get to a useable desktop environment" .

What ?

I have installed NixOs many times since I slowly started looking at it in 2016 or 2017...

And I have had many problems with corner-stone situations, some needed packages broken in the current branch, or learning the architecture of nix package expressions in nix language, or with other package-managers not interfering really well like python pip or conda or perl cpan...

But I have NEVER had a problem of having a non-useable desktop environment.

NEVER. Just to tell you, although it is only my personnal experience of NixOs.

Have you checked the MD5sum of the installation ISO that you had downloaded?

Alexander Sosedkin

@publicvoit

> For example, when GitHub would be out of business or the service is down for some other reason, NixOS would probably be dead. Its main repositories are on GitHub and there is no obvious fall-back concept to other repositories hosted on different services.

This is just plain false. Flakes and channels can point anywhere; the only thing that'd need special care to move is the registry repo that points to all the other repos.

@monk Yes. And at least in my case, they all point to GitHub.

Alexander Sosedkin

@publicvoit my point is, you can point them anywhere, just find a suitable hosting

@monk And my point is: all defaults of the NixOS installer are pointing to a hacked platform where anything could be manipulated already.

Copying a tainted dataset still results in a tainted dataset, independent of the trustworthiness of a different hoster.

Alexander Sosedkin

@publicvoit OK, so, take the last commit before the compromise, move it somewhere, replay history, update registry, update default registry URL, rebuild installation media. All the technical stuff is already there, about as ready as it could possibly be, so it's mainly the question of convincing the community that all of that is actually necessary.

@monk So, yes. Let's trust Microsoft once more and delete everything since 2021-04. 😜

https://msrc.microsoft.com/blog/2023/09/results-of-major-technical-investigations-for-storm-0558-key-acquisition/

Results of Major Technical Investigations for Storm-0558 Key Acquisition | MSRC Blog | Microsoft Security Response Center

Results of Major Technical Investigations for Storm-0558 Key Acquisition

Alper Çelik Oct 1, 2023

@publicvoit @monk
https://discourse.nixos.org/t/could-microsoft-master-key-hack-impact-nix-via-github/33652?u=alper-celik
Is interesting discussion about possiblity of an attack.

Could Microsoft master key hack impact Nix via Github?

Interesting discussion here about the possible implication of MS’s recent master key hack on GitHub’s security. Speculation that any repo on Github could now be compromised too, including nixpkgs. Any thoughts on this?

NixOS Discourse

@monk @publicvoit I can understand why someone would say it, though. By default, the flakes registries and the previous channels both point to GitHub URLs. They don’t have to, but it’s not obvious that they could be different.

Two big things I would worry about if GitHub became unusable for any reason: 1) nixpkgs is friggen’ huge, in terms of size and activity, so picking a forge successor must be done carefully. 2) issues and PRs are hard to migrate.

@monk @publicvoit none of these problems are actually specific to GitHub, I think. It’s just what Nix uses currently. Communicating the change when necessary takes the same amount of work regardless of the host. Maybe the issues/prs migration can be easier with better tooling elsewhere.

A disaster plan sounds like a good idea. I hope someone has one, but I haven’t heard of it.

Raito Bezarius Sep 29, 2023

@publicvoit While I agree with you, there are issues, I worry that your framing is flirting with baseless conspiracies as you seems to be ignoring that there are many safeguards in place to avoid letting GitHub corrupt the whole project.

Even if we didn't use GitHub, you have to understand that NixOS / Nixpkgs cannot force anyone we are consuming packages of to migrate somewhere else.

Either case, I think this is kinda FUD…

@raito I can't follow your accusation.

The facts show that Microsoft got compromised since at least 2021-04.

There is no claim by MS I know of that GH is completely separated from MS infrastructure that got compromised.

Current NixOS setups are pulling from GitHub which belongs to Microsoft. Yes, this can be changed but that's not the point here at all.

As far as I know, you can't protect yourself from a bad actor that has more or less full access to the GH infrastructure and backends.
1/2

Raito Bezarius Sep 29, 2023

You are jumping from MSFT got compromised at time T to MSFT is still compromised and all GH repos are compromised with full capabilities for attackers. This is one of my accusation.

> As far as I know, you can't protect yourself from a bad actor that has more or less full access to the GH infrastructure and backends.

Assuming this without proof is, to be honest, conspiracy.
I don't like Microsoft neither, but this is ridiculous.

@raito The bad actors had years of more or less full access to MS infrastructure.

Actors that can pull off such an attack are perfectly well aware of what to do so that they keep access when the original attack vector is not available any more.

This is standard procedure for each intrusion attack.

Furthermore, in such a situation, the original bad actors can provide any sort of access to interested parties.

Yes, it's hard to digest but that's absolutely standard IT sec reasoning.

@raito Furthermore, it's not just me who tries to explain the implications. Please read other sources that quote various security experts and how they judge the impact of this incident.

Assuming that nothing happened to GH is understandable from a project's point of view (effort! trust!) but nothing more than wishful thinking without any proof.

In IT sec, you always(!) assume the worst case just because of that. You can never be sure otherwise.

LisPi Sep 30, 2023

@publicvoit @raito Does NixOS not independently sign its updates and use checksums of the aggregate repo contents (not trusting git's mostly sha1-only setup) like Guix?

If it does those two things, there's very little a malicious host could do other than denial of service.

If it doesn't then uh yeah, it's broken and really should fix that post haste.

alastair87 (old account)Sep 30, 2023

@publicvoit @raito One thing GitHub may have going to it is that it's still descended from the pre-acquisition infrastructure. It has now been tightly integrated into Microsoft though and I assume much of it is hosted on Azure. But they still use e.g. Ruby, MySQL because that's what it was built with to begin with. Much as Hotmail continued to run on UNIX for a long time after Microsoft first bought it.

Yes, there is no proof or indication that anything happened to any GH repository yet. 👍

However, in IT security, you don't rely on lucky guess. A compromised network is still a compromised network and needs to be restarted from a clean status.

It doesn't look like MS is going to setup major parts of their infrastructure to introduce trustworthy hosts again.

So where's the FUD in terms of reasoning?

Raito Bezarius Sep 29, 2023

> So where's the FUD in terms of reasoning?

> It *doesn't look like* MS is going to setup major parts of their infrastructure to introduce trustworthy hosts again.

I think you answered yourself very well.

In IT security, lucky guess are not primitives to build threat models. Hypotheses, assumptions, economics, politics, technical measures and careful analyses are.

What you are doing is just lucky guessing that MSFT didn't do any form of "reasonable" due diligence.

@raito After years of having a (potential) state actor in the back-end of MS, I'd be very interested in your assumptions that they really did not perform lateral movements and expand to linked networks.

No tech measures can mitigate or contain such an attack that lasted for so long in retrospect.

From an politics/economic perspective, we agree. We see what the economic decisions were already.

But that is strongly orthogonal to IT sec reasoning. Trade-offs won, as usual. But no proof.

Raito Bezarius Sep 29, 2023

@publicvoit But then, assuming the level of catastrophe you are describing.

What value is there in using a modern computer? NixOS/Nix is not the only thing affected. systemd is in GitHub, systemd developers are from Microsoft, etc, etc.

What is the usable advice we can get out of your whistleblowing?

@raito I'm not a whistleblower at all! I just quoted articles published by MS & independent sources.

I never said that we should stop using NixOS, systemd, ...

We just need to be aware that there is no such thing as a trustworthy cloud infrastructure most probably anywhere except you deal with it yourself to some degree.

All projects that rely on a pot. compromised infrastructure need to invest in mitigation measurements against malicious infrastr. which wasn't discussed so far AFAIK.

Raito Bezarius Sep 29, 2023

> All projects that rely on a pot. compromised infrastructure need to invest in mitigation measurements against malicious infrastr. which wasn't discussed so far AFAIK.

We already touched base on some obvious mitigation measures we all enjoy thanks to the concept of Git repositories.

We have many more because of how nixpkgs works, but I admit I am slightly annoyed because you seem to be ignoring them and you didn't contact any expert matter, I assume?

Raito Bezarius Sep 29, 2023

> I never said that we should stop using NixOS, systemd, ...

> We just need to be aware that there is no such thing as a trustworthy cloud infrastructure most probably anywhere except you deal with it yourself to some degree.

Right, but what you are saying is that NixOS is particularly reliant on GitHub whereas *everyone* is reliant on GitHub so…

LisPi Sep 30, 2023

@publicvoit It doesn't have anything like Guix's fallbacks to (among other things) the #SoftwareHeritage archive?

That's all kinds of unfortunate.