Infrastructure used to maintain and distribute the Linux operating system kernel was infected for two years, starting in 2009, by sophisticated malware that managed to get a hold of one of the developers’ most closely guarded resources: the /etc/shadow files that stored encrypted password data for more than 550 system users, researchers said Tuesday.

The unknown attackers behind the compromise infected at least four servers inside kernel.org, the Internet domain underpinning the sprawling Linux development and distribution network, the researchers from security firm ESET said. After obtaining the cryptographic hashes for 551 user accounts on the network, the attackers were able to convert half into plaintext passwords, likely through password-cracking techniques and the use of an advanced credential-stealing feature built into the malware. From there, the attackers used the servers to send spam and carry out other nefarious activities. The four servers were likely infected and disinfected at different times, with the last two being remediated at some point in 2011.

An infection of kernel.org came to light in 2011, when kernel maintainers revealed that 448 accounts had been compromised after attackers had somehow managed to gain unfettered, or “root,” system access to servers connected to the domain. Maintainers reneged on a promise to provide an autopsy of the hack, a decision that has limited the public’s understanding of the incident.

In 2014, ESET researchers said the 2011 attack likely infected kernel.org servers with a second piece of malware they called Ebury. The malware, the firm said, came in the form of a malicious code library that, when installed, created a backdoor in OpenSSH that provided the attackers with a remote root shell on infected hosts with no valid password required. In a little less than 22 months, starting in August 2011, Ebury spread to 25,000 servers. Besides the four belonging to the Linux Kernel Organization, the infection also touched one or more servers inside hosting facilities and an unnamed domain registrar and web hosting provider.

A 47-page report summarizing Ebury's 15-year history said that the infection hitting the kernel.org network began in 2009, two years earlier than the domain was previously thought to have been compromised. The report said that since 2009, the OpenSSH-dwelling malware has infected more than 400,000 servers, all running Linux except for about 400 FreeBSD servers, a dozen OpenBSD and SunOS servers, and at least one Mac.

https://arstechnica.com/security/2024/05/ssh-backdoor-has-infected-400000-linux-servers-over-15-years-and-keeps-on-spreading/

Linux maintainers were infected for 2 years by SSH-dwelling backdoor with huge reach

Ebury backdoors SSH servers in hosting providers, giving the malware extraordinary reach.

Ars Technica
@dangoodin
Not to put too fine a point on it, but holy shit!
@dangoodin hm, a library putting a backdoor into OpenSSH? Now where have I heard that before?
@dangoodin Still a better track record than some other OS's, one could imagine. 🤔
@decapitae @dangoodin I mean, root access to the kernel update processes? Pretty sure the reason this seems so small scale is likely because of adoption rate hits from non-Linux servers slowing it down.

@dangoodin

About 2 years ago I decided to wipe my laptop and install Ubuntu, made sure I got all the latest and everything couple of weeks later my bank account got hacked. Probably a coincidence but there was some weird stuff going on with Firefox that I've never seen any other browser do...

@dangoodin Someone rooted kernel.org... and they used it to send spam?!

Seems a bit unambitious compared to e.g. backdooring the kernel or something.

@sgf @dangoodin Back then all we did at hosting companies I worked for was turn down giant piles of money offered to us by spammers for IP blocks. Spam made tons of money. We had to have a separate vetting and abuse department. Of course the companies like OVH that would literally do anything for money are now public and the ones that followed the rules are gone.

@sgf @dangoodin

A bit like:
Boss: We've broken into Fort Knox!
Team: What are we going to do now? The Gold maybe?
Boss: We will...
(Team leans forward to hear of this brilliant plan)
Boss: .. use their photocopiers, all of them, free photocopying for us!

@sgf @dangoodin Do we celebrate their ineptitude, or fear that someone so inept could do it and others have already compromised the kernel...
@sgf @dangoodin An attempt to modify the kernel sources would likely be quickly discovered due to git hash mismatches. Something along those lines has in fact happened once: https://lwn.net/Articles/57135/
An attempt to backdoor the kernel

As has been widely reported elsewhere, an attempt was recently made to slip a back door into th [...]

LWN.net
@mansr I wasn't entirely serious, but I was also wondering if an effective attacker would be able to extend the attack. e.g. using ssh agent forwarding to attack those logging in, steal signing keys for commits and impersonate a commit in a way no one is likely to notice. But I know approximately nothing about kernel dev processes or security, so this was mostly a joke. :)

@dangoodin

Interesting story, but just let it be emphasized, this has happened FIFTEEN YEARS AGO! Just so that people don’t start panicking.

@mcepl

Your sentiments would resonate with me more if the kernel.org folks showed even a shred of transparency. Instead, they attempted to sweep this very serious breach under the carpet, a la Microsoft. Given kernel.org's opacity in 2011, who's to say there haven't been equally serious breaches since then?

@dangoodin

Perhaps, but then the article should be about that not about fifteen years old vulnerability.

@mcepl

The article is about important information that has never been revealed until now.

@dangoodin

??? https://lwn.net/Articles/457142/ ???

Again, if there was something specifically unknown before, you should make the article about that, shouldn’t you?

What am I missing?

kernel.org compromised [LWN.net]

@mcepl

I made it clear in my post there's no indication the source code was tampered with.

The breach that hit kernel.org was much more serious than we knew previously. Kernel.org should be called out for their opacity the same way Microsoft should be.

@mcepl

There is no vulnerability. There's only stonewalling by people we entrust with a massive responsibility. The two-years that the Ebury attackers had root access to the infrastructure is entirely new information. Feel free to bury your head under the sand. Some of us find the new details important. Apparently, you don't and would rather not see articles reporting them.

@dangoodin

I am absolutely NOT saying that they did nothing wrong! What I am saying is that you (at least for me) buried the lead on what was the article really about under the information which is long time known. “Linux maintainers were infected for 2 years by SSH-dwelling backdoor with huge reach” meant nothing new to me, because I knew about that. So, when I read the first paragraph which repeated what I knew from LWN, I dropped the whole article.

@mcepl

Wait, you're saying you already knew about the 2-year compromise of kernel.org by Ebury? How?

At any rate, I think both or us have said all we have to say about this. Can we move on?

@dangoodin

I am not paid to read the whole discussion under https://lwn.net/Articles/461552/ , but I somehow assumed that the problem was there for some time, because kernel.org maintenance team was just one guy since 2008. But yes, re-reading the article now doesn’t show any dates when the compromise actually started.

And yes, we can certainly move on.

Kernel.org's road to recovery [LWN.net]

@dangoodin In comparison here's FreeBSD's report of an intrusion also a little over a decade ago: https://www.freebsd.org/news/2012-compromise/
FreeBSD.org intrusion announced November 17th 2012

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

The FreeBSD Project
@dangoodin @mcepl Given how the kernel dev team is spamming CVEs for everything—even if it’s not exploitable—since they became a CNA, as well as their tendency to bury security fixes before CNA status, is also a good indicator for how they prefer to hide problems.
@dangoodin I see you prefer to assume malign intent, but for anyone else who is open to considering that Linux kernel folks aren't trying to hide something, see this note: https://lwn.net/Articles/973873/
Linux maintainers were infected for 2 years by SSH-dwelling backdoor(ars technica) [LWN.net]

@mcdanlj

So the compromise of kernel.org was the subject of an FBI investigation and later of a court case? Do I have that right? How do I go about learning more?