Mastodawn

Christine Lemmer-Webber Mar 5

I am convinced we are on the verge of the first "AI agent worm". This looks like the closest hint of it, though it isn't it quite itself: an attack on a PR agent that got it to set up to install openclaw with full access on 4k machines https://grith.ai/blog/clinejection-when-your-ai-tool-installs-another

But, the agents installed weren't given instructions to *do* anything yet.

Soon they will be. And when they are, the havoc will be massive. Unlike traditional worms, where you're looking for the typically byte-for-byte identical worm embedded in the system, an agent worm can do different, nondeterministic things on every install, and carry out a global action.

I suspect we're months away from seeing the first agent worm, *if* that. There may already be some happening right now in FOSS projects, undetected.

A GitHub Issue Title Compromised 4,000 Developer Machines

A prompt injection in a GitHub issue triggered a chain reaction that ended with 4,000 developers getting OpenClaw installed without consent. The attack composes well-understood vulnerabilities into something new: one AI tool bootstrapping another.

@cwebber meanwhile people I talk to are like "wait why do you want guarantees your open source supply chain doesn't have LLM-sourced code in it. it has literally never occurred to me that this would be a thing someone would desire"

@mcc @cwebber I concur with the assessment, and have been sharing similar warnings. In fact, we are beginning to see a pivot in stealer activity to install OpenClaw, etc. for exactly these purposes. It's a botnet, compute miner, and worm all in one.

@mttaggart @cwebber i wonder if i can install a virus detector rigged with the single signature of an openclaw executable

@mcc @cwebber You could, but I would not recommend doing so. Instead perhaps a purposed YARA lookup with a single rule to look for the filename/string? Not sure why you'd be so restrictive on detections, but you can.

@mttaggart @mcc @cwebber Do we know what is being used for inference? At this point in time it's unlikely that they can use a self-hosted model, so there will be network calls.

@dvshkn @mcc @cwebber So the trick here is if you install OpenClaw in secret on a user's machine who isn't checking carefully, you might hide easily in network traffic. Use of tools like Claude Code would make the same API calls, which is likely for users who would be targeted with these attacks.

The real insane part is if multiple instance of OpenClaw were running on the same machine, so not even the process name looked suspicious. But of course process names are a poor indicator and can be changed.

Tiota Sram Mar 5

@mttaggart @dvshkn @mcc @cwebber this does suggest a good defense: block outgoing network traffic to the big inference providers and you're likely to be safe from the less-targeted versions of this.

@dvshkn @mttaggart @cwebber one thing i wonder is if it's in principle possible to firewall claude/copilot endpoints. in the old days of the internet this would have been possible, in the present day the claude/copilot api servers are probably mixed in with the aws/azure IP pool and more than likely move around…

@mcc @mttaggart @cwebber I think openrouter is another good inference endpoint to check for

@mcc @dvshkn @cwebber It's very easy and being done, although in big places you'll hear screams from your devs. api.anthropic[.]com can be blocked today.

@mttaggart @dvshkn @cwebber …that… should have occurred to me. I guess I got too used to the threat model of "is Windows 10 phoning home / searching bing without telling me", where Microsoft has the ability to ship IP lists. Probably only Microsoft can really do this.

… I guess if the attacker really thought ahead they could do DNS lookup through the firefox DoH server or something but they don't have much reason to try that.

Kirtai 🏳️‍⚧️Mar 5

@mcc @cwebber
Reminds me of the people who ask "Why do you want bootstrapping? Don't you trust our code?"

Nope, I don't.

Daniel Lyons Mar 5

I think there is a valuable distinction between LLM-sourced code and LLM tool calls. Both are potentially problematic but have different threat vectors.

LLM-sourced code is a non-deterministic system writing deterministic code. We can still code review it.

LLM tool calls is a non-deterministic system taking non-deterministic actions via deterministic tools. This can’t be code reviewed and must be sandboxed.

@dandylyons @cwebber there are various ways I could respond to this post, but instead:

I'd like you to consider *the specific two posts in this thread you are responding to* and ask yourself if your comment is remotely relevant, or if you are simply pattern-matching on anti-LLM sentiment and responding with aggression/a thread derail.

Daniel Lyons Mar 5

@mcc @cwebber The original post was all about an LLM taking non-deterministic shell level actions at runtime. And you conflated that with deterministic code written by an LLM.

What I wrote is very relevant.

@dandylyons @cwebber it is about an attack based on covertly deploying LLM development tools, with the possible intent of later using them to leverage a second stage attack. If the LLM development tools were already installed, installing openclaw would not have been necessary and the attack could have worked a different way. We are discussing a situation where *the developer of a piece of software I use merely having LLM tools on their computer represents a risk to me*

Christine Lemmer-Webber Mar 5

@mcc exactly put

bituur esztreym Mar 5

@cwebber @mcc @dandylyons
not forgetting the second post - the one that appropriately begins by "meanwhile" - wasn't conflating anything, it was contrasting the gravity of the situation with the surreallistically ingenuous state of mind of some people.

@dandylyons @cwebber in other words, if Christine's analysis holds, llm development tools create so much downstream risk to your users that *a malicious party would try to covertly install llm development tools for later exploitation*. That is the subject of discussion. Whether it is safe to install these things *at all*.

0xC0DEC0DE07EA Mar 5

@mcc @dandylyons @cwebber I cannot believe that we went from arguing about making all software memory-safe as a way of cutting out a way in which computers could be coerced into taking arbitrary instructions from a potentially malicious source to a bunch of the industry abandoning any concept of separation between data and instructions and installing highly non-deterministic, ambiguous arbitrary code execution systems on their machines…

0xC0DEC0DE07EA Mar 5

@mcc @dandylyons @cwebber we invented The Game for computers, why?!

@mcc @cwebber Which to me sounds like "why do you want guarantees your code is remotely reliable or was at least developed by someone actually thinking about it?" which is just a ridiculous question on its face.

How could you not want those guarantees?

(Someone actually thinking about it and having intentionality makes for a very different kind of code to review compared to statistical slop where I might as well just lookup the prompt and rewrite it myself instead it'll be faster.)