Christine Lemmer-WebberMar 5

I am convinced we are on the verge of the first "AI agent worm". This looks like the closest hint of it, though it isn't it quite itself: an attack on a PR agent that got it to set up to install openclaw with full access on 4k machines https://grith.ai/blog/clinejection-when-your-ai-tool-installs-another

But, the agents installed weren't given instructions to *do* anything yet.

Soon they will be. And when they are, the havoc will be massive. Unlike traditional worms, where you're looking for the typically byte-for-byte identical worm embedded in the system, an agent worm can do different, nondeterministic things on every install, and carry out a global action.

I suspect we're months away from seeing the first agent worm, *if* that. There may already be some happening right now in FOSS projects, undetected.

A GitHub Issue Title Compromised 4,000 Developer Machines

A prompt injection in a GitHub issue triggered a chain reaction that ended with 4,000 developers getting OpenClaw installed without consent. The attack composes well-understood vulnerabilities into something new: one AI tool bootstrapping another.

Show threadmccMar 5
@cwebber meanwhile people I talk to are like "wait why do you want guarantees your open source supply chain doesn't have LLM-sourced code in it. it has literally never occurred to me that this would be a thing someone would desire"
Show threadDaniel LyonsMar 5

@mcc @cwebber

I think there is a valuable distinction between LLM-sourced code and LLM tool calls. Both are potentially problematic but have different threat vectors.

LLM-sourced code is a non-deterministic system writing deterministic code. We can still code review it.

LLM tool calls is a non-deterministic system taking non-deterministic actions via deterministic tools. This can’t be code reviewed and must be sandboxed.

Show threadmccMar 5

@dandylyons @cwebber there are various ways I could respond to this post, but instead:

I'd like you to consider *the specific two posts in this thread you are responding to* and ask yourself if your comment is remotely relevant, or if you are simply pattern-matching on anti-LLM sentiment and responding with aggression/a thread derail.

Show threadDaniel LyonsMar 5

@mcc @cwebber The original post was all about an LLM taking non-deterministic shell level actions at runtime. And you conflated that with deterministic code written by an LLM.

What I wrote is very relevant.

Show threadmccMar 5
@dandylyons @cwebber it is about an attack based on covertly deploying LLM development tools, with the possible intent of later using them to leverage a second stage attack. If the LLM development tools were already installed, installing openclaw would not have been necessary and the attack could have worked a different way. We are discussing a situation where *the developer of a piece of software I use merely having LLM tools on their computer represents a risk to me*
Show threadmccMar 5
@dandylyons @cwebber in other words, if Christine's analysis holds, llm development tools create so much downstream risk to your users that *a malicious party would try to covertly install llm development tools for later exploitation*. That is the subject of discussion. Whether it is safe to install these things *at all*.
Show thread0xC0DEC0DE07EAMar 5
@mcc @dandylyons @cwebber I cannot believe that we went from arguing about making all software memory-safe as a way of cutting out a way in which computers could be coerced into taking arbitrary instructions from a potentially malicious source to a bunch of the industry abandoning any concept of separation between data and instructions and installing highly non-deterministic, ambiguous arbitrary code execution systems on their machines…
Show thread0xC0DEC0DE07EA
@mcc @dandylyons @cwebber we invented The Game for computers, why?!
Mar 5 at 10:28pmWeb