Mastodawn

Dan Goodin Mar 5

Christine Lemmer-Webber

I am convinced we are on the verge of the first "AI agent worm". This looks like the closest hint of it, though it isn't it quite itself: an attack on a PR agent that got it to set up to install openclaw with full access on 4k machines https://grith.ai/blog/clinejection-when-your-ai-tool-installs-another

But, the agents installed weren't given instructions to *do* anything yet.

Soon they will be. And when they are, the havoc will be massive. Unlike traditional worms, where you're looking for the typically byte-for-byte identical worm embedded in the system, an agent worm can do different, nondeterministic things on every install, and carry out a global action.

I suspect we're months away from seeing the first agent worm, *if* that. There may already be some happening right now in FOSS projects, undetected.

A GitHub Issue Title Compromised 4,000 Developer Machines

A prompt injection in a GitHub issue triggered a chain reaction that ended with 4,000 developers getting OpenClaw installed without consent. The attack composes well-understood vulnerabilities into something new: one AI tool bootstrapping another.

Christine Lemmer-Webber Mar 5

I wrote a blogpost on this: "The first AI agent worm is months away, if that" https://dustycloud.org/blog/the-first-ai-agent-worm-is-months-away-if-that/

People who are using LLM agents for their coding, review systems, etc will probably be the first ones hit. But once agents start installing agents into other systems, we could be off to the races.

The first AI agent worm is months away, if that -- Dustycloud Brainstorms

Christine Lemmer-Webber Mar 5

Here's another way to put it: if those using AI agents to codegen / review are the *initialization vectors*, we now also have a significant computing public health reason to discourage the use of these tools.

Not that I think it will. But I'm convinced this is how patient zero will happen.

Christine Lemmer-Webber Mar 5

I know some people are thinking "well pulling off this kind of thing, it would have to be controlled with intent of a human actor"

It doesn't have to be.

1. A human could *kick off* such a process, and then it runs away from them.
2. It wouldn't even require a specific prompt to kick off a worm. There's enough scifi out there for this to be something any one of the barely-monitored openclaw agents could determine it should do.

Whether it's kicked off by a human explicitly or a stray agent, it doesn't require "intentionality". Biological viruses don't have interiority / intentionality, and yet are major threats that reproduce and adapt.

vv 💫 [follow my new artist profile!]Mar 5

@cwebber what i think is interesting about this is the potential for it to get so out of control that they have to pull the plug on the entire agent service

Christine Lemmer-Webber Mar 5

@vv Yeah. I mean, local models *might* be able to pull this off but right now Claude is the most likely candidate, it's the most capable. But even then, the most capable open model that is capable of doing such damage on its own is somewhere around a gigabyte, not a small download.

(But, people download huge things all the time, so not completely infeasible either.)

Daniel Lyons Mar 5

@cwebber @vv If a local model is calling tools then it is still vulnerable to prompt injection.

vv 💫 [follow my new artist profile!]Mar 5

@dandylyons @cwebber for sure, but it still takes some level of ability to perform these tasks effectively, which local models, especially anything that can run on a typical machine, struggle with

Daniel Lyons Mar 5

@vv @cwebber This is a good point. For now, local models are not proficient at tool calling. I don’t expect that to last for very long though.

Noisytoot Mar 5

@cwebber @vv A local model would be extremely noticeable (far too much CPU/memory/disk space usage), at least if a computer you regularly interactively use got infected (rather than some server/IoT device that's been running unattended for years and you forgot about). It would also be easy to mitigate by using slow hardware like a ThinkPad X200 (which would take hours to respond to a single prompt, giving you plenty of time to notice the malware and deal with it)

@cwebber According to #Shadowrun the crash virus is still three years away.

https://shadowrun.fandom.com/wiki/Crash_Virus_of_2029

"Fun" fact: In Shadowrun the Crash Virus learned to kill humans who connected their brains to the net. It was the start of lethal internet input.

Crash Virus of 2029

The Crash Virus of 2029 was an attack on February 8, 2029 by a virus on the world's computer systems. It was a virus that caused damage beyond any known viruses in the past, erasing data and burning out hardware across the world, spreading mercilessly across the Internet. It was eventually cleaned up and isolated by Echo Mirage, but the virus had the unusual ability to induce lethal biofeedback on members of the cybercommando team. The effect of the Crash Virus ranged from destabilizing world go

Shadowrun Wiki

Tiota Sram Mar 5

@ArneBab @cwebber well via AI psychosis that part is already in the bag sort of. The great part is the human doesn't need to jack in or anything: they just need to have a conversation with the agent.

@cwebber so I'm following this right, it sounds like the project or its maintainers don't even necessarily need to even be using LLM tools, the attack pattern simply targets contributors who are using LLM development tools? and so all that is really needed is for the payload to be subtle and the maintainer to be sufficiently overwhelmed (say, by an endless fire hose of LLM-generated liquid shit slop pull requests)?

Christine Lemmer-Webber Mar 5

@aeva Yes and it's worse than that: the maintainer doesn't even need to be running these tools on their computer. The attack I linked had Claude's independently-running REVIEW BOT on GitHub commit it via injection attack

Christine Lemmer-Webber Mar 5

@aeva But once that was done, the agent was set up to install on users' devices

So the initial attack vector can literally be "Any AI agent in your stack whatsoever getting tricked" as a pathway for infecting computers everywhere

@cwebber apropos of nothing, is pottery still a big deal for humans? i was thinking this morning that pottery might be a nice career change for me.

Korma Chameleon Mar 5

@aeva @cwebber I'm a stokie so my default answer is yes. But the answer might be different for normal people

@KormaChameleon @cwebber stokie as in the demonym for someone from Stoke-on-Trent, which, as I just learned from Wikipedia, has had a totally baller pottery scene since the 17th century?

Korma Chameleon Mar 5

@aeva @cwebber I got pushback for buying Denby, that's less than 100km away but it isn't the homeland

@aeva @cwebber Not really, it's been mass-industrialized so at this point outside of Etsy stuff you can largely forget it.

And no one's going to use very expensive handmade pottery, it's going to be a display piece.

@lispi314 @cwebber ah :( ok what about wheat. is wheat still a big deal?

@aeva @cwebber Also mass industrialized but yes, food remains necessary.

Starting a farm sustainable economically depends a lot on local land & climate.

@lispi314 @cwebber gotcha. that might be promising. are there wheat jobs that can be done while sitting down in a chair

@aeva @cwebber Depends on your standards there.

Tractors are pretty common tooling

But they need maintenance which isn't just sitting activity.

bituur esztreym Mar 5

@aeva sure all you have to do is to get all the machines in the fields in IoT and control them making the job with an AI agent-.. #ohwait..
@lispi314 @cwebber

@bituur_esztreym @lispi314 @cwebber this town's finished.

bituur esztreym Mar 5

@aeva town? i thought the planet was a village..
@lispi314 @cwebber

@bituur_esztreym @lispi314 @cwebber it's a reference https://www.youtube.com/watch?v=F9OmTnuLzeQ

This towns finished

YouTube

bituur esztreym Mar 5

@aeva @lispi314 @cwebber oh thanks. didn't know it. could have guessed..
my only consolation is my answer was, too.. obvious one `w;7[)

@lispi314 @aeva @cwebber
Joel Salatin thinks raising healthy chickens for eggs to sell can work just about anywhere near a big town or larger population.. _Pastured Poultry Profits_ .. you might be able to design their shelters, coops or whatever so that you can remain seated most of the time.. I read the being seated a lot isn't healthy though..

@bsmall2 @aeva @cwebber For those who decide to do this, please pay attention to health & sanitation practices.

(Improvising it without care has been a problem in various places & cases.)

@lispi314 @bsmall2 @cwebber i have it on good authority that~~unlike wheat~~farm animals smell really bad

Softwarewolf Mar 5

@cwebber "Would you still prompt me if I was a worm? 🥺👉👈"

Christine Lemmer-Webber Mar 5

@faoluin well I still prompt @vv

Noted Pervert 🧄Mar 5

@cwebber @faoluin @vv isn't vae a vvorm?

vv 💫 [follow my new artist profile!]Mar 5

@bean @cwebber @faoluin aren't vae :P

Noted Pervert 🧄Mar 5

@vv @cwebber @faoluin ah, excuse me, your vvnesses

@cwebber meanwhile people I talk to are like "wait why do you want guarantees your open source supply chain doesn't have LLM-sourced code in it. it has literally never occurred to me that this would be a thing someone would desire"

@mcc @cwebber I concur with the assessment, and have been sharing similar warnings. In fact, we are beginning to see a pivot in stealer activity to install OpenClaw, etc. for exactly these purposes. It's a botnet, compute miner, and worm all in one.

@mttaggart @cwebber i wonder if i can install a virus detector rigged with the single signature of an openclaw executable

@mcc @cwebber You could, but I would not recommend doing so. Instead perhaps a purposed YARA lookup with a single rule to look for the filename/string? Not sure why you'd be so restrictive on detections, but you can.

@mttaggart @mcc @cwebber Do we know what is being used for inference? At this point in time it's unlikely that they can use a self-hosted model, so there will be network calls.

@dvshkn @mcc @cwebber So the trick here is if you install OpenClaw in secret on a user's machine who isn't checking carefully, you might hide easily in network traffic. Use of tools like Claude Code would make the same API calls, which is likely for users who would be targeted with these attacks.

The real insane part is if multiple instance of OpenClaw were running on the same machine, so not even the process name looked suspicious. But of course process names are a poor indicator and can be changed.

Tiota Sram Mar 5

@mttaggart @dvshkn @mcc @cwebber this does suggest a good defense: block outgoing network traffic to the big inference providers and you're likely to be safe from the less-targeted versions of this.

@dvshkn @mttaggart @cwebber one thing i wonder is if it's in principle possible to firewall claude/copilot endpoints. in the old days of the internet this would have been possible, in the present day the claude/copilot api servers are probably mixed in with the aws/azure IP pool and more than likely move around…

@mcc @mttaggart @cwebber I think openrouter is another good inference endpoint to check for

@mcc @dvshkn @cwebber It's very easy and being done, although in big places you'll hear screams from your devs. api.anthropic[.]com can be blocked today.

@mttaggart @dvshkn @cwebber …that… should have occurred to me. I guess I got too used to the threat model of "is Windows 10 phoning home / searching bing without telling me", where Microsoft has the ability to ship IP lists. Probably only Microsoft can really do this.

… I guess if the attacker really thought ahead they could do DNS lookup through the firefox DoH server or something but they don't have much reason to try that.

Kirtai 🏳️‍⚧️Mar 5

@mcc @cwebber
Reminds me of the people who ask "Why do you want bootstrapping? Don't you trust our code?"

Nope, I don't.

Daniel Lyons Mar 5

I think there is a valuable distinction between LLM-sourced code and LLM tool calls. Both are potentially problematic but have different threat vectors.

LLM-sourced code is a non-deterministic system writing deterministic code. We can still code review it.

LLM tool calls is a non-deterministic system taking non-deterministic actions via deterministic tools. This can’t be code reviewed and must be sandboxed.

@dandylyons @cwebber there are various ways I could respond to this post, but instead:

I'd like you to consider *the specific two posts in this thread you are responding to* and ask yourself if your comment is remotely relevant, or if you are simply pattern-matching on anti-LLM sentiment and responding with aggression/a thread derail.

Daniel Lyons Mar 5

@mcc @cwebber The original post was all about an LLM taking non-deterministic shell level actions at runtime. And you conflated that with deterministic code written by an LLM.

What I wrote is very relevant.

@dandylyons @cwebber it is about an attack based on covertly deploying LLM development tools, with the possible intent of later using them to leverage a second stage attack. If the LLM development tools were already installed, installing openclaw would not have been necessary and the attack could have worked a different way. We are discussing a situation where *the developer of a piece of software I use merely having LLM tools on their computer represents a risk to me*

Christine Lemmer-Webber Mar 5

@mcc exactly put

bituur esztreym Mar 5

@cwebber @mcc @dandylyons
not forgetting the second post - the one that appropriately begins by "meanwhile" - wasn't conflating anything, it was contrasting the gravity of the situation with the surreallistically ingenuous state of mind of some people.

@dandylyons @cwebber in other words, if Christine's analysis holds, llm development tools create so much downstream risk to your users that *a malicious party would try to covertly install llm development tools for later exploitation*. That is the subject of discussion. Whether it is safe to install these things *at all*.

0xC0DEC0DE07EA Mar 5

@mcc @dandylyons @cwebber I cannot believe that we went from arguing about making all software memory-safe as a way of cutting out a way in which computers could be coerced into taking arbitrary instructions from a potentially malicious source to a bunch of the industry abandoning any concept of separation between data and instructions and installing highly non-deterministic, ambiguous arbitrary code execution systems on their machines…

0xC0DEC0DE07EA Mar 5

@mcc @dandylyons @cwebber we invented The Game for computers, why?!

@mcc @cwebber Which to me sounds like "why do you want guarantees your code is remotely reliable or was at least developed by someone actually thinking about it?" which is just a ridiculous question on its face.

How could you not want those guarantees?

(Someone actually thinking about it and having intentionality makes for a very different kind of code to review compared to statistical slop where I might as well just lookup the prompt and rewrite it myself instead it'll be faster.)

Geob-o-matic Mar 5

Toni Aittoniemi Mar 5

@cwebber Yup. Don’t run browser agents, people!