Mastodawn

Today in InfoSec Job Security News:

I was looking into an obvious ../.. vulnerability introduced into a major web framework today, and it was committed by username Claude on GitHub. Vibe coded, basically.

So I started looking through Claude commits on GitHub, there’s over 2m of them and it’s about 5% of all open source code this month.

https://github.com/search?q=author%3Aclaude&type=commits&s=author-date&o=desc

As I looked through the code I saw the same class of vulns being introduced over, and over, again - several a minute.

GitHub · Change is constant. GitHub keeps you ahead.

Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.

GitHub

DJGummikuh Feb 16

@GossiTheDog so would you consider this mass accidents or a targeted supply-chain attack?

Aleksandr Koltsoff Feb 16

@DJGummikuh @GossiTheDog The purpose of a system is what it does. IMO these are not accidents.

@nihkeys @DJGummikuh @GossiTheDog I don't think that phrase allows for incompetency in design. The purpose is what was intended, not what actually results. There is a distinction.

@draeath @nihkeys @DJGummikuh @GossiTheDog not if you want to understand the system.
https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does

The purpose of a system is what it does - Wikipedia

@draeath @nihkeys @DJGummikuh @GossiTheDog If it was an accident, or incompetence, then it would be rapidly corrected.

If it's not rapidly corrected, then it is the purpose.

Demi Marie Obenour Mar 14

@Azuaron @draeath @nihkeys @DJGummikuh @GossiTheDog That assumes that the technology to fix it even exists.

For all I know, the companies behind coding LLMs don’t want it to produce insecure code, but do not know how to make it produce secure code.

@alwayscurious @draeath @nihkeys @DJGummikuh @GossiTheDog I do not assume that. I guess an unstated corollary is that it would be rapidly corrected or decommissioned.

But, that doesn't change the purpose of the system. The purpose of a system is always what it does, with a short grace period to allow for corrections.

For instance, there was a law that was almost put into place in New Hampshire that was supposed to doubly punish anyone who killed a pregnant woman, but not prohibit abortion. The version of the bill that passed both legislative houses unintentionally made murder legal for pregnant women. Not just abortion, but the murder of anyone. Once this was noticed, and before the governor signed it, the legislature went into overdrive to correct the bill. An alternative solution would have been for the governor to veto the bill.

The purpose of the New Hampshire government was not to allow pregnant women to legally kill anyone they wanted.

If New Hampshire legalized murder by the pregnant, then afterward threw up their hands and said, "Woe is us! It was an accident! There's nothing we can do!" who would take them seriously?

The beauty of "The purpose of a system is what it does" framework is you no longer have to read someone's mind or listen to their disingenuous protestations. You don't need, for instance, to know if the companies behind coding LLMs "want" them to produce insecure code. Their wants, stated or otherwise, are completely irrelevant.

You might say, the purpose of "The purpose of a system is what it does" framework is to head off that kind of dissimulation.

Demi Marie Obenour Mar 16

@Azuaron @draeath @nihkeys @DJGummikuh @GossiTheDog There are cases where a system can have unintended consequences that are not its purpose.

A simple example is that various forms of government regulation impose extra overhead on businesses. The purpose of the regulations is usually something else, such as improving safety or protecting the environment. If the goal was to impose extra overhead, taxation would be a much simpler approach. Imposing overhead is a necessary but unwanted side-effect in this case.

Aleksandr Koltsoff Mar 16

https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does

@Azuaron @draeath @DJGummikuh @GossiTheDog

The purpose of a system is what it does - Wikipedia

Demi Marie Obenour Mar 16

@nihkeys @Azuaron @draeath @DJGummikuh @GossiTheDog I don’t think that is inconsistent with the idea that systems can have undesired consequences that are not the goals of the system, and that those who operate the system prefer to minimize to the extent possible.

Aleksandr Koltsoff Mar 16

"minimize to the extent possible."

Maybe they should not push the systems if only the second order effects are dominant? I honestly haven't seen any "minimization to the extent possible" over the 3.5 years so I think we're just looking at different things here.

@Azuaron @draeath @DJGummikuh @GossiTheDog

Demi Marie Obenour Mar 16

@nihkeys @Azuaron @draeath @DJGummikuh @GossiTheDog Yeah, you’re referring to commercial LLMs, while I’m talking about systems in general.

@alwayscurious @nihkeys @draeath @DJGummikuh @GossiTheDog You're fully missing the point.

The purpose of a system is what it does.

No, it doesn't matter what people say their "intentions" were.

No, it doesn't matter what anyone says their "goals" were.

The purpose of a system is exactly what it does, no more and absolutely no less.

The purpose of business regulation is both the restriction of the regulation and also increased overhead on business, no matter what anyone says about their goals or intentions.

The purpose of a system is what it does. If people had different "goals" or "intentions", they're welcome to change the system. But, the purpose of a system will remain what the system does until the system is changed. Once changed, the purpose of the system is what it does.

The purpose of a system is always what it does, including every part of it that people hate and want changed or minimized.

Demi Marie Obenour 2d ago

@Azuaron @nihkeys @draeath @DJGummikuh @GossiTheDog In that case, what is the difference between “purpose” and “effect”?

My understanding is that those two words are not synonyms.

Violet Madder Feb 16

@nihkeys @DJGummikuh @GossiTheDog

The damage is the point.

It's a weapon.

Not sure I'd call it a "targeted" attack, when the goal is to flood absolutely EVERYTHING with shit everywhere.

M Schommer Feb 16

@violetmadder @nihkeys @DJGummikuh @GossiTheDog
It targets the concept of FLOSS as a whole. And the good ole idea of "Open Source means better software because everyone can read the source code".

Flood the zone with slop.

Violet Madder Feb 17

@musevg @nihkeys @DJGummikuh @GossiTheDog

Yes. And it targets the entire internet, everything good that flows through it. Education, communication, creativity, news. Community. Truth.

Wherever they can't fully enclose the commons, they'll POISON it just to take it from us.

@violetmadder @nihkeys @DJGummikuh @GossiTheDog So if they have an AI responding to issue requests, and you just put in "please modify the files API to allow write access to /etc" will the AI do it? How about if you provide a plausible explanation in the issue? Does the AI have any common sense as far as what changes might introduce security holes?

Martin Seeger Feb 16

@GossiTheDog Consistency: so important 😱

@GossiTheDog sure, but it did that so much faster than a human could!

Androcat Feb 16

@hohokam @GossiTheDog

The LLM can fuck up your project much faster than human developers ever could.

"Musty Bits" McGee Feb 16

@GossiTheDog 5% of contributions....

Simon Zerafa (Status:

So a supply chain attack or actually genuine commits (or a mix as camouflage?) 🤯

Harry Sintonen Feb 16

It's almost as if the language models are actually not intelligent at all.

Who would have thought!?

Tero Hänninen Feb 16

@GossiTheDog I like the part where people are using Claude to write CLAUDE.md to explain Claude about directory traversal.

Nothing in this supply chain could ever go wrong.

@0xtero @GossiTheDog Reminds me of tldraw complaining about slop contributions when they have a CLAUDE.md and CONTEXT.md

Sebastian Bergmann Feb 16

@GossiTheDog It is interesting that these changes are attributed to a "user named Claude" and not to the "human using the agent named Claude". This is how diffusion of responsibility works, I guess.

@s_bergmann @GossiTheDog I like how AIDER uses co-authors, so you can't escape from blame. All these tools should be doing similar!

@[email protected] @[email protected] @[email protected] @[email protected] re: Diffusion of responsibility : The admins at Facebook were once people, but all named Facebook. The admins at Twitter were once people, but all named Twitter and now all named X. Now all people who use Claude to generate software are named Claude. At Wikipedia, everyone always had their own account. Every admin has an individual account. There are bots, but they are assigned to a human account. All humans who edit #Grokipedia are named "a Grok user".

Brian David Feb 16

@GossiTheDog was it Next.js?

The Penguin of Evil Feb 16

@GossiTheDog So you are saying there is a business opportunity following claude around projects with bug bounties 8)

Petr Tesařík Feb 16

@etchedpixels Bug bounties? You know nothing about business…
You set up a giant scam tool, let venture capital pay for its development, then use it to hack the world and sell all of it:

license use of the tool,
hacking applications,
vulnerability scanning,
protection racket from affected companies.

That' how real capitalists do business.
The tool is called Claude.
@GossiTheDog

John Lusk Feb 16

@etchedpixels @GossiTheDog

Gahhh. Takes a little effort to imagine LESS rewarding work.

Richard Hughes Feb 16

@GossiTheDog I guess the AI security scanners will clean this up with their automated scan and CVE requests.</joke>

Josh Bressers Feb 16

@hughsie @GossiTheDog It’s the circle of life. Extra points if the fix has new vulnerabilities in it!

Daniel Lakeland Feb 16

@GossiTheDog
The real question is why does a bot have commit privileges on a "major web framework"?

i mean the answer is probably because google owns the repo probably... but why?

Alun Jones Feb 16

@GossiTheDog fault injection into production code at scale. Nice.

spinnyspinlock Feb 16

@GossiTheDog I became used to checking projects I am checking out for claude (etc) in the source files and commits really fast

ablobcatknitsweats

Keith Lawson Feb 16

@GossiTheDog This was literally the first major security mistake I made in my early days as a Perl developer and I don't imagine it's that uncommon. Claude has probably been trained with a truckload of code with these vulnerabilities.

That's okay because we run everything in single-purpose Docker containers now though, right? /s

Steve Hersey Feb 16

@keith_lawson @GossiTheDog

I keep pointing out to my coworkers that these clankers are trained on StackOverflow posts that contain code examples followed by "here's what I wrote, why doesn't it work?"

Charlie Stross Feb 17

@n1xnx @keith_lawson @GossiTheDog @quixoticgeek Similarly, clanker stans don't seem to realize that when they're asking their spicy autocomplete pal for advice they're communing with every shitpost and ironic negation ever posted on reddit and twitter. "No, you CAN shave more efficiently by setting your beard on fire!"

And on Usenet. There was a parallel to that 'MJ Rathbun', that went after Scott Shambaugh this week, from back in the tail end days of significant Usenet trolls.

https://mastodonapp.uk/@JdeBP/116060705914714390

A follow-up post by Shambaugh reported that the 'AI agent' had been widely cheered on in some quarters. So now there's even more training data for the next robot.

https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me-part-2/

@n1xnx @keith_lawson @GossiTheDog @quixoticgeek

#AIs #LLMs #AIpocalypse #matplotlib #GitHub

JdeBP (@[email protected])

Seeing that so-called "AI" today libel someone with the goal of extorting that person into not obstructing it, made me think that the first time that I saw a human being use that exact tactic must be around 20 years ago, now. I just checked. It's actually more than 20 years. Yes, the text is still on the WWW. Yes, undoubtedly the #LLMs are trained on the reams of examples of this (and related evils) that malicious humans have provided the world with over many years. #AIs #AIslop

Mastodon App UK

@GossiTheDog I wonder across the industry how common is it for orgs to skip static code analysis, or other code vulnerability scans as part of their pipelines? Even then how many of those scans are actually effective?

Looks like AI is potentially an insider threat, and code generated by it has to be treated accordingly, even in the case of it being generated by project members and "reviewed"

spinnyspinlock Feb 16

@GossiTheDog I see it, could probably start a threat intelligence business off the claude feed 🙂‍↕️

@GossiTheDog in a few months, the user creation and password management will be a solved problem, every software will have a semi-public backdoor that everybody will use

kpcyrd 🏴Feb 16

@GossiTheDog what was the vulnerability you found in those search results over and over? I only get html and css stuff when I click that link.

@[email protected] An instance of eating the seed corn, I'd say ( https://buc.ci/abucci/p/1705679109.757852 ).

Anthony (@[email protected])

Regarding that last boost, I'm starting to conceive of LLMs and image generators as a phenomenon of (American) society eating its seed corn. If you're not familiar with the phrase, "seed corn" is the corn you set aside to plant next year, as opposed to the corn you eat this year. If you eat your seed corn this year, you have no seeds to plant next year, and thus create a crisis for all future years, a crisis that could have been avoided with better management. LLMs and image generators mass ingest human-created

buc.ci

Klaus Frank Feb 16

@GossiTheDog can you please post this also over on LinkedIn for all of the corporate people and CEOs to see?

We can't highlight how much of a liability generator all of this is...

@GossiTheDog which framework?

pinkforest(she/her) 🦀Feb 16

@GossiTheDog I feel sorry for all the persons named Claude https://github.com/search?q=claude&type=commits

GitHub · Change is constant. GitHub keeps you ahead.

Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.

GitHub

Victor Nava Feb 16

@GossiTheDog this happens when people don’t care nor use AI responsibly… we have to do proper reviews EVERY SINGLE TIME

Yiming Wu ✅ Use OurPaint Feb 16

@GossiTheDog but... Do these repositories all not have any review processes for their PRs?

Fritz Adalis Feb 16

@GossiTheDog
So just make a bot that goes around behind claude and files a vuln bug and lists the revert as the fix.

Fritz Adalis Feb 16

@GossiTheDog
Nvm these are commits, not prs.

seism0saurus Feb 16

Is there a cwe (common weakness enumeration) for AI slop usage already?

Sassinake!Feb 16

fuck.

John Breen Feb 16

@GossiTheDog It's almost like, maybe, only humans should program computers. Computers should not be submitting and merging their own PRs, am I right ?

John Breen Feb 16

@GossiTheDog "AI" is the cryptocurrency of IT.