Mastodawn

jonny (good kind)

It's so cool that anthropic is setting up a double-sided protection racket where it will profit from the massive token burn of attackers and defenders with a tool specifically designed to generate exploits and their only observable mitigation is a clientside system prompt that sternly warns the LLM to be good and not do malware
https://red.anthropic.com/2026/mythos-preview/

Claude Mythos Preview \ red.anthropic.com

jonny (good kind)10h ago

sure they are doing """alignment""" to the models, and maybe they have some more sophisticated serverside mitigations. but the fact that the system prompt text is in the package at all rather than all being entirely serverside does the opposite of inspire confidence. Even the system prompt is fine with hacking as long as you go "it's ok I am good"
https://neuromatch.social/@jonny/116325221458366596

jonny (good kind)10h ago

so this simultaneously raises the floor of doing open source at all to "if you can afford brute force generating exploits against your repos for days at a time" while simultaneously causing so many false positives that bug bounties are crumbling and the info giants will pull labor from open source projects by just generating them badly in-house - don't roll your own crypto becomes "now you have to roll your own crypto because nobody else is, and then pay an AI company to secure it for you."

The end of the curl bug-bounty

tldr: an attempt to reduce the terror reporting. There is no longer a curl bug-bounty program. It officially stops on January 31, 2026. After having had a few half-baked previous takes, in April 2019 we kicked off the first real curl bug-bounty with the help of Hackerone, and while it stumbled a bit at first … Continue reading The end of the curl bug-bounty →

daniel.haxx.se

jonny (good kind)10h ago

you know that problem where it's actually in Google's best interests to sabotage their traditional search results to force everyone to use the AI results because then you never leave the site and direct prompt advertising becomes extremely valuable? yeah, it's like that for code, where it's actually in anthropic's best interests for all the code to be entirely unmaintainable and unsecurable except for with LLMs

jonny (good kind)9h ago

i feel bad constantly fixating on the informational capitalists while there is so much material harm being done in the world, and then simultaneously remind myself that this is literally capitalism's gamble to finally and fully enclose not only the material world but also our minds. If informational reality comes to be owned by 4 megacorporations, then it's all fascism forever baby.

happyborg 8h ago

@jonny I don't feel bad for that because the best thing I can do is use my understanding and skills. I know what I'm rubbish at too.

What I don't understand is why so few of those I came to know in the FOSS, privacy and security space do not get this, and are raving about it actively embracing "AI" and the snake oil hype.

Neil Moffatt 8h ago

@jonny All while Marx is still attacked as some kind of evil imposter ... mostly because he foretold the decay of humanity capitalism would herald.

You can enable a lot of evil by capturing and controlling information.

In many cases, the informational evil precedes and facilitates the actual harms. See: JK Rowling's TERF campaign and how genocides tend to work (you can't just kill a bunch of people out of the blue you have to get ordinary people to hate or at the very least not care for them first)

@jonny the cyberpunk wiki is starting to look like the necronomicon now

jonny (good kind)9h ago

jonny (good kind)9h ago

@Viss i can't wait for the phase of the grift where "they can't control it" and release a series of whitepapers on how the only mitigation is to constantly refactor your code with a background churn of 10 exploit generation agents to not present a stable attack surface

jonny (good kind)9h ago

@Viss like their entire corporate voice is laying the groundwork for one day claiming "hey everyone now that we are too big to fail and integrated everywhere, we are unhappy to announce that we have lost control of the models but can't shut them off because their so important and everyone needs to subscribe to our active countermeasures protection suite or a rogue AI that we are no longer responsible for will hack you."

The Orange Theme 9h ago

@jonny @Viss Meanwhile, I'm in every vibe-coded web app going "../../../../../" popping 0-days lol.

jonny (good kind)8h ago

(for the literal reader out there, i am not claiming this is actually their secret plan or whatever, i am saying that whenever anthropic goes like "we didn't fully understand the model..." or invoke emergence or otherwise write as if the model is some unknowable god, that's always in service of product

João Santos 10h ago

@jonny I was going to say LLMs also struggle to understand LLM generated code, but then I remembered that it will only lead to more tokens being used, and increase Anthropic’s revenue, so it's a win-win for them.

jonny (good kind)9h ago

@jmcs exactly

cynicalsecurity

@jonny which is why I am really uncomfortable with their "vulnerability hunting" partnership which looks a lot like a closed club where "responsible disclosure" could well mask "no disclosure" or "only to the right US people disclosure"...

Claus Cramon Houmann 8h ago

@cynicalsecurity @jonny yes!

Joshua Barretto 9h ago

@jonny I've noticed a thing happening a lot recently where I build something, someone asks how it works, and is shocked at how little code there is behind the scenes. Maybe this isn't distinctly a thing that follows from LLMs alone, but it feels like people increasingly expect complexity where little is warranted, and that can't be a good omen

jonny (good kind)9h ago

@jsbarretto my friend i am watching this happen in real time too as projects that should take 1k lines suddenly take 10k lines of disjointed nonsense, and it's like "no it's not magic it's just thinking about how things should work a little bit"

@jonny
This feels really insightful, I gotta say!

Chris [list of emoji]4h ago

"First one's free."

datarama 10h ago

It's not specifically designed to generate exploits; it's a general-purpose LLM that turns out to be very good at writing exploits.

jonny (good kind)10h ago

@datarama they are more or less explicitly telegraphing that defense by attempting to generate exploits against yourself will be at least part of the product. their description in second image reads to me like a product brief - these are the things you should expect from the fully automated luxury self-hack solution

Pete Alex Harris🦡🕸️🌲/∞🪐∫10h ago

@datarama @jonny

I have come to the conclusion that making a thing as general-purpose as possible is a fool's quest because nearly all purposes are stupid.

All the well-defined and non-stupid goals can be pursued better with the right set of specific-purpose tools.

Inevitable end of LLMs on the internet is a competition to see who can throw money into the money burning pit faster.

For one short moment we generated amazing share holder value.

degenerating degenerate 8h ago

@jonny It's good that you're taking on board the state of Anthropic's server backend code has nothing to say about the value of their LLM offering.

jonny (good kind)8h ago

@hopeless having difficulty parsing the sentence, but yeah given the nature of LLMs it does indeed matter what deterministic code wraps them and enacts their ability to do stuff like "execute commands" and "write code" and whatnot, to say nothing of being material safeguards both as literal filters of behavior but also orchestrators of the multi-agent chains that seem to be necessary to keep these things in the bounds of plausible behavior

degenerating degenerate 8h ago

@jonny Yeah... my point is... in the era of OpenClaw, it seems none of that affected Anthropic's ability to produce an effective hacking machine, affects their LLM usefulness, affects their profits... or generally matters.

jonny (good kind)8h ago

@hopeless
if it didn't matter, there would be no system prompts, there would never need to be a new feature in Claude code, they would never need to develop and release a new product as they are doing now, because it would all be driven by the quality of the LLM right? You seem to think my argument is that "their non-LLM tech is bad, so therefore everything is bad" when what I am actually saying is "their non-LLM wrapping tech and how they are marketing is a signal for how they intend to use it as a profit taking tool, and their displayed competence and techniques do not inspire confidence that they can adequately safeguard it from being a weapon"

jonny (good kind)8h ago

@hopeless
There are more thoughts in the world than whether something is good or bad

pomubieng 8h ago

@jonny aquest és un dels motius pels quals a la UE no es permet la investigació privada amb patògens d’alta letalitat en animals: prohibit tractar la tuberculosi en animals, prohibit desenvolupar vacunes contra ella, prohibit el seu diagnòstic en laboratoris privats…

@jonny isn’t it great when you sell the sickness and the cure 🫠

Claudius 4h ago

@jonny Ferengi rules of Acquisition 🤷