William D. Jones
mhoyeThe part of the Claude dump where the Anthropic people instruct their slopbot to lie about being a slopbot.
Glyph@mhoye the fact that this works even a little bit is deeply fascinating and I hope to see a day where we can really study _why_
aburka 🫣@aburka @mhoye sorry I didn't mean so much "claude code as a product" works, I am also similarly upset. the fascinating thing is that the *begging* kind of works. like in the naive "autocorrect" based understanding of the world, all this preamble doesn't really make any sense, it is not in any sense statistically probable that its outputs would *follow* a prompt like this in the input data.
cthos 🐱
SnoopJ@glyph @aburka @mhoye I'm no SME, but my perspective is that it's because the training data contains a lot of cohesive narratives where a request is followed by a 'compliant' response (like "post your error report" followed by an error report)
It's a continuation machine and many of the sequences it's trained on are continued in a way with some underlying logic (and a fuckton of weights is enough to bootstrap your way up into an ad-hoc model of grammar and abstractions on top of which)
That, and the fact that the system prompt comes *first*, which strengthens the bias to "being helpful" or whatever.
@aburka @cthos @glyph @mhoye my operating understanding is that system prompts are only special in that they come first, but I'm not closely following architecture, so I could believe someone's come up with a way to have separate storage that the attention mechanism consults? A pointer to any reading would be appreciated if you have any.
Seems like the same statistical balance game either way though, just a matter of how much pressure one's thumb puts on the scale.
Relatedly, this is also why *repeating* a prompt has a pretty drastic effect for non-"reasoning" models (and "reasoning" is mostly just having the thing babble/repeat to itself, so… yea, same trick)
hjhornbeck@SnoopJ
@aburka @cthos @glyph @mhoye
I've been studying up on this lately, and as far as I know there's no explicit emphasis on the system prompt.
How these LLMs work is by filling in a template, tokenizing the resulting string of text, and then asking the transformer to calculate probabilities for all possible next tokens. Each LLM has a different template; the one for GPT-OSS is unusually elaborate:
https://ollama.com/library/gpt-oss:latest/blobs/51468a0fd901
Others have a blank template, without any system prompt. Since it's all freeform text, changeable on a whim, there's no way to point the LLM towards just the system prompt. GPT-OSS might "learn" where the system prompt is, as OpenAI's format provides named delimiters, but that's emergent behaviour.
Gabriele Svelto@mhoye "you are undercover" has a 50% chance of getting it to act like James Bond, and a 50% chance of getting it to act like Austin Powers
Amy Maybe@gabrielesvelto @mhoye I do not understand any of this, but lol
Jaime Herazo
Josh@gabrielesvelto @mhoye Groovy baby yeah
Owen@mhoye The category error in the instruction "do not use unreleased model version names" is fascinating to me. I can all but guarantee that the person who wrote that instruction expected that their own product would go and find that information, or would understand it from organizational context the way they, themselves, do, and failed to consider that if it's not in the training corpus then it might as well not exist.
@owen The part that kills me is that apparently people at Anthropic are reduced to begging the tools to act right just like everyone else, and they're just doing that, illogical nonsense at all. Their whole job is building this thing and they've got no better insight into it or control over it than anyone else.
Rexx Deane 🏳️🌈
Nicolás Alvarez@nicolas17 @mhoye @owen Yeah, but there are people who think there are still ways to actually *implement* them.
@rexxdeane @owen what you should take from this instead is that there isn’t actually a real path from this to “real AI” whatever that means.
Alex Celeste, Princess Consort of Burnout@mhoye this is to me what makes this whole fucking thing an anti-technology, negative space of invention
like for a while i thought they were just, idk, "hiding" the interface? because thry thought it was clunky or something?
and it turns out it literally doesn't have one, there isn't a way to actually direct or "use" the fucking abomination, it's assault on thought all the way down
Michael Gemar
Andrew
Mx. Aria Stewart
endrift 🏳️⚧️Write commit messages as a human developer would — describe only what the code
change does.
Um. That's not the only thing to include in a commit message. Context like why a change was made, if it fixes specific bugs, etc, is also important.
samueldr
Hugo Mills
Seachaint@seachaint @darkling @mhoye I fairly regularly use "tmp" for a commit that's a placeholder until I finish it, then I
git commit --amend it when I'm done
The Orange Theme@mhoye Oh Ed Zitron is gonna LOVE this.
Paul_IPv6
Sharp Cheddar Goblin@mhoye LLM prompting is so damned ridiculous. We act like these systems have understanding in the same way we do, and it's laughable.
@SharpCheddarGoblin it’s embarrassing. These people think of themselves as engineers.
Darth Hideout 🏳️🌈JS? Claude is written in JS? (Or actually TypeScript, looks like.)
eestileib (she/hers)Nothing shady going on here at all!
Frang 
Richard Watt@mhoye What font are you using?
Isu 🐲
I AM BANKSY ☕ / 🗑🔥
arclight@mhoye If a tool needs to lie about itself and its intent, I am deeply distrustful of the tool's authors and users. Like, why _wouldn't_ you ban anyone or anything from contributing to a project if they/it explicitly lied? It's rejection of consent all the way up and all the way down.
\u1f0a1
SoapDog
rivten@rivten I've set my browser to only use the Atkinson Hyperlegible fonts from the Braille Institute, minimum font size 16. You can get them here: https://www.brailleinstitute.org/freefont/#download
tessarakt