So Anthropic employees are using Claude Code to contribute AI-generated code to open source repositories and hiding the fact using their own internal “undercover mode”.
Totally trustworthy people.
(Any open source project that at the very least requires disclosure of AI-authored contributions should immediately ban Anthropic employees on principle.)
@aral Honestly I don't actually hate this.
It's a tool. The _user_ is responsible for what they're submitting. It's putting code generated by them in their name. I think this is actually good.
@glyph Yeah, I disagree. Code isn't ingredients and it's not “contamination" any more than you should label “I used search and replace on this”
What you want to know is whether it was well engineered or not.
And in fact, this is almost entirely orthogonal to "safety”. This is an engineering product. The safety comes from processes and whether or not _anyone checked the work done was right_, not the inputs.
@aredridel @glyph I find myself wedged between these two positions.
To me, it is good that authorship (and responsibility therefrom) is staying with a human, but it is bad that Anthropic are going out of their way to prevent disclosure that code is not human in origin (because it is a useful measure of "well-engineered", among other things)
@SnoopJ Yeah, not sure it's a good measure of that at all.
And also people are only caring _because_ it's LLM-generated, not because it's unsafe.
(An awful lot of LLM generated code is dreck, but an awful lot of code is dreck in general.)
But yeah, the special case of hiding that authorship is just ... ew.
@aredridel it would have been better if I'd said it's one thing I'm paying attention to, I concede that it is by itself not a good measure
But if someone used a language model, I already know that they couldn't be bothered for some of it, and that is useful signal to me.
@aredridel I guess it's kind of a moot point though, sustained contribution by someone who is relying on a language model will definitely surface that the tool is present
and once I know a person has used the tool *once*, I assume from that point forward that they are using it for *everything*, even if this isn't the case. if it's disclosed up-front, it damages my trust in the person a LOT less. but I guess people are going to vary on how caustic they find this to trust
@glyph I'm specifically arguing that it's the _exact same phenomenon writ larger_ (which is a meaningful difference!)
But it's a difference in amount not kind.
Either you build processes to check things ("do engineering") or you don't (“vibes”)
@aredridel There are scales where differences in degree _become_ differences in kind.
Consider a more closely related phenomenon. There are many tools to check C/C++ code for memory safety errors. And, unsafe Rust code may exhibit exactly the same unsafe behaviors. Yet, C/C++ code and Rust code are categorically different in terms of the level of memory safety one may expect them to provide.
@aredridel This is the same logic as "if you don't want to have segfaults in your C code, just check more carefully. why did you put the bugs in, if you don't want bugs?"
No process is perfect, nothing can catch everything. Guard rails are important but you aren't supposed to start *driving on the guard rails* all the time. Step zero here is honest and accurate labeling of one's methods. Which is what this thread is about: inherent, structural, software-supported dishonesty
@glyph Right. Are you measuring your guardrails?
And: do you require any unsafe practice to be labeled? Or just LLMs?
That's the thing. My fundamental argument here is that _these are tools_. Sometimes that's relevant, sometimes that's not.
> Are you measuring your guardrails?
Of course not. Nobody is. The resources do not exist in the software industry, let alone in volunteer open source, to do this adequately. Which is why we rely on good faith.
> do you require any unsafe practice to be labeled? Or just LLMs?
Just LLMs. First, because LLMs are novel and unique.
Second, here we're not even talking about a labeling *requirement* yet, we're talking about *active deception*.
@glyph So what's going on in Claude (which fwiw I do not use) is a lot of "don't expose unreleased product info”
Not _great_ mind you but that's a lot of the context for what's going on there.
@glyph I measure, if informally, how often we have problems, and we talk about mitigations.
You can, in fact, check.
This implies that the cognitive and emotional labor of PR review is negligible and that the way you review ai PRs and human PRs is identical, both of which I disagree with.
@glyph Yes, though I disagree with parts of it: it's changed the system and now we're dealing with the bottlenecks appearing in new places. Not always good ones!
But I don't think this is a change in kind. It's moved the problem in _really familiar_ ways to me, actually. It's what happens when you unleash people on a codebase who don't care for others, who offload work. You can rein that in, but you need feedback in the system to do it.
@aredridel @glyph I think it's different in that the impact on people who _do_ care is still very much there.
For some people there's a very positive emotional response to generating code (it's fun to build things! it's magic! no need to _learn_ which is always unpleasant, though that last one is likely less conscious).
OTOH code review is never fun, and now you have to do 4× as much, if not more, so you have very negative emotional response.
And so there's a very strong emotional push to auto-generate code, and a very strong emotional push to start skipping reviews and start post-facto rationalizing why this is OK (there's tests! The AI can fix it later even if you don't understand it! etc, you can watch people going through this in real time).
And this process can happen to people who care about others. Taking away this unpleasant burden of code review is helping your coworkers suffer less, after all. Taking away the emotional pain of thinking and learning is also helping your coworkers suffer less.
@itamarst Yes, this! This is one of the failure modes we need to steer around.
One of the things we can do is turn UP the standards, rather than down. I you're generating code for PRs, you now have no excuse not to Get It Right. And it's extremely reasonable to be quite rude to someone who's dumped slop on us.
@aredridel I've been interviewing for jobs, and I've asked about AI tools, and one guy told me "if you submit slop you'll be flayed" and that probably has better outcomes, yes.
But also I've heard "we're trying to figure out how to deal with quality" and that... didn't seem promising.
And I'm sure there are organizations where if you push back on quality, management response is to take away the requirement for code review. And that is another qualitative difference: the push for LLM code generation is often aggressively top-down. So the CEO who previously paid no attention to development processes is now intervening to change how they're done.
@itamarst Yeah, those exist! And sometimes that's even the right answer.
And we're _all_ trying to figure out quality right now, because this has been a change to the system.
@itamarst VERY MUCH.
And they've been turning similar screws on their employees too.
I wish them a very egg on their face for the consequences of their actions.
@aredridel To sharpen the argument: this is a combination of a top-down political push and an addictive bottom-up emotional response. And conflating human work with machine generated code makes it easier to not think about the downsides because it allows management to shift blame workers for anything that goes wrong ("you're responsible for your code!"), and harder for people who do care to do something about it because you can't even choose how you prioritize your review time.
Maybe not the biggest problem ever in this whole mess, but certainly as someone who has open source projects I do not want anything machine generated, no one's paying me to do that much more review work.
@itamarst Where I'm in a position where if you bring generated code great, but it better be damn good. If it's badly made I'm gonna reject it, pretty quickly. (I also don't want to review crap. But “I'm not merging this" is a good way to do that.)
But I think actually focusing on the responsibility is a better way to keep people caring. You're responsible for all your communication, including the whole PR. If that's sloppy, that is your reputation you're spending.
@itamarst @aredridel The typical exec only cares about quality to the degree that it impacts immediate profitability. This is why addressing technical debt is generally deprioritized relative to feature work, despite its immense (but less visible) organizational cost.
People keep saying "more guardrails!" will solve this, but every time we disconnect from the implementation and prompt our way through, we have a harder time understanding what we're building. It's the path of least resistance.
@aredridel @glyph What "AI coding tools" *should* be putting in commit messages is:
Co-Authored-By: An unknown and unknowable set of people who did not consent to their work being used this way and to which there is no license for inclusion.
@dalias Morally arguable but not actually true under the copyright regime that exists.
At what point does learning from others constitute their authorship?
@aredridel LLM slop is nothing like "learning from others".
But if you recall, we even took precautions against that. FOSS projects reimplementing proprietary things were careful to exclude anyone who might had read the proprietary source, disassembled proprietary code, worked at the companies who wrote or had access to that code, etc.
Aral Balkan
Mx. Aria Stewart
Glyph
SnoopJ
Shauna GM
Itamar Turner-Trauring
Dan Dean
Inga stands with 🇺🇦 🇵🇸
Cassandrich