Timnit Gebru (she/her).20h ago

This "careful" "AI Safety" company that just accidentally leaked its entire source code to the world is the one that African governments are entering into agreements with to include in infrastructures from health care to god knows what.

These are the products people have to use to make sure that they don't get dinged in their performance reviews for "not using AI."

These are the products teachers have to use in schools so that "students aren't left behind."

https://www.theguardian.com/technology/2026/apr/01/anthropic-claudes-code-leaks-ai

Claude’s code: Anthropic leaks source code for AI software engineering tool

Nearly 2,000 internal files were briefly leaked after ‘human error’, raising fresh security questions at the AI company

The Guardian
Show threadTimnit Gebru (she/her).20h ago

I appreciated this article by @mttaggart
infosec.exchange.

I get the temptation especially in this world we're all living in where you have to produce something super fast all the time.

But my question is, what are people's arguments for how functioning software can be created with these tools?

What about new architectures, new ways of thinking, new programming languages, etc? Who will create those?

https://taggart-tech.com/reckoning/

I used AI. It worked. I hated it.

I used Claude Code to build a tool I needed. It worked great, but I was miserable. I need to reckon with what it means.

Show threadTimnit Gebru (she/her).20h ago

I'm not even talking about the data stealing, exploitation, environmental pillaging, pollution, environmental racism etc.

I'm talking about the way people use the tools. Like what do advocates of using these tools say will happen to software engineering in the future? That it just won't need to exist because everyone will be able to create software using these tools?

Show threadTimnit Gebru (she/her).20h ago
That it will just take a different form, which is fine?
Show threadMichał "rysiek" Woźniak · 🇺🇦20h ago

@timnitGebru I think this is relevant to these questions, albeit handles them on a different level:
https://freakonometrics.hypotheses.org/89367

> Someone still has to reread, compare, test, contextualize, and sometimes rewrite. And if no one seriously takes on that work, the cost does not disappear. It reappears later in the form of errors, urgent fixes, loss of trust, and eventually litigation. What is presented as a productivity gain is often just an accounting displacement.

If No One Pays for Proof, Everyone Will Pay for the Loss

This post was initially written in French, Si personne ne paie pour la preuve, tout le monde paiera pour le sinistre Let’s start with a truism. In ordinary life, just as in economic life, we have to make decisions without ever knowing everything. Every decision involves some uncertainty, and therefore some risk. Some risks are … Continue reading If No One Pays for Proof, Everyone Will Pay for the Loss →

Freakonometrics
Show threadTimnit Gebru (she/her).19h ago
@rysiek Great article.
Show threadMichał "rysiek" Woźniak · 🇺🇦19h ago

@timnitGebru it really is.

And boy does the Claude Code leaked codebase support that assessment. Have you seen @jonny 's thread on this? If not:
https://neuromatch.social/@jonny/116324676116121930

Show threadMichał "rysiek" Woźniak · 🇺🇦19h ago

@timnitGebru the whole thing is great, but somewhere down the thread there are truly astonishing gems like:

> So the reason that Claude code is capable of outputting valid json is because if the prompt text suggests it should be JSON then it enters a special loop in the main query engine that just validates it against JSON schema for JSON and then feeds the data with the error message back into itself until it is valid JSON or a retry limit is reached.

Thousand monkeys, thousand typewriters…

Show threadMarcel Waldvogel

@rysiek @timnitGebru
I was so baffled to learn how *mandatory* output verification is implemented. Any sane developer would have resorted to a compact loop along the lines of

`do { result = tool_call(…) } while (!is_valid(result));`

Zero overhead besides the wasteful repetitive tool calls in the hope of eventually getting the format right.

Instead, they have complex, expensive instructions for the LLM to do that.
https://neuromatch.social/@jonny/116326861737478342

jonny (good kind) (@[email protected])

Attached: 3 images OK i can't focus on work and keep looking at this repo. So after every "subagent" runs, claude code creates *another* "agent" to check on whether the first "agent" did the thing it was supposed to. I don't know about you but i smell a bit of a problem, if you can't trust whether one "agent" with a very big fancy model did something, how in the fuck are you supposed to trust another "agent" running on the smallest crappiest model? That's not the funny part, that's obvious and fundamental to the entire show here. HOWEVER RECALL [the above JSON Schema Verification thing](https://neuromatch.social/@jonny/116325123136895805) that is unconditionally added onto the end of every round of LLM calls. the mechanism for adding that hook is... JUST FUCKING ASKING THE MODEL TO CALL THAT TOOL. second pic is registering a hook s.t. "after some stop state happens, if there isn't a message indicating that we have successfully called the JSON validation thing, prompt the model saying "you must call the json validation thing" this shit sucks so bad they can't even ***CALL THEIR OWN CODE FROM INSIDE THEIR OWN CODE.*** Look at the comment on pic 3 - "e.g. agent finished without calling structured output tool" - that's common enough that they have a whole goddamn error category for it, and the way it's handled is by just pretending the job was cancelled and nothing happened.

neurospace.live
Apr 2 at 3:11amWeb
Show threadMichał "rysiek" Woźniak · 🇺🇦11h ago

@marcel if all you have is a hammer, and if you charge by each hammer stroke, every single problem looks like a nail.

@timnitGebru