abadideaI can't stop thinking about the LLM-generated compiler that passes all the unit tests but emits inner loops that benchmark over 150,000x slower than a gcc debug build. I couldn't possibly have intentionally come up with such a funny demonstration of the point of genuine expertise https://harshanu.space/en/tech/ccc-vs-gcc/
Nina Kalinina@0xabad1dea makes two of us. The CCC isn't the flex AI proponents think it is, but there aren't enough people who can understand that it should have been a cautionary tale rather than a sensational headline. :(
@0xabad1dea like, I'll bait; great stuff, unsupervised agent produced something that can compile some C code that in a certain definition can be called "working", but absolutely not ready for any sort of production usage.
The agent has multiple reference implementations, extensive testing suite, and C is literally based on an extremely well defined standard. AI proponents claim that we're in an era where all we need is to provide a specification, and the agents will just implement the thing for us. This CCC thing is proof that they quite literally can't; it's difficult to think about a commercial software project that would have a specification better defined than the C standard. And a vanilla C compiler isn't all _that_ complicated, it's literally the kind of thing many undergrad SWE students build as a student project (yes yes lots of caveats and simplifications). You'd think Anthropic could improve on their CCC with the agents until they get the compiler working at least as well as the tcc would, but 1/2
The agent has multiple reference implementations, extensive testing suite, and C is literally based on an extremely well defined standard. AI proponents claim that we're in an era where all we need is to provide a specification, and the agents will just implement the thing for us. This CCC thing is proof that they quite literally can't; it's difficult to think about a commercial software project that would have a specification better defined than the C standard. And a vanilla C compiler isn't all _that_ complicated, it's literally the kind of thing many undergrad SWE students build as a student project (yes yes lots of caveats and simplifications). You'd think Anthropic could improve on their CCC with the agents until they get the compiler working at least as well as the tcc would, but 1/2
Mad Engineering@0xabad1dea @nina_kali_nina Ai is about where.i was.as a junior in college. Had some impressive accomplishments, but you wouldn't trust their code to be efficient, elegant, or maintainable.
@madengineering @0xabad1dea unlike juniors, it cannot produce novel things; anything I've seen reportedly done by AI ended up being a rip off of something that was in the training data set, at best translated to a different programming language
@madengineering @0xabad1dea just the other day I've read a post how awesome claude is, it got a 37 year old game binary and produced a typescript port (that no one has ever seen published yet). But on a closer evaluation it turns out this AI typescript port is likely based on a currently maintained C port, and the JS port of the game is definitely in the training data too. Stealing machines be stealing.
pixx@nina_kali_nina
...the sad thing is a tool that could look at that binary and point you at the reponof the port would have a genuinely impressive advancement for search
The technology they're building could actually be useful if they stopped throwing money in the furnace, and were more conservative with applying it. Though of course you likely don't n3ed ML for that anyways and I'm pretty sure the mass buildout is using ai as an exuse for massive resource and land grabs thst they already wanted to do sooo
Chip@nina_kali_nina @madengineering @0xabad1dea I’ve seen this myself as well. Reviewed a draft PR for some enhancement for our internal compiler generated by Claude. It’s weird and too complicated and my spidey sense is tingling. I go look for the same module name in the rust compiler. It’s basically the same structure, but simplified.
And I think that’s generally what people are seeing when they see an LLM being “creative”. They’re just unable to comprehend how vast a pool of human work it’s cribbing from.
@madengineering @0xabad1dea @nina_kali_nina And unlike juniors, it cannot internalize the lessons you teach it. You can correct it, and it will take that into account. But as soon as that slides off the context window, it’s gone.
@0xabad1dea @nina_kali_nina @bytex64 I've improved since college.
So Claude, consider this a challenge. I've upped my game, now up yours.