simonw

Using coding agents to track down the root cause of bugs like this works really well:

> Three out of three one-shot debugging hits with no help is extremely impressive. Importantly, there is no need to trust the LLM or review its output when its job is just saving me an hour or two by telling me where the bug is, for me to reason about it and fix it.

The approach described here could also be a good way for LLM-skeptics to start exploring how these tools can help them without feeling like they're cheating, ripping off the work of everyone who's code was used to train the model or taking away the most fun part of their job (writing code).

Have the coding agents do the work of digging around hunting down those frustratingly difficult bugs - don't have it write code on your behalf.

Nov 1 at 9:01pmWeb
Show threadqa34514324Nov 1

I have tested the AI SAST tools that were hyped after a curl article on several C code bases and they found nothing.

Which low level code base have you tried this latest tool on? Official Anthropic commercials do not count.

Show threadsimonwNov 1
You're posting this comment on a thread attached to an article where Filippo Valsorda - a noted cryptography expert - used these tools to track down gnarly bugs in Go cryptography code.
Show threaddelusionalNov 1
These are not "gnarly bugs".
Show threadtptacekNov 1
They're not?
Show threadtptacekNov 1
They're also using "AI SAST tools", which: I would not expect anything branded as a "SAST" tool to find interesting bugs. SAST is a term of art for "pattern matching to a grocery list of specific bugs".