I'm fundamentally a tool builder, and LLM coding agents work one million times better if you give them good tools, and I wrote a thing about this

https://john.regehr.org/writing/zero_dof_programming.html

zero_dof_programming

@regehr article ok but. i don't understand why these metric-providers need to be called "oracles". it's the same pretension as the use of the word "story" in ticket systems. oracles deliver divine prophecies. what these things do is underwhelming in comparison. it irritates me greatly.

@regehr ok. after fabian explained the origins i now think your use of the word is accurate, and it hints that zero degrees of freedom are impossible to achieve.

it is an extension of the original least squares problem: every optimizer games the error metric.

here, the game has more brownian motion and the quantifier isn't fuzzy, but you also struggle to find a complete, ungameable metric.

essentially you can not prove oracle completeness.

@regehr all this, and your post, inspired this thread:

https://mastodon.gamedev.place/@lritter/116299707577071848

LR (@[email protected])

1. Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." 2. every optimizer targets the error metric. 3. therefore, every error metric is a bad measure. #s0up

Gamedev Mastodon