John RegehrI'm fundamentally a tool builder, and LLM coding agents work one million times better if you give them good tools, and I wrote a thing about this
LR@regehr article ok but. i don't understand why these metric-providers need to be called "oracles". it's the same pretension as the use of the word "story" in ticket systems. oracles deliver divine prophecies. what these things do is underwhelming in comparison. it irritates me greatly.
@lritter this is the standard term from software testing
Fabian Giesen@regehr @lritter to be fair, I think the original usage goes back to things like non-deterministic Turing machines, where an Oracle whispers to you the exact right sequence of non-deterministic choices to make to get the result you want, and in that case (certainly in the presence of things like uncomputability etc.) some form of divine intervention (or at least divine wisdom) might legitimately be required.
@regehr @lritter it's just like with anything, you start up building a religion to give you divine input to find a counterexample to the Goldbach Conjecture or whatever, but once the infrastructure exists and works you stop asking the Oracle questions like "what is the meaning of life" and start asking it questions like "is it sunny outside? I can't be bothered to pull back the curtains"
@regehr @lritter as an aside, the idea that the arcane (the concept) in Arcane (the show), rather than purely a mechanical facet of how that world works, is strongly implied to have agency and starts "acting out"/getting pissy when treated as a mere utility to be tapped on a whim, is IMO a genuinely inspired world building idea and a cool fix for this kind of thing
@regehr @lritter Don't know when complexity theorists started using it, but evidently it was already standard terminology by 1973: https://web.archive.org/web/20230319210201/https://cse.ucdenver.edu/~cscialtman/complexity/Relativizations%20of%20the%20P=NP%20Question%20(Original).pdf (see usage on first page)
@regehr @lritter Turing used it in this sense in https://pure.mpg.de/rest/items/item_2403325_2/component/file_2403324/content (1939, p.40 in PDF) so it seems safe to say that usage of the term is about as old as CS itself, if not older
@rygorous @regehr the references you gave are legit.
this ties into a mathematical intuition i had about the cyclical nature of time¹, which allows us to see into the future by stringing together causations in reverse.
the turing machine is a pure forward process, whereas our "oracle" is a hard-won emulation of a reverse information process, much like our maxwellian daemons do in fact need an energy source² to reverse entropy.
—
¹ only where events are repeatable
² fortunately abundant
@regehr ok. after fabian explained the origins i now think your use of the word is accurate, and it hints that zero degrees of freedom are impossible to achieve.
it is an extension of the original least squares problem: every optimizer games the error metric.
here, the game has more brownian motion and the quantifier isn't fuzzy, but you also struggle to find a complete, ungameable metric.
essentially you can not prove oracle completeness.
@regehr all this, and your post, inspired this thread:
LR (@[email protected])
1. Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." 2. every optimizer targets the error metric. 3. therefore, every error metric is a bad measure. #s0up