i love it when my program's execution conditions are "still active if unsure"

what in the fuck kind of world have we arrived at where in the optimal conditions where the "program" works fully as intended the program is "RUNNING" as long as the execution environment is "NOT SURE" (????????) if the program is "RUNNING"

this is what counts as benchmarking, with code links because this shit makes literally no sense and boils your brain if you try and read it:

• A list of 5 hardcoded 1-sentence prompts
• that get checked against a hardcoded set of if switches to give a name to the prompt, rather than idk labeling the prompts themselves. If a prompt doesn't have a matching word in that switch, the test is marked as pass.
• the task name extracted from the if/switch block selects from a map of functions that print the LLM output into a hardcoded string template > a python file
• where the python code has a hardcoded list of possible function names that the LLM could have generated like validate_email, is_valid_email, etc. if any of those names is defined, get the function by fucking evaling the name.
• if none is found, just look for ANY FUNCTION THAT TAKES ONE PARAMETER IN THE globals() DICT AND SEE IF THAT IS AN EMAIL VALIDATION FUNCTION
• call that in your tests by oh wait no yeah just completely redefining the test prompts in the test code again, that's fine. and the output too so the only thing the tests test are the tests when tested on test data.
• actually half the tests just test for the existence of keywords that are inevitably in the output since they are also in the prompt and they are the most sampled training data in the world
• when you actually run the benchmarks, the plugin actually causes one of the test cases to fail because it invents several function names
• the email validator it writes is just "anything with an @ and a period."
• the only output that's actually reported is lines of code, lower is better.

there is an as-yet unmerged PR to "fix the correctness benchmarks" and a "robustness audit" that is wonderful:

https://github.com/DietrichGebert/ponytail/pull/83

• someone raised an issue like "hey this makes the models worse"
• the LLM self-diagnosed the problem as being that all the tests are based off extracting code from fenced code blocks (true)
• the prior prompt text failed because all the LLMs just made up function names for one of the prompts, so now the prompt text just says the name of the function and what it should accept and return. (the models by default just copy/paste the most common response on stack overflow)
• two whole new set of tests, defined different than all the other tests, are added. one is just more of the tests testing the tests and the other is sweet mother of mercy what the hell is that
• the robustness audit passes if every test fails, the only thing that matters is if ponytail fails more than the baseline. therefore ponytail is good. untested is whether the test output is meaningful or possible to fail.