@glyph I am constantly torn between the obvious (to me) knowledge that LLM are just reconstructing parts of their training corpus and the beliefs of management and vibe coders that the results they see are truly novel.
Where I am, having a serious discussion about LLM merits is taboo currently, so the steam train continues to barrel towards the proverbial broken bridge.
@glyph Coming from a data analysis background, I am very interested in verifying my assumptions using data. The method in your article seems to be useful to me to try to guage whether we should be using LLM, but at the corporate level rationality seems to have been throw under the bus.
Our current measurements don't even attempt to target productivity. The measurement currently is literally how many hours per day I (and the rest of the team) were using an LLM. There is an incentive currently to use unlimited tries in an attempt to get it right without having to think about it directly.
Part of me thinks that management think they can watch the LLM prompts (which are all recorded and visible to management) and when someone gets something right the prompt itself will be an asset to the company. But anyone who has tried to replicate a prompt sequence knows that it breaks every time the model is retrained or you try it on another companies LLM. That may make it an asset in the very short term, but the costs are being completely discounted.