I got into an argument with #Microsoft #Copilot today.
I asked for PowerShell code to trim the first N digits from every filename in a directory. Testing showed it actually removed 2·N characters. When I reported the bug, Copilot insisted that was impossible and said I must have run it twice.
I knew the cause and the fix, but I treated it as a learning exercise to see what it would take to get Copilot to correct itself.
1/4
I verified I ran code only once, reran the test, and reported identical results of second test. Copilot argued in detail how it was impossible for the code to rename each file twice.
I've been down this path before, both with humans & LLMs, unable to convince them of what I can plainly see. So, I had Copilot modify the code to print each old and new filename.
Renaming '1234elephant.dat' → 'elephant.dat'
Renaming 'elephant.dat' → 'hant.dat'
I pasted that output into the chat.
2/4
With this evidence, Copilot declared:
Okay—that output is the smoking gun. The pipeline is seeing the renamed file as a new item while it’s still enumerating the directory, so it’s processing each file twice.
The fix is to take a snapshot of the files first, then rename from that fixed list so changes to the directory don’t affect what gets processed.
Here’s a version that will not double‑rename, with logging:
3/4
I reviewed & tested the new code. The problem was finally fixed.
So, this is the 100 billion dollar planet-destroying machine that will replace us?
I've heard it said LLMs are an intern you must closely supervise, but unlike a human intern that rapidly improves, the AI intern has enormous inertia and won't get much better, at least not any time soon.
I had a little fun and vented a little bit at. At least the AI openly admitted the mistake—something you don’t always get from a human.
4/4
Scott Miller 🇺🇦 🇺🇸
Rob
Olivier Guinart