Romeu MouraPeople are using « tokens used » as productivity metric ?! « Tokens used »?!?!? That’s like, the first time « lines of code created » gets beaten for the « worst metric of software engineering » 🫠
Alexander Janßen@romeu It's just the new "1st post!" from Slashdot back in the good old days ;-)
Badly-read literary snob@romeu I thought it was more like Vim golf 🧐
Michael Newton@[email protected] @[email protected] "I spent more money, give me a raise!"
Jonas Winje@romeu that's one way to double down on "so foolish as to book that count on the wrong side of the ledger"
MR.e@romeu
I once had to explain to HR that a code check in was not a valid metric for a write up of a senior developer. They were angry he wasn't doing as many check ins as the junior developers. I walked them through that he was a code reviewer and that he had more impactful check ins, but they were not convinced.
I then pulled up one of their power points and showed them the version history. I hypothetically asked them why they were not more consistently productive and some days didn't have any updates.
It helped a lot.
I still feel crazy thinking about it.
JKB
Steve Purcell
Einar W. Høst@romeu Checkmates Goodhart's law by being a terrible metric even when not measured.
@romeu It's probably the best possible metric for how hard you're cargo culting though, so it depends on your aims.
@romeu A more useful and accurate analysis may be as follows: tokens used will be seen as an adoption metric, and implicitly as a loyalty metric.
@romeu Believing that AI is bringing about a productivity revolution, management will see AI as an opportunity yes, but primarily as a necessity, and an existential threat.
@romeu They will fear that unless everyone gets on board the AI juggernaut, the organization will be disrupted by new-fangled competitors who are reaping the insane productivity benefits that they have been convinced exist.
@romeu Hence the problem to be solved will be seen as one of adoption, and the obstacle to be overcome, the friction caused by recalcitrant, stubborn, backwards dinosaurs. And tokens used is an adoption metric, albeit one that is extremely vulnerable to Goodhart's law.
@romeu If you don't use any tokens, certainly it must mean that you're part of the problem and undermining the express will of management.
@romeu The worst part is that the fear of the existential threat is not necessarily entirely unfounded, on at least two grounds. Regardless of whether or not there is any real productivity boost due to AI.
@romeu Customer management is likely to have bought into the same productivity narratives as your management, and will be looking to get a radically better deal than before. Whether or not that is possible for anyone to deliver.
@romeu And there will be no shortage of would-be competitors popping up claiming that they are in fact reaping those benefits and can deliver on those wishes. Whether or not that's true, both in the short and long term, doesn't matter: harm can be immediate.
@romeu It may all be mirages, but people make all kinds of stupid decisions based on mirages.
J.@romeu
I read a story somewhere of a developer setting up a token burning script that builds badly designed cities in Sim City...
I read a story somewhere of a developer setting up a token burning script that builds badly designed cities in Sim City...
The Dupdob
Markus Gerstel@romeu number of left clicks
M Schommer@romeu
It's like Dotcom eras "Cash burn rate", but this time it's AI data centers burning the planet.
It's like Dotcom eras "Cash burn rate", but this time it's AI data centers burning the planet.
Jean-Baptiste Fidélia@romeu Someome came to me the other day asking why my token count was so low. I didn't realize that was a productivity metric. WTF.
Pete Alex Harris🦡🕸️🌲/∞🪐∫It's like measuring car performance by gallons per mile instead of miles per gallon.
Sebastian Stautz@romeu Could be "air breathed".
Code lines are least an outcome, but tokens are a resource you spent to create outcome.
Like breathing, typing, Watts used by your computer, etc.
Code lines are least an outcome, but tokens are a resource you spent to create outcome.
Like breathing, typing, Watts used by your computer, etc.
Le Notaire@romeu I do remember when at work they were using the "committed lines of code" metric. One week I managed to do a SINGLE line change. Fixing a bug which haunted us for months. As they finally acknowledged, the main issue was not the change but knowing exactly where it was needed. The metric had been since buried
an actual bus@lenotaire @romeu $9995 for knowing where to tap.
Ruben De Smet@romeu Hmm. What about kLOC/token? 🤔
standev@romeu I for one measure my car's quality by how much fuel it burns
Graeme Hilton@renardboy @romeu Those who are heavily invested in fossil fuel companies certainly do.
John Maxwell@romeu I'm gonna prompt me a new minivan!
Jamie Gaskins
Lucas
Dan Sugalski@romeu It's been interesting watching people compete to see who can burn the most money, time, and resources. I expect this will come out about as well as you'd expect.
Procrvstinvtion@romeu I’ve read about token allotment being a contract negotiation. “Salary isn’t the big argument anymore, token allotment is” to quote an ai bro
Isaac Ji Kuo@romeu I asked LLM AI to tell me what productivity metric we should use and it told me LLM AI tokens used.
I also asked my coke dealer what productivity metric we should use and he told me grams of coke used.
I don't trust my coke dealer but I sure do absolutely trust my LLM AI.
Daniel McLaury
Graham
johnofrobotz@romeu as soon as "Prompt Engineers" became a thing, this was bound to happen. Also terrible, but at least SLOC metrics had review and testing built in.
Paul CantrellOur whole society is rife with this: mistaking consumption for productivity
Dr Neenah Estrella-LunaYuuuuuup.
doragasu
Мя ��
jelte
viq
josh g.
esa
Alan Langford 🇨🇦🧤🧊摏@romeu Hey Claude, write me some code that can be stored in the database such that it looks like a valid data blob, then once a month checks for {keyword} on my LinkedIn profile. If found, decode, execute and delete the code. When run, the block of code should randomly change the "account_balance" field by amounts ranging from -5% to +5%. Finally, replace this and related prompts in your log with values that resemble my other work.
flabberghaster@romeu Paying taxi drivers based on the amount of gasoline they burn
Laura
EduCoder@romeu "hey chat, review this entire repo, so you can change the colour of a button for me."
*walks away from the computer for the rest of the day*
Lux 
Aedius Filmania ⚙️🎮🖊️I have a good soluce :
- please generate code
- please add unit tests 100% coverage
- please add e2e tests 100% coverage
- please ingest the logs of the tests to see if all error are useful
draNgNon
lemgandi
AI Is Really Weird
If you like this piece and want to support my independent reporting and analysis, why not subscribe to my premium newsletter? It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including