Mastodawn

Mre. Dartigen [maker mode]Jul 16

When I suggest that people will game whatever metrics we put in place, I'm often met with shocked indignation. We would never game the numbers! And yet we do.

I took my car in for service this morning and I was asked if it was ok that they split the bill across two transactions. "You're being measured on number of cars through?" I asked. The answer was obviously yes, and this way I counted as two cars.

It's not just a matter that the numbers are now wrong, we have now introduced waste into the system. There were two credit card transactions rather than one. Two receipts instead of one. There was additional time for the workers to explain why they wanted to do it this way. Overall, this was complete waste, but because they felt they were being judged on the count of cars through, it was justified.

If people think that they'll be judged based on measurements then they'll game those. The more judgement, the more the numbers will be inaccurate, and the more waste will be introduced into the overall system.

You might think that I'm opposed to measuring anything then but that's not at all true. I'm a big proponent of measuring those things we want to improve. I'm just a realist and recognize that we have to design our measurements very carefully. If we measure the wrong things, or in the wrong way, we'll drive the wrong behaviours and that's our problem to solve.

@mike_bowler I guarantee that FedEx drivers are measured on the number of delivery ATTEMPTS rather than the number of packages that were delivered. they seem to "attempt" to deliver a package to me quite often without setting foot out of the delivery van.

Blackoverflow Jul 15

@mike_bowler
My current belief is that the ones designing the measurements must be the "workers" themselves.

Measures can be used to judge work.
Measures can't be used to judge people.

But even then, i fear, measures a group is used to follow die hard, even if they show bad side effects.

In my experience, no, it's almost always management. There's a constant demand to show the people above you that their numbers are going up and to the right. And to stratify their employees into "better" and "worse" with quantitative metrics rather than qualitative metrics.

This even bleeds into SEO when your product/company needs to have a lot of 5 star reviews to get more business.

Or your company needs to have big quarterly earnings for the board to give you a bonus.

Or a manager showing how good they did at cutting costs without too much lost productivity.

@blackoverflow @mike_bowler

Legit_Spaghetti Jul 16

@JessTheUnstill @blackoverflow @mike_bowler This whole discussion is a classic example of Goodhart's Law: If a measure becomes a target, it ceases to be a good measure.

https://en.wikipedia.org/wiki/Goodhart%27s_law

Goodhart's law - Wikipedia

@Legit_Spaghetti
To which I will add this quote:

What gets measured gets managed - even when it's pointless to measure and manage it, and even if it harms the purpose of the organization to do so - V. F. Ridgway

@JessTheUnstill @blackoverflow @mike_bowler

@JessTheUnstill @blackoverflow @mike_bowler Pretty sure they don’t mean that current metrics *must have been* selected by the workers. They mean that for a metric to be useful, it *needs to be* selected by the workers.

Ah okay, misread it.

But even still, some work simply isn't well quantified. Even if it's something like Agile where the team itself assigns "story points", when management tries to normalize that across teams and determine relative velocities and such it becomes a huge mess of story point inflation.

@bob_zim @blackoverflow @mike_bowler

Blackoverflow Jul 16

@JessTheUnstill
I completely agree with you.

If measurements are misused by management, they lose any usefulness and become dangerous.

Maybe, with great precaution, it may be possible that a team designs measurements intended for reporting to management.

But I don't have much hope that this works often.

@bob_zim @mike_bowler

groff 🇺🇦Jul 15

@mike_bowler I think what they were doing was charging you double.

Mike Bowler Jul 15

@geoffl LOL. They didn’t, but I could see someone unscrupulous trying that.

@mike_bowler
We talk a lot about how truck drivers need better training due to speeding, aggressive driving, etc.
Most truck drivers are paid by the mile. If you could increase your wage by 10-15%, doing what practically everyone else is doing wouldn't you? And wouldn't you be frustrated by the guy pulling a U-Haul trailer slowing you down? Wouldn't it suck if the more stressful your job is (traffic , etc) the less you are being paid?
Every truck driver should be paid a fair hourly wage.

Mike Bowler Jul 15

@corvus I used to know a trucker, who I think was independent. All I can say for sure is that he managed his own gas costs and had optimized his driving speed based on that. For him, driving too fast lowered his profit, just as driving too slow did.

@mike_bowler
I was always a company driver. My brother owned his truck working as an independent but was still tied to a particular carrier that paid by the mile. Vanishingly few truckers are truly independent, getting their own loads and making their own schedules.

❣️a standard deviant +/- gravy Jul 16

@mike_bowler
i nudge that it's not the metrics, it's the incentive structures wrapped around metrics that is usually what needs adjustment.
?
@corvus

Mike Bowler Jul 16

@melioristicmarie @corvus Yes, it's the fact that the metrics have become targets, that drives the wrong behaviors.

❣️a standard deviant +/- gravy Jul 16

@mike_bowler
targets for? targets for prizes given to the highest? the lowest? targets for hitting people with sticks for higher or lower?

the value of the metric, as you point out, can be made meaningless by incentives.

Mre. Dartigen [maker mode]Jul 16

@melioristicmarie @mike_bowler @corvus In many cases as well, the incentive becomes so punishing as to incentivise gaming.

When the result for low metrics is 'you will lose your job', people will game the metric because they don't want that outcome. If the metric is poorly designed such that it's impossible to always get high numbers that makes it even worse.

(Which happens more often than many think, and is not explicitly stated more often than many think. Management might not outright say 'if you don't meet the KPIs then you'll be fired', but whoever keeps having low KPIs is going to be first on the list when they decide to cut positions.)

Don_Rubiel Jul 15

@mike_bowler
This is related but hard to read...
https://en.wikipedia.org/wiki/Atrocities_in_the_Congo_Free_State
"The baskets of severed hands, set down at the feet of the European post commanders, became the symbol of the Congo Free State. ... The collection of hands became an end in itself. Force Publique soldiers brought them to the stations in place of rubber; they even went out to harvest them instead of rubber ... They became a sort of currency. They came to be used to make up for shortfalls in rubber quotas, to replace ... the people who were demanded for the forced labour gangs; and the Force Publique soldiers were paid their bonuses on the basis of how many hands they collected.

A missionary holds up a Congolese man's arm at the elbow, and points to his missing hand

In theory, each right hand proved a killing. In practice, to save ammunition soldiers sometimes "cheated" by simply cutting off the hand and leaving the victim to live or die. Several survivors later said that they had lived through a massacre by acting dead, not moving even when their hands were severed, and waiting till the soldiers left before seeking help. In some instances a soldier could shorten his service term by bringing more hands than the other soldiers, which led to widespread mutilations and dismemberment"

Atrocities in the Congo Free State - Wikipedia

Snaprails Jul 15

@mike_bowler When a measurement becomes a target it ceases to be a measurement.
Seen it so many times over the years. Figures get fiddled, business becomes stats driven, customers and staff suffer.
I once explained to a manager why the stat he cared about so much was meaningless nonsense. The only reply was “but we can’t not have measurements” 🙄

Maria Langer | 📝 🎬 ⚒️🛥️Jul 15

@mike_bowler @kajord I saw this way back in the 1980s when I was an auditor. The company I worked for had a division that did sales and the employees got bonuses based on the number and value of sales. They would go to existing customers, cancel services, and then sell them the same service as a new sale. Talk about waste!

Charlie Stross Jul 16

@mike_bowler This is EXACTLY why the USSR's central planning system persistently failed to work, in miniature. Setting productivity targets with simple performance metrics gives workers an incentive to game the metrics, not do the job better.

jaKa Močnik Jul 16

@cstross also, mao's great leap forward, at tremendous human cost. @mike_bowler

@mike_bowler the UK introduced all sorts of targets for public services. The results were predictable, everyone scored above average but there was no significant improvement in the actual services.

Gaming the system became more important than actually improving the service. Decades later the problem persists...

Also see Lake Woebegone effect about averages...

as a large fashion model,Jul 16

@mike_bowler > If we measure the wrong things, or in the wrong way, we'll drive the wrong behaviours and that's our problem to solve.

I think the core issue here is not the measurement, but the judgment on those measurements, as you mentioned in the previous paragraph. Just adding this because I don't think the "right" kind of measurement can (or should) ever be achieved when it's a target. Even in a perfect setup people can still forge records etc. Unless you go 100% totalitarian surveillance.

Mike Bowler Jul 16

@dngrs It is possible to get a set of overlapping metrics to counteract that gaming effect, but the point does still stand that once it's a target, people will game it.

Anꞇóin Ó B.Jul 16

Looks like the examples section for the Wikipedia page on Perverse incentives is set to grow.

My favorite is the cobras. 🐍 🐍 🐍 🐍

adamrice Jul 16

@mike_bowler There’s a management aphorism: “you measure what you treasure,” but I’m convinced it’s the other way around.

My job (technical translation) is one for which meaningful metrics would be almost impossible to obtain, but management needs its KPIs, so it has set some criteria that are easy to measure, and based on them, I am doing great.

Karl Baron Jul 16

@mike_bowler The fast food places around here have all started having those screens showing when your order is ready. And every time, the orders are marked as ready/disappear from the screen long before they are actually ready. 100% that is because they are gaming the order completion time metrics.

Ryan Finnie Jul 16

@mike_bowler One of my most annoyed fast food pet peeves: "I'll have your total at the window"

It means 1) they're gaming the system to lower drive-thru times and 2) my order will probably be wrong because they didn't actually enter it.

@mike_bowler
The hard part is aligning goals. Traditionally we've depended on bosses for that, whether they're dukes or CEOs, control, rather than consensus, and enforcement is never quite as good at alignment as it thinks it is. Thus drift, and work to measure.

If you can get consensus, both on what is to be done, and in how to see it, then good measurement is excellent feedback. Without that it's just another set of numbers with questionable relevance.

Jon Wood Jul 16

@mike_bowler I sometimes wonder on a global scale just how much waste and general crap service is being generated by metrics intended to do the opposite. Support calls hung up causing the customer to requeue at 59 minutes hold time. Split payments like you say. People pushed to use a crappy mobile app when buying coffee. Hospital staff spending 90 minutes filling in paperwork for a 60 minute consultation. It’s mind boggling.

Mike Bowler Jul 16

@jon "perverse incentives" is the general term for this. All the undesirable behaviour that's driven by poorly thought out metrics. Once you start to look for it, it's everywhere.

Neil Kandalgaonkar Jul 16

@mike_bowler I want to slightly disagree with you. We should always have the best metrics we can design. But as you say, all metrics can be gamed.

Which means that optimizing metrics has diminishing returns, rather quickly. Our real focus should be organizational ethos.

And that is a subtle and tricky thing to maintain. At minimum workers must have faith they will be fairly judged, by humans who understand their context – the very context that metrics often ignore

D. Moonfire Jul 16

@mike_bowler Goodhart's Law ruins it for everyone.

Tony Yarusso Jul 16

@mike_bowler If we have a ticket open too long middle management tells us to close it and recreate a new one with the same request so the reports to upper management look better.

Klaus Stein Jul 16

“I'm a big proponent of measuring those things we want to improve” – this is fine and completely different from measuring things to judge people.

The problem is not the measuring, the problem is thinking that any scheme that will make the measured number better will improve the situation without understanding that applying anything will change what you are measuring.

Measuring tells you something about the situation. Now try to improve the situation, not the numbers. You may need a different measurement afterwards.

@mike_bowler Any known metric will cause behavior to change so your metrics should be unknown to all 😉

Emmy, Gendermancer Jul 17

@mike_bowler this obviously applies so much more broadly, but: as a person that works in manufacturing, and quality control at that, this is incredibly frustrating. Line go up or else, and line can and will go up if it needs to. And it does.

"We're not tracking your productivity to punish you," but there's no way to avoid that. "Please report mistakes and losses. We are doing this to help you, and we understand that everyone makes mistakes." You literally cannot guarantee that, whether you're a supervisor, a department head, or the sole owner. If you have numbers, you have to use them, and something WILL persuade you to punish someone over them eventually.

So the system gets gamed, and not just by the low level employee. Few people are willing to admit "I accidentally destroyed this $3,000 tool" or "I scrapped $15,000 of parts and thereby also guaranteed we will be two weeks late on delivery," and a lot of leaders are unwilling to accept that setback. If the risk is low, can be rationalized away, or the blame could be shifted if need be, leaders will cover up or ignore things.