Cat HicksDec 15

"Goodhart’s Law is really a statement about the process of trying to make policy based on proxy measures of “internal states of complex systems” which are not themselves directly observable. "

Got to this piece from a link from Ben Recht and it finally nailed something that's always bothered me about the constant reference to Goodhart's law across software.

https://backofmind.substack.com/p/goodhart-as-epistemologist

goodhart as epistemologist

what did that "law" really say?

Dan Davies - "Back of Mind"
Show threadDaniel LakelandDec 15

@grimalkina

I'm not sure if the quoted text is a good assessment. Even taking what Goodheart actually said.

Suppose you turn on cruise control on your car, and then drive over a hilly freeway with varying wind speed and soforth. As you drive, the position of the accelerator pedal will be a primary *cause* of your speed. But your speed will have zero correlation with that pedal's position because your speed will be essentially constant while the pedal moves up and down

Show threadCat HicksDec 15
@dlakelan I mean, there are plenty of statistical models that are capable of handling more complex relationships than linear positive correlations? There IS a true causal relationship in what you're describing you just need a better theory of the mechanism to measure it correctly?
Show threadDaniel LakelandDec 15

@grimalkina

Oh yeah, for sure. I'm just saying that if you have a good feedback system with a good mechanism, it will destroy many statistical correlations that are observable in the data before you turned it on. That's more or less to be expected, and I think that's what Goodheart was trying to say

Show threadCat HicksDec 15
@dlakelan oh yeah, that's fair. It's hard to talk/write about this because people move so immediately into the realm of "what people are inferring about measures" and away from the comments about actual analysis. At any rate I agree with the "ought to target the thing you care about" as a counter to criticisms that all gamification will always render measures useless. But take your point that it's a simplistic summary of the actual law!
Show threadDaniel Lakeland

@grimalkina

This point made in the linked article is good though. you HAVE to have measurements, and you HAVE to respond to them. You don't have any choice other than to be passive and useless. The key questions should always be things like "how does this measurement relate to the thing I care about" and maybe "how does that relationship change with time and adversary action?"

Dec 15 at 5:55pmWeb
Show threadDaniel LakelandDec 15

@grimalkina

Another one I heard and I think is correct, is that basically **all** measurements are indirect measurements. When I look at a volt-meter needle I'm measuring the position of the needle, not the voltage of the circuit. Even if it's an LCD I'm measuring the position of the dark spots on the display. And then, maybe your voltage is actually the voltage across a resistor which lets you infer current, and that current infers magnetic field, and that magnetic field infers position of...

Show threadCat HicksDec 15
@dlakelan yes this is very familiar to social scientists :). We seem to have a very explicit awareness of operationalization that other areas are allowed to elide.
Show threadDaniel LakelandDec 15

@grimalkina

A servo motor, and that servo motor infers position of a valve, and the valve position infers water flow, and the water flow infers power generation of a turbine, and the power generation of a turbine infers current in another circuit, and that infers voltage on a transmission line, and all of that lets you decide if you should cut off a circuit breaker which relates to the safety of a hospital operating room... etc etc even in physical science or engineering it's all indirect

Show threadDaniel LakelandDec 15

@grimalkina

The difference in engineered systems is that we get to design the system so that the relationship has a reasonably "flat" transfer function, that is the position of the needle tells us a lot of low noise information about the power transmission line.