Cat HicksDec 15

"Goodhart’s Law is really a statement about the process of trying to make policy based on proxy measures of “internal states of complex systems” which are not themselves directly observable. "

Got to this piece from a link from Ben Recht and it finally nailed something that's always bothered me about the constant reference to Goodhart's law across software.

https://backofmind.substack.com/p/goodhart-as-epistemologist

goodhart as epistemologist

what did that "law" really say?

Dan Davies - "Back of Mind"
Show threadDaniel LakelandDec 15

@grimalkina

I'm not sure if the quoted text is a good assessment. Even taking what Goodheart actually said.

Suppose you turn on cruise control on your car, and then drive over a hilly freeway with varying wind speed and soforth. As you drive, the position of the accelerator pedal will be a primary *cause* of your speed. But your speed will have zero correlation with that pedal's position because your speed will be essentially constant while the pedal moves up and down

Show threadCat HicksDec 15
@dlakelan I mean, there are plenty of statistical models that are capable of handling more complex relationships than linear positive correlations? There IS a true causal relationship in what you're describing you just need a better theory of the mechanism to measure it correctly?
Show threadDaniel LakelandDec 15

@grimalkina

Oh yeah, for sure. I'm just saying that if you have a good feedback system with a good mechanism, it will destroy many statistical correlations that are observable in the data before you turned it on. That's more or less to be expected, and I think that's what Goodheart was trying to say

Show threadCat HicksDec 15
@dlakelan oh yeah, that's fair. It's hard to talk/write about this because people move so immediately into the realm of "what people are inferring about measures" and away from the comments about actual analysis. At any rate I agree with the "ought to target the thing you care about" as a counter to criticisms that all gamification will always render measures useless. But take your point that it's a simplistic summary of the actual law!
Show threadCat Hicks
@dlakelan and in general your example of cruise control is a great one, thank you for that, it reminds me a lot of the work we had to do when I was an efficacy scientist to convince school districts that "kept the failure rate steady despite increasing adversity" could count as a big educational success in the context of all that was happening for learners, as much as "test scores went up" in other cases
Dec 15 at 5:45pmWeb
Show threadDaniel LakelandDec 15

@grimalkina

Oh for sure. Use it please! It's not my example I heard it on the internet somewhere. If you keep the car speed constant while you go from flat ground to a steep incline that's already a big indicator your car is "succeeding" and I think it should be understandable to the education audience.