I knew the Dunning-Kruger effect wasn't valid, but thanks to this post I finally understand why: "The Dunning-Kruger Effect is Autocorrelation"
https://economicsfromthetopdown.com/2022/04/08/the-dunning-kruger-effect-is-autocorrelation/
The Dunning-Kruger Effect is Autocorrelation – Economics from the Top Down

Do unskilled people actually underestimate their incompetence?

Economics from the Top Down
@joeposaurus Thanks for sharing this. I find it pretty remarkable that it took so long to figure this out. — And it makes me wonder what other (more or less) famous/seminal papers are flawed in one way or another. Or any paper for that matter.

@joeposaurus

Ha!

Also good to know that the fact that I think that I fully get this argument doesn't make it any more likely that I'm an idiot.

@joeposaurus

That the test to measure effect was faulty doesn't mean the results are faulty ("broken clock is right twice a day"). DK is so easily observable in real life that it can't summed up as false, even if a test to measure it should prove to be badly executed.

@iju @joeposaurus From the article: "what happens to the Dunning-Kruger effect if it is measured in a way that is statistically valid? According to Nuhfer’s evidence, the answer is that the effect disappears."

Edit: although: "the spread in self-assessment error tends to decrease with more education. In other words, professors are generally better at assessing their ability than are freshmen."

@bluekieran2 @joeposaurus

Yes, if we assume the test to measure the effect was valid, and the only problem wasn't the analysis.

Edit: and as for your edit: big part of academic training centers on identifying your own lack of understanding.

@iju @joeposaurus I guess you can't go wrong ending with "further research is required" ;)

@joeposaurus And even the graph from the original study pretty much says outright "people are bad at guessing their relative performance, but they tend to guess closer to their actual performance."

Which flies 100% against the way people invoke "Dunning-Kruger." It's just saying "lol this person is stupid" dressed up in phony intellectualism.

@joeposaurus Everything before Figure 11 is a misinterpretation of D-K’s graph.

@joeposaurus sorry, but no. The author of this "debunk" is significantly confused.

His argument for autocorrelation completely fails to see that what the research looks for - and found - is not a correlation, but a lack of perfect correlation (and how divergent it is).

Their counterexample - best illustrated by figure 7 - is a good show of their confusion: yes, in a sample of people that have a completely uncorrelated notion of perceived vs actual performance, the DK effect is at full glory.

@joeposaurus .. that equivalent to say that in a society in which individuals assess their performance completely randomly, people are quite terrible at it (and for the poor performers, it's gonna be seen as overconfidence, while for the best performers, it's gonna be underconfidence).

Also, I'm not claiming DK study is super solid (because I don't know), but this debunk is rubbish.

@joeposaurus In other words, the null hypothesis (no effect) of the DK effect is points on a straight line (like the "x=x" example). All other patterns (once denoised) show some kind of effect to exist.

@vriesk @joeposaurus

Glad you point out the flaw in this invalid attempt to deny the existence of the DKE . Well argued, thanks!

@joeposaurus I love this article. This doesn't attack our observations, but it tears the math used to defend the concept to shreds
@ATLeagle @joeposaurus except even if performance and assessment are totally uncorrelated, as in this article’s model, that is DK effect, compared to the idea that people can assess their own competence. https://andersource.dev/2022/04/19/andersource.github.io/2022/04/19/dk-autocorrelation.html goes into it a bit.
I can’t let go of “The Dunning-Kruger Effect is Autocorrelation”

Organizing my thoughts regarding an argument about the Dunning-Kruger study and statistics in general

andersource
@joeposaurus It's a bad post. The D-K effect is that people are bad at estimating their own skill, to the point that the regression coefficient between perceived test score and actual test score is very weak (but not zero, or, as in some popular depictions, negative). Mechanically it means people who do worse on tests overestimate their ability and people who do better underestimate it, but that's another way of representing the finding that the regression is meh.

@joeposaurus It was an interesting read, but I have the impression that it fails to address what I thought was an essential point of the DK effect: i.e. that the spread between real and perceived ability was bigger for the first quartile than for the last quartile.

The spread is bigger on the real data, meanwhile it's exactly the same on the random data.

Am I missing something there? 🤔

@jpetazzo @joeposaurus @jpetazzo Woah, that article is so wrong! Of course if the charts are based on statistical noise the results are going to look like this. Perceived ability is a key value in these studies, and the point of DK effect is to show that it is not always correlated with actual ability. If the results are random, perceived abilities will of course not be correlated with actual abilities... If anything, that study shows that random evaluation is not correlated with actual ability.
@joeposaurus Dunning-Kruger is one of many studies meant to give an air of legitimacy to hierarchy (i.e. that some people "deserve" to be ruled over). It gives individuals an excuse to stop listening to people challenging their power.
@joeposaurus
Um ... yeah ... the FayeDunaway-BettyCrocker effect. I know all about it. 👍

@joeposaurus

/IroniemodusON

Blödsinn ich versteh von Mathe gar nix und weis aber genau das es den Dunning-Krüger Effekt gibt!

Erlebe ihn doch andauernd, jene die überzeugt davon sind das die Erde eine Scheibe ist plärren die ganze Zeit rum und wenn sie irgend wann merken das ihnen die anderen schon gar nicht mehr zu hören schweigen sie und Wählen die Hohlerdepartei.

DAS KANN KEIN ZUFALL SEIN!

@joeposaurus

/goes on with his Besserwissermode ..

And what about all those who jumped on the Dunning-Krueger-Effect train because they have no clue about Math, statistics and auto-correlation but are self proclaimed specialist who are mad about getting opposed?

They all proved the effect to be true!

you see!
you see!

@joeposaurus count me among the folks who don't get/agree with the argument in this post. I'm not sure it's wrong, but it feels off to me. No opinion on whether the DK effect is real though
I can’t let go of “The Dunning-Kruger Effect is Autocorrelation”

Organizing my thoughts regarding an argument about the Dunning-Kruger study and statistics in general

andersource

@jayalane @shauna @joeposaurus

This great, but the web address is wrong (its two addresses smushed together).

@jayalane @shauna @joeposaurus Thanks for this. This article did a really good job articulating — and putting into statistical terms — what I was thinking when reading the original: the dependent/independent bit is really *the* key insight.

I also share the author’s irritation with the original (and discussion around it). Part of a bad trend.

Incidentally, that link didn’t work for me but I clicked around and found it… at what claims to be the same URL? Weird.

@joeposaurus Here's an interesting rebuttal to why this article may not be valid: https://andersource.dev/2022/04/19/dk-autocorrelation.html
I can’t let go of “The Dunning-Kruger Effect is Autocorrelation”

Organizing my thoughts regarding an argument about the Dunning-Kruger study and statistics in general

andersource

@joeposaurus Seems to me that what the data says is that:
a) Everyone is bad at assesing their own skill level.

b) Everyone, with the exception of the most skillful quartile, tend to over estimate their skill level, where as if the data were totally random, then you'd expect the two lines to cross at the median point on the x-axis.

You can make the point that the original claim was that the least skillful were uniquely bad at assessing their own skill, there is still a real effect there.

@joeposaurus It's a fun little criticism of the Dunning-Kruger effect, but I think it's a fundamental mistake to evaluate knowledge and learning as a statistical problem. Taxonomies of learning outcomes often assess the complexity of an argument, and monocausal explanations will be seen as a sign of poor understanding. Yet, the student often feels confident because they know an explanation that is true, and they don't see yet that it is also too simple.
@joeposaurus Okay but we're going to need a new term to describe all those tech bosses who think they're great but aren't.
@joeposaurus The top of this absurdity is the fact, that Nuhfer et al use a invalid measure of "skill". Education level is not a valid measure of skill. This is not proving the invalidity of an effect – it's proving the invalidity of the writer and the authors of this study. The invalidity of the paper by Dunning & Kruger does not prove the absence of the effect.
@joeposaurus A rather good explanation of the origin of D-K… Have you seen this, @MarcAbrahams . I understand the argument and yet examples of outrageous D-K abound. or is it an illusion?
@joeposaurus Wäre das was für @Fischblog oder Kolleg:innen?
@joeposaurus
Read beyond the headline, citizen. The author goes on to unintentionally prove the existence of Dunning-Kruger effects in their own experiments.

@joeposaurus

> "The Dunning-Kruger Effect is Autocorrelation"

I might be wrong, but I think this analysis is based on a conceptual misunderstanding.

Not the fact that the two curves are correlated is interesting (can be replicated with random data) but the fact that they are not perfectly correlated is.

The problem of "too juicy to not be true" might apply (but the other way as intended).

This convinces me more:

https://andersource.dev/2022/04/19/dk-autocorrelation.html

(Thanks @jayalane )

Anyway: Need more thinking.

I can’t let go of “The Dunning-Kruger Effect is Autocorrelation”

Organizing my thoughts regarding an argument about the Dunning-Kruger study and statistics in general

andersource
@joeposaurus
The first sentence of the article actually demonstrates the validity of Dunning-Krueger.
D/K doesn’t assert anything about skill, it’s an observation about the ability to understand.
@joeposaurus ok, I actually tried reading this gibberish, and I’m starting to think the author is trolling. Or…
@joeposaurus
Fabulous, the author over looks the possibility that the carefully crafted data set the study created is capable of excluding the effect….(?) but may contain flaws introduced by their own over confidence.
@joeposaurus
The fault lies not in our data sets,
But within ourselves.
@joeposaurus does this mean that Dunning and Kruger over estimated their own scientific ability?
@joeposaurus this article is fucking dumb. They replicate the DK experiment and say the imp part from that experiment is the difference between real test and percceived score so they plot y-x against x. Then do random plot graph and get the same y-x plot. Not shit, Sherlock. In both cases that's just the graph of y = -x + C. The shit they specifically were pointing out is compared to percentile. There's a reason it was in the x-axis in DK. Plotting y = -x is shit we did in maths lessons at 16
@joeposaurus TL:DR essentially in this replicated experiment, they added an autocorrelation themselves and then were surprised when the results they obtained were due to autocorrelation
@engravecavedave @joeposaurus
They were overly confident their model could address the DKE.