Greg Egan“A study that claimed OpenAI’s ChatGPT can positively impact student learning has been retracted nearly one year after publication. The journal publisher, Springer Nature, cited “discrepancies” in the analysis and a lack of confidence in the conclusions—but not before the paper racked up hundreds of citations and made the rounds on social media.”
David J. Atkinson@factsentinel @gregeganSF It mirrors the massive issue of poor replication of results. It’s true in all the sciences, I think medicine has it the worst but it is true mostly in social and soft science areas. Papers get retracted sometimes, but the papers that cited the retracted papers don’t get the scrutiny needed. I think authors rarely know if a previous reported result important to their own research has been retracted. This is why skepticism is good. Never believe one study. You need lots, and results that replicate.
𝓙. 𝓜.@meltedcheese @factsentinel @gregeganSF and well, who has time to chase papers two levels deep (i.e., a paper in the bibliography of a paper cited by the paper you're currently looking at) or more these days? ;)
@gregeganSF @factsentinel @tpfto Exactly. We should have AI track this for us. The AI tech is called “non-monotonic reasoning” and it is the basis for considering future hypothetical situations given certain assumptions. The hard for AI, and extremely hard for people, is unwinding all the inferences you’ve made that are traceable to assumption that should be retracted — e.g., a result from a retracted paper. Same idea.
@gregeganSF @tpfto @factsentinel And THIS is why the #scientific method is nevertheless the most credible source of Truth: everything is questionable and subject to disproof. It’s hard, but we do spot and disprove errors in results Alternatives are proposed and tested FOREVER.