JonathanSep 3, 2023

If you exercise some degrees of freedom in conducting your experiment, you can make almost any stupid hypothesis (appear to) satisfy p<0.05.

False positive rates can be as high as 60%.

These researchers (appeared to) prove, to p<0.05, that listening to kids music made participants 1.5 years younger.

Show threadJonathanSep 3, 2023
Show threadJonathan
You can make anything "statistically significant".
Sep 3, 2023 at 9:09pmWeb
Show threadJonathanSep 3, 2023

Searching for the statistically significant effect.

But why?

So you have something to publish.

Show threadJonathanSep 3, 2023

Authors conducted a real study of whether listening to kids music made the participants feel younger.

It did.

They then tested whether it MADE them younger ...

Show threadJonathanSep 3, 2023

...

It did.

Show threadJonathanSep 3, 2023

By adjusting the degrees of freedom in the experiment, you can generate frighteningly high false positive rates.

That is, you can adjust the methods to get a positive result that is NOT TRUE.

Show threadJonathanSep 3, 2023
Just collect data, and if it fails, collect a bit more and test again until you get a positive.
Show threadJonathanSep 3, 2023
Here's how they suggest researchers avoid this bias.
Show threadJonathanSep 3, 2023
Same applies to RCTs. In addition to being the wrong tool for testing masks, etc. this is no doubt why so many fail, esp the meta studies where authors have huge degrees of freedom to exclude whatever "biased" or "methodologically wrong" study they want.
Show threadJonathanSep 3, 2023
Here is an excellent demonstration of the effect. Results - NOT significant at the beginning,
- BECOME stat sig at a certain point, but
- MORE data shows it is in fact not stat sig.
Show threadJonathanSep 3, 2023
Authors conclude with my point, science is the search for truth, yielding to pressures of the every-day such as publish or perish.
Show threadJonathanSep 3, 2023

Which is why the World Health Organization tossing money to idiots to run stupid meta studies to continue the "droplet" tradition is just egregious patronage to support the status quo of the system.

This money could go to real science.

What a waste.

What bad science.

What a blow to credibility.

Show threadJonathanSep 29, 2023

Main p-test thread
https://mastodon.social/@jmcrookston/111120455774214705

Thread about Randomized Controlled Trials
https://mastodon.social/@jmcrookston/111003157891215097

Show threadTJ RadcliffeSep 3, 2023
@jmcrookston This was a huge problem for particle physics in the '80s, as the complexity of detectors increased so we had thousands of channels to choose from, allowing us to put arbitrarily complex cuts on the data. Spend a few hours like that and you can find *enormously* significant peaks in almost any dataset, as happened at the German GSI lab, which spawned a minor industry for a few years and turned out to be garbage. https://link.springer.com/article/10.1007/BF01290325
Can extended objects explain the GSI e+ e− lines? - Zeitschrift für Physik A Hadrons and nuclei

The anomalous electron-positron coincidences observed in heavy-ion collisions have been interpreted as signal for the pair decay of hitherto unknown neutral objects with masses around 1.8 MeV. We discuss the decay modes of such extended composite particles when they are bound to a nucleus. In particular we investigate the angular correlation of the emitted pair and the competing single-photon decay channel. We confront the particle hypothesis with recent negative results from experiments searching for resonances in Bhabha scattering. The induced pair decay of a metastable 1++ state in secondary collisions with target atoms is discussed as a possible explanation.

SpringerLink
Show threadJonathanSep 3, 2023
@tjradcliffe Interesting!
Show threadTJ RadcliffeSep 3, 2023
@jmcrookston My PhD work was a precision measurement of the electron-positron annihilation cross-section in the energy range of 1 to 3 MeV, which was part of the search for evidence of this non-existent "GSI particle". We didn't find it, but did confirm that Dirac's prediction for the cross-section was correct in that energy region, which is kind of dotting an i, but it would have been huge if the value had been wrong!
Show threadMathew SamuelSep 3, 2023

@jmcrookston
Heh. You need to stop when you randomly have a subset of data that validates your preconceived notion.

Basic science stuff.

🤦🤦

Show threadmyrmepropagandistSep 3, 2023

@jmcrookston

Are there any awards for excellent well-designed studies that found nothing?

Awards for replication studies that either confirm or fail to confirm or muddy established results?

Show threadVirginicusSep 3, 2023
@futurebird @jmcrookston I admire this group https://www.jasnh.com/
Journal of Articles in Support of the Null Hypothesis