Kenneth Baillie
Ed Palmer@kennethbaillie the editorial discussed the conf ints. effectively in terms of probability mass and gave a direct interpretation of them as if they were Bayesian. But of course a direct probability interpretation isn’t possible without casting priors.
Romain Barthélémy@DocEd @kennethbaillie is it about the shameful spinning in the conclusion of BLING III?
@rombarthelemy @DocEd What was the spin?
@kennethbaillie @rombarthelemy this was for A2B, but also interested to hear the spin!
@rombarthelemy @kennethbaillie where’s the spin?
@DocEd @kennethbaillie second sentence. It’s just the definition of the CI. It doesn’t belong to a conclusion. It is aimed at producing confusion suggesting that the results is almost positive to be in concordance with the metanalysis they also published
@rombarthelemy @kennethbaillie politely disagree. The primary analysis used pre-defined adjustment criteria to increase precision and landed with p 0.04. So in fact, the conclusion is an undersell. Of course the alpha threshold is arbitrary.
@DocEd @kennethbaillie that’s two primary outcomes, so p-value threshold should have been lowered then…
@rombarthelemy @kennethbaillie I personally would have only reported the adjusted analysis. I think being this purist to frequentist analysis is problematic. If you are this pure, then you need to also correct for family wise error for all previous studies. Silly in the context of a totally meaningless and arbitrary threshold. 0.05 isn’t magic.
@rombarthelemy @kennethbaillie could I recommend this book if this is something that interests you. One of the best things I’ve read on this subject.
@rombarthelemy @kennethbaillie important to remember that frequentist methods were developed for agriculture. They are surprisingly ill suited to interpretation of clinical trials. But we have to work with what we have.
Lars Mølgaard Saxhaug@DocEd @rombarthelemy @kennethbaillie I’m not sure I agree, as much as I favour bayesian methods. Frequentist methods have a history well before Fischer and frequentist focus on error controll does make sense from a regulatory point of view where you wan’t to controll the false discovery rate among numerous potential new drugs (yes, that’s not alpha I know).
@DocEd @rombarthelemy @kennethbaillie I think the larger problem is following habit rather than thinking through the actual question and the best way to answer it. Always using the same arbitrary alpha, always applying two-sided point null hypothesis tests etc are more at fault than frequentism itself. If bayesianism takes over but with the same pervasive statistical illiteracy, we’re no better off.
@load_dependent @rombarthelemy @kennethbaillie yes, that’s certainly important. I would like to see more decision theory incorporated into the planning of clinical trials. I tend to have a lower alpha requirement for mortality end points than other non-patient centered outcomes. And to be fair to Fisher, he was quite against the Neyman-Pearson approach to “error” and promoted the use of p values as continuous measures of evidence.
@load_dependent @rombarthelemy @kennethbaillie the more I look into this area, the more I see that there are political or historical reasons under pinning the way we do things, rather than scientific. For example, many simple tests used today are in use because they have quite elegant analytic solutions. But that just isn’t necessary anymore when you can use things like MCMC and prioritise using a model that is suited to the situation.
@load_dependent @rombarthelemy @kennethbaillie which applies to frequentist and Bayesian methods alike.
@DocEd @kennethbaillie they suggest that the risk of a clinical effect that is not detected by the study is high. But. This risk is known. It’s 10%. By design
@rombarthelemy @DocEd FWIW, I think the quoted text above is an honest attempt to describe our best assessment of the truth. Seems like a perfectly reasonable interpretation of 95%CI. You might ask, why not do an another trial so you can find out? But I think opportunity cost of doing that is important. There are probably better things we could all do with our time and energy.
@kennethbaillie @DocEd sure but it is the case for every CI of every estimate of any RCT. So the most probable is that the effect is not clinically significant. And sure it’s time to move on. This sentence adds nothing but confusion
@rombarthelemy @kennethbaillie I would say that your statement is actually incorrect. That certainly is not the most probable interpretation. This is why frequentism is so difficult to interpret. Frequentism is about long term error control, so making probabilistic statements at the trial level (like you’ve just done) is fraught with difficulty.
@rombarthelemy @kennethbaillie in fact, from a pure frequentist standpoint, the probability that the confidence intervals contains the true effect is either zero or one (it either does not, or does). We just don’t know which, so we control the long term error rates so that we are more often correct in our assertions. Using those rules of frequentism, I will assert that there is an effect here. And I am likely to be correct in the long term at an error rate that I am comfortable with.
@DocEd OMG it reminds me how happy I am to have left Twitter
@rombarthelemy sorry if the discourse has reminded you of Twitter. Apologies if my rebuttal has been too robust.