JAMA Touts Long COVID Exercise Trial with Clinically Insignificant Results; Most LC Exercise Trials Ignore PEM, Per Sick Times

By David Tuller, DrPH

In its current “Medical News in Brief” section, JAMA is touting and amplifying the questionable claims of a flawed trial to treat or prevent Long COVID published by one of the journals under its umbrella—JAMANetworkOpen. The JAMA headline: “Resistance Training Improves Long COVID Outcomes.”

Technically, the headline is true. In the trial, participants who received a program of resistance exercise reported statistically significant benefits on the primary outcome, a measure of physical capacity, compared to those who did not receive the intervention.

(The investigators also reported benefits on a few of many secondary outcomes. However, conducting multiple analyses increases the possibility of positive results from chance alone, and the investigators did not perform standard statistical tests to address that concern. That renders any claims about them questionable at best.)

The JAMA news-in-brief summary noted: “Intervention participants showed a greater overall improvement in distance traveled during the Incremental Shuttle Walk Test from baseline, an increase of 83 m vs 47 m in the control group after 3 months.” As I highlighted the other day in a post about the trial itself, there is less here than meets the eye. 

The news-in-brief item, like the trial, doesn’t mention a key fact: The findings for the primary outcome, a standard measure called the Incremental Shuttle Walk Test (ISWT), were not clinically significant. The difference of 36.5 meters between the two arms was way below the 47 meters the investigators themselves had designated as the minimal clinically important difference (MCID) for the ISWT. As they described in the trial protocol, they used that value in determining the desired sample size, citing an authoritative 2022 analysis.

That wasn’t the only issue with the MCID in the trial. In my prior post, I forgot to mention another critical point: Whatever benefits participants in the intervention arm did or did not attain, the final results in both groups remained far below the average performance of healthy adults on the ISWT.

Overall, the participants achieved a mean distance of 389 meters at the end of the 12-month exercise program. In contrast, a 2013 study of “age-specific normal values for the ISWT” found a distance of 824 meters for 40- to 49-year-olds, 788 meters for 50- to 59-year-olds, 699 meters for 60- to 69-year-olds, and 633 meters for 70-year-olds and up. In other words, the participants in this trial remained seriously disabled across the board—another salient detail ignored by the investigators.

To hype statistically significant but minimalist results as if they demonstrated substantial improvement is obviously not helpful to patients. It is also deceptive. No one relying on the JAMANetworkOpen paper or the JAMA news-in-brief follow-up for clinical guidance would have any idea how poorly both arms did compared to healthy adults unless they took the time to dig a bit more deeply on their own. 

Ignorng this information, like ignoring the fact that the difference between the two trial arms did not meet the threshold for clinical significance, is a form of misrepresentation. These methodological lapses reflect poorly on the investigators. Similarly, the journal’s decision to allow the paper to be published in this form reflects poorly on the integrity and competence of its editorial processes.

This isn’t the first time a Long COVID exercise trial has misrepresented the findings on the ISWT in this manner. In February, I wrote a post about a paper in European Respiratory Journal (ERJ), called “Post-Hospitalisation COVID-19 Rehabilitation (PHOSP-R): a randomised controlled trial of exercise-based rehabilitation,” in which something similar occurred. Participants in all arms of the trial—including the two intervention arms—performed way below normal levels for healthy people. In touting their findings, the investigators failed to address this point.

(Trudie Chalder was one of many co-authors of that study. As I wrote then: “It is a truth universally acknowledged (or at least universally acknowledged by smart researchers), that if the list of authors on an article includes Trudie Chalder, King’s College London’s mathematically and factually challenged professor of cognitive behavior therapy, then the article in question should most assuredly be expected to be short on, or utterly devoid of, intelligence and logical reasoning.”)

**********

Most Long COVID exercise trials ignore PEM, per Sick Times

A recent article in The Sick Times also addressed the issue with an article headlined “Less than 20% of Long COVID trials involving exercise even mention post-exertional malaise.” 

The news organization analyzed the registration records for clinical trials and found that fewer than 20% assess or take into account post-exertional malaise, the defining characteristic of ME/CFS. Many people with Long COVID—the proportion is debated—experience PEM and quality for ME/CFS diagnoses. For these patients, exercise programs geared toward rehabilitation can be contra-indicated.

Here’s an excerpt from the piece, written by Toronto journalist Simon Spichak: 

“According to an analysis conducted by The Sick Times, PEM is systematically neglected in research trials testing exercise interventions that could trigger it. Dozens of low-quality trials on exercise in Long COVID haven’t provided any answers and potentially harmed trial participants, and many never ended up publishing their results.

“The flawed idea central to these trials is that exercise is a panacea, and gradually increasing it could treat Long COVID, Jaime Seltzer, researcher and scientific director of the advocacy group #MEAction, told The Sick Times. As a result, the research won’t advance our understanding of the disease’s actual underlying mechanisms, said Seltzer, calling it “money down the drain.”

You can read the full article here.

(View the original post at virology.ws)

#mcid #meaction

Another Exercise Trial with Clinically Insignificant Findings

By David Tuller, DrPH

A recent study from JAMA Network Open, called “Resistance Exercise Therapy After COVID-19 Infection: A Randomized Clinical Trial,” demonstrates some of the flaws that so often mar papers in this field of research. The trial’s reported results do not warrant the optimistic conclusion that the intervention “may be a generalizable therapy for individuals with persisting physical symptoms after COVID-19 infection.”

Not surprisingly, the study received positive media attention. An article disseminated on Yahoo, headlined “Long-COVID Patients Should Focus on This Training,” included the following quote from the study’s lead investigator and primary author: “Our study shows the benefits of strength training for recovery after COVID-19 and suggests that people suffering from persistent symptoms after a COVID-19 infection could benefit from this type of training.” 

Not so fast! A core problem here is one we have seen before: Statistically significant results are being touted as evidence of benefits even when they fall below the threshold deemed to be clinically significant.

The trial was conducted in Scotland by a team of investigators led by Colin Berry, a professor of cardiology at the University of Glasgow. Participants included 233 patients with confirmed cases of COVID-19, falling into three categories: those who had been hospitalized and discharged but had continued to experience symptoms for at least four weeks; those who had not been hospitalized but continued to experience symptoms for at least four weeks; and hospitalized patients convalescing from their acute disease.

This mish-mash of patients seems rather odd. The cases in these three categories would likely differ significantly from each other. Tossing them all together in one bigger study does not seem like the best way to achieve the most interpretable results for each separate group. Whatever.

All participants received care as usual. Half were randomized to also receive a 12-week resistance exercise training program, tailored to their individual levels of disability. In this case, the designated primary outcome was the Incremental Shuttle Walk Test (ISWT), a commonly used measure for assessing cardiovascular fitness and exercise capacity, at three months after randomization—essentially, at the end of the training program. The trial also included a host of secondary indicators–self-reported, others objectively measured.

It is important to note that no one reasonably objects to exercise interventions for Long COVID patients unless they suffer from post-exertional malaise (PEM), in which case these therapeutic approaches are contra-indicated. It is widely understood that exercise is generally good for people! Those recovering from acute illness are likely to gain muscle strength from resistance exercise—unless this approach is contra-indicated because of PEM. 

In any event, the participants in this trial do not seem to have been assessed at baseline for PEM, although data are provided for its presence at at the end point. It would have been interesting to see if there were any changes from start to finish. 

**********

Reported benefits lower than the minimal clinically important difference

But here’s the main problem for the investigators and their primary outcome. Per the statistical analysis plan outlined in their protocol, the ISWT’s threshold for clinical significance—known as the minimal clinically important difference (MCID)—is 46 meters. The investigators cited an authoritative 2022 paper from the European Respiratory Society (ERS)–“Use of exercise testing in the evaluation of interventional efficacy: an official ERS statement”–for this MCID. They used it to informed their power calculations for determining the needed sample size. (MCID values for a given measure can vary per the methodology used, population studied and other factors.) 

In the trial, participants in the intervention arm performed better than the control group by an average of 36.5 meters. Let’s acknowledge the obvious: 36.5 meters is quite a bit lower than the MCID of 46 meters that the investigators themselves referenced in their statistical analysis plan. Oops!

Researchers with a robust sense of integrity would have reported such a salient detail, however disappointing or embarrassing or contrary to their expectations. Yet the JAMA Network Open paper made no mention of it, effectively disappearing a very revealing data point. This omission, in and of itself, serves to misrepresent the findings and is arguably a form of research misconduct. Readers deserve to know that the trial’s primary outcome did not reach its own threshold for clinical significance.

(A published comment to the article highlighted the omission and the fact that the results fell below the established MCID; in their rebuttal, the investigators simply ignored the point, in effect affirming the validity of the criticism. These issues have also been discussed on the Science for ME forum. Another published comment on the journal’s site highlighted some serious ethical concerns with the trial; the comment author posted an X thread about it.)

Interestingly, a pre-print of the article posted earlier this year did seek to navigate the MCID issue without acknowledging a problem. Rather than citing the MCID of 46 meters referenced in the protocol, the pre-print flatly declared that “the effect size exceeded the minimum clinically important difference of 35.0.” The reference for this lower MCID was a different study–an earlier one, from 2019—that calculated an estimate of 35 to 36.1 meters in patients with chronic obstructive pulmonary disease. 

That section of the pre-print is not part of the published text. Perhaps peer reviewers raised questions about it. Perhaps the investigators themselves thought better of including it, for whatever reason, given that the 2022 paper was a much more comprehensive overview of walking tests; also, one of the two co-authors of the 2019 article was a co-author of the 2022 paper as well. In any event, as long as the investigators fail to confront the MCID issue or offer a reasonable account of their shifting positions, it is hard to take what they have written at face value.

And how about those many secondary outcomes? A few yielded positive results, but many did not. In particular, as the investigators acknowledged, “measures of physical activity, including accelerometry and patient-reported fatigue or perception of frailty, did not improve.” 

Given that the trial was unblinded, responses to any self-reported or subjective measures would be infused with an unknown amount of bias. This tendency would likely be reflected in modestly positive responses from those who received the intervention–even in the absence of actual benefits. 

Beyond this key factor, the investigators failed to include any attempt to adjust for multiple outcomes. This is a standard strategy to compensate for the fact that, given a great many different statistical tests, some are likely to come up as positive by chance alone. Considering the amount of testing conducted in this study, adjusting the results for multiple outcomes might have meant that some positive findings were no longer statistically significant. (Whether positive findings for these secondary outcomes were clinically significant is another question.)

In other words, don’t pay attention to any of the rosy assertions arising from this trial–and the slanted way in which the results have been presented.

(View the original post at virology.ws)

#exercise #longCovid #mcid

🚨New blog post 📝: Your Study Is Too Small (If You Care About Practically Significant Effects)

#effectsize #precision #poweranalysis #research #Psychology #MCID #SESOI #samplesize

https://open.substack.com/pub/mzloteanu/p/your-study-is-too-small-if-you-care?r=3b457w&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Trudie Chalder Is Co-Author on Another Bad Exercise Paper

By David Tuller, DrPH

It is a truth universally acknowledged (or at least universally acknowledged by smart researchers), that if the list of authors on an article includes Trudie Chalder, King’s College London’s mathematically and factually challenged professor of cognitive behavior therapy, then the article in question should most assuredly be expected to be short on, or utterly devoid of, intelligence and logical reasoning.

This is certainly the case with a recent publication in European Respiratory Journal titled “Post-Hospitalisation COVID-19 Rehabilitation (PHOSP-R): A randomised controlled trial of exercise-based rehabilitation.” Professor Chalder is one of more than three dozen co-authors, so it is unclear how much she can be held responsible for the article’s poor quality and unwarranted claims. Nonetheless, this new trial continues what appears to be her impressive streak of being involved with scholarship that can accurately be described as rubbish.

The new trial, conducted at the University of Leicester and Northumbria University, was funded by the Medical Research Council and the National Institute for Health Research. The design was not terrible. Specifically, unlike some other Long Covid rehabilitation studies, this one excluded participants with post-exertional malaise (PEM). That’s a good move, given that the presence of PEM is a contra-indication for an exercise-based rehab program.

So this was a study of Long Covid patients who do not meet criteria for an ME/CFS diagnosis—a key point.

The real problem here is that the reporting of the results stinks.

In the study, 181 participants who were experiencing prolonged symptoms after a Covid-related hospital stay were randomized to either an eight-week face-to-face exercise rehabilitation program, an eight-week remote exercise rehabilitation program, or care as usual. The primary outcome was the change in the Incremental Shuttle Walking Test (ISWT). Among the many secondary outcomes were questionnaires measuring health quality of life and symptom burden.

The conclusion: “Exercise-based rehabilitation improved short-term exercise capacity in Post-COVID syndrome following an acute hospitalization.”

First, let’s note that the conclusion does not explicitly state that the study is not about “post-COVID syndrome” overall but only about non-ME/CFS “post-COVID syndrome.” That’s a major limitation of the findings that should have been emphasized prominently throughout, given the significant numbers of Long Covid patients who do experience PEM and qualify for ME/CFS diagnoses. Health care providers will read the conclusions and assume they can be extrapolated to all Long Covid patients. That is clearly not the case.

Beyond that unacceptable oversight, let’s review the data in a bit more detail and see if that claim holds up.

The face-to-face intervention group had a drop-out rate of 29% and the remote intervention group had a drop-out rate of 39%. These drop-out rates are quite high. Remarkably, the article includes no substantive discussion of this. It is hard to argue persuasively that an intervention is successful or effective when so many participants apparently decided not to continue with it, for whatever reasons. Did they find it unhelpful? Too difficult? Harmful? We don’t actually know.

The article also overlooks the fact that the participants were, on average, almost as unhealthy after the eight-week intervention as they were beforehand. After the intervention, the average increase in the IWST for the face-to-face and remote groups was, respectively, 52 meters and 34 meters more than the increase in the care as usual group. But the average meters walked remained way, way below the levels of healthy people in the same age range.

The average age of participants in the trial was 59. A 2013 study called “Age-specific normal values for the incremental shuttle walk test in a healthy British population” found that the average distance walked during the ISWT by those in their 40s, 50s, 60s, and over 70 were, respectively, 824 meters, 788 meters, 699 meters, and 633 meters. By comparison, those in the face-to-face group increased from 285 to 312 meters, and those in the remote group from 353 to 388 meters.

It should have been obvious to any intelligent or even minimally competent researcher with that an exploration of both of these issues–the high drop-out rate and the continued poor health of the participants–was essential to put the purportedly “positive” findings in context. There is simply not that much positive to report about interventions that left participants severely disabled and that substantial numbers were unable or unwilling to complete.

Oh, and on top of that, there were null results for the trial’s seven quality-of-life and symptom burden questionnaires: EuroQol five-dimension five-level questionnaire (EQ5D), Patient Health Questionnaire (PHQ9), the Generalised Anxiety Disorder (GAD7) 7-item scale, Dyspnoea-12, the Functional Assessment of Chronic Illness Therapy Fatigue Scale (FACIT), the DePaul Symptom Questionnaire, and the Montreal Cognitive Assessment (MoCA). In other words, whatever incremental improvements might have occurred, participants did not report any overall benefits in the trial’s many subjective measures of well-being.

And to mention one other odd point…According to a 2019 study cited by the authors, the “minimal clinically important difference” (MCID) for the ISWT is 35 meters. In other words, while the results for the face-to-face intervention surpassed that threshold, the results for the remote intervention did not quite reach it. Yet here’s the opening of the discussion section:

“In this fully powered randomised controlled trial, we demonstrated that both face-to-face and remote exercise-based rehabilitation significantly improve exercise capacity compared to usual care alone in those previously hospitalised with COVID-19. These between group improvements exceed the established MCID (35m), highlighting improvements of clinical relevance in those with post-COVID syndrome.”

This last statement is simply not true in relation to the remote intervention, at least when it comes to the final, adjusted, intention-to-treat analysis. It is either a mistake or a deliberate effort to fudge the facts. I assume the latter, because it is very obvious that 35 is a bigger number than 34. With more than three dozen people on the manuscript, it is hard to believe that no one noticed this discrepancy. Either way, this indisputable error requires a correction. (To be clear, a correction won’t make the rest of the paper any better.)

In fact, the authors might have pointed out that there are multiple studies of the MCID of the ISWT, such as a 2008 analysis that found it to be 47.5 meters, and one from 2015 concluding that it was 70 meters. Even thought they chose to cite the MCID most favorable to their argument and ignore the others, they still presented false information to bolster their case.

So here’s the bottom line: Despite some marginal improvements among those who actually were able to or decided to complete the interventions, the trial documented that exercise-based rehabilitation failed dramatically to restore participants’ health. Moreover, participants felt no better subjectively on any measures than beforehand. Given those telling details, along with the fact that significant numbers of participants abandoned the trial’s intervention arms, the boast that these rehabilitation programs “improved short-term exercise capacity” is hard to take seriously.

**********

Disclosure: My academic position at the University of California, Berkeley, is largely supported by donations to the university via the campus crowdfunding platform from people with ME/CFS, Long Covid, and related disorders.

(View the original post at virology.ws)

#MCID #TrudieChalder