Breaking Down Cass Review Myths and Misconceptions: What You Need to Know – The Quackometer

https://lemmy.world/post/14288447

Breaking Down Cass Review Myths and Misconceptions: What You Need to Know – The Quackometer - Lemmy.World

Seen the “98% of studies were ignored!” one doing the rounds on social media. The editorial in the BMJ put it in much better terms: “One emerging criticism of the Cass review is that it set the methodological bar too high for research to be included in its analysis and discarded too many studies on the basis of quality. In fact, the reality is different: studies in gender medicine fall woefully short in terms of methodological rigour; the methodological bar for gender medicine studies was set too low, generating research findings that are therefore hard to interpret.”

Why Hilary Cass' NHS report is wrong about trans health care

The recent NHS report from Dr. Hilary Cass isn't the take-down of gender-affirming care that conservatives want it to be.

Advocate.com
Ms Reed is not an objective source. Nor does it appear she has much experience with systematic reviews.
It’s not a lie, they were mostly dismissed even according to your own article, they were dismissed and rolled into one conclusion.

101 of 103 studies were not dismissed. All systematic reviews classofy their source studies based on the quality of the work. Of the 103, two were classed as high quality, 58 as moderate quality and the remaining 43 as low quality. For synthesis, only high and moderate quality studies were drawn on. That’s more than half, not 2%.

So yes, Erin is lying.

You can’t say she’s lying until we do a systemic review of why the Cass study dismissed everything but 2 studies for the numbers it used to reach us conclusion. You can’t say she’s lying without that review no more than I can support Erin by reading each study that was dismissed. What I can tell you is that dismissing that many studies is not normal scientific analysis. It reeks of bias.

You can’t say she’s lying until we do a systemic review of why the Cass study dismissed everything but 2 studies

This is the lie. They didn’t dismiss all but two studies, they actually included 60. More than half of the 103 studies identified for the review.

What I can tell you is that dismissing that many studies is not normal scientific analysis.

It’s key part of synthesising multiple sources into a meta-analysis. Including poor quality studies dilutes the quality of the overall analysis.

It reeks of bias.

By design, it’s biased towards higher quality research.

Synthesis is a paragraph summary inclusion only it means they didn’t use data from the study, it is dismissal. I’m done arguing that with you.

They have absolutely used the data from those 60 studies. You can read where they say explicitly that in the report if you cared to.

You are utterly mistaken and firm on your conviction, these are not the qualities of skepticism.

“Don’t seek refuge in the false security of concensus”

That’s not what synthesis means. I’ve written synthesis reports before and the data you include from those reports is once you have dismissed them as inaccurate, is entirely selectivel whatever you want them. We even have a phrase for it in law, Summarily dismissed.
And of the 103 reviewed they included data from 60. It is a lie to say they “dismissed all but two.”
So you don’t know what you are talking about. Gotcha.

You can read the review for yourself

t.co/82Rjs2L1pA

Let me know where you find the bit where they dismiss 101 out of 103 papers.

There only inclusion in the report is to half explain why the were discluded, exactly what I said.
They did not dismiss 98% of the data.
Putting 98% of the data in supplementary table 4 is not including the data.

Supplementary Table 4 (from the first review) is a list of each of the 53 studies included in the review and how the were scored based on the Newcastle-Ottawa scale.

The “data” is in supplementary tables 6 and 7. Only studies that were scored as low quality were excluded from the synthesis.

“They dismissed 98% of data” is a lie.

No it’s not. None of the dismissals are scientifically supported, and their data is incomplete presented in a way that isn’t inclusive of what those studies actually say.

Nothing was dismissed at all (and “statistics” has nothing to do with it so curious to mention it).

Studies were scored for quality on the well established Newcastle-Ottawa Score. High and Moderate quality studies were included in the synthesis. Low quality studies were not, but their outcomes are still reported.

Outcomes from each study were included in tables 3, 5, 6 and 7.

'They dismissed 98% of the data" remains a lie.

You can’t remove a study from your a scientific paper without having statistical analysis to back it up. Each of those removed studies all had statistical analysises of how confident they remained in their data even with the gaps. Because there aren’t completed 100% studies in science it just doesn’t happen so you use the data you have. And the idea that some trans people don’t make it to the completetion of a study due to personal reasons or even suicide isn’t that rare. Not using 98% of the data because of that would be stupid.

You can’t remove a study from your a scientific paper without having statistical analysis to back it up.

You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality.

Each of those removed studies all had statistical analysises of how confident they remained in their data even with the gaps.

Studies are also scored low on quality if, for example, they don’t control for important sociodemographic confounders. Study that do control these, will have more reliable results.

You can read how the scoring works in supplementary material 1.

Not using 98% of the data because of that would be stupid.

“They dismissed 98% of the data” remains a lie. Repeating it doesn’t change anything.

You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality.

That’s wrong when you are trying to be scientifally correct. A science paper without math isn’t science my dude.

It’s remarkably common in systematic reviews. You give the impression that this is a new or foreign concept to yourself and are just encountering these ideas for the first time.

Search on pubmed or the bmj or the Cochrane library for other systematic reviews using the Newcastle-Ottawa score. You’ll trip over them.

And comparing trans healthcare data to Facebook polls is ridiculous

One of the studies reviewed recruited patients over Facebook and polled them.

Again I’ve written these reports. It is absolutely not common practice to discluded data.

Again I’ve written these reports.

I am forced to strongly doubt this given your wholly misunderstanding of the basic concepts on assessing methodical quality…

Certainly, you’ve never authored a systematic review for a reputable medical journal.

But don’t take my word for it…

…cochrane.org/…/13_5_2_3_tools_for_assessing_meth…

It is absolutely not common practice to disclude data without scientific reason and analysis.

You mean such as using a method like the Newcastle-Ottawa score to assess data quality?

It is explicitly taught not to do it that way in college.

If your college course covered systematic reviews and didn’t include a review of study assessment methods, ask for a refund.

And is not scientific to do that without a statistical threshold

Statistics are not required to assess that a study without a comparator is weaker than one with.

“They dismissed 98% of data” remains a lie.

13.5.2.3 Tools for assessing methodological quality

The Newcastle method is not seen as a scientific basis for dismissal on its own.

98% of the data was dismissed in the synthesis and were not used in the conclusion that there wasn’t enough scientific evidence to support transition when 98% of the science says that is wrong.

It’s not used for “dismissal” it’s used to score studies on their likelihood of bias. Studies without appropriate controls for example are more susceptible to bias than those with.

98% of the data was dismissed in the synthesis

Demonstrably false, only low quality studies were excluded from the synthesis which account for less than half of the 103 reviewed. A lie is a lie no matter how often repeated.

and were not used in the conclusion that there wasn’t enough scientific evidence to support transition when 98% of the science says that is wrong.

That’s not what the conclusions say, for example:

Synthesis of moderate-quality and high-quality studies showed consistent evidence demonstrating efficacy for suppressing puberty

And

Evidence from mainly pre–post studies with 12-month follow-up showed improvements in psychological outcomes

“They dismissed 98% of data” remains a lie.

That was published a month before Cass came out and so hasn’t anything to do with the two systematic reviews being discussed above. It doesn’t even mention them.

I’m uncertain what expertise a business graduate can bring to assessing the quality of a systematic review in medicine.

Readers are free to Google the author and subsequently make a judgement on their objectivity on the subject matter.

And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with. That like the Cass report are not scientific reasons to disclude the criticism or the data respectively.

And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with.

If they made a scientific argument about these review papers under discussion I might but this is just a polemic using unscientific language like “cis-supremacy” in a low impact obscure journal.

That like the Cass report are not scientifically sufficient reasons to disclude the criticism or the data respectively.

Newcastle-Ottawa scoring is a scientific method for weighting the methodical quality of scientific studies.

And I can garuntee you that the Cass report was not peer reviewed like all of the studies they dismissed were because it would have been torn apart.

It was peer reviewed since thats BMJ policy, unless you have evidence to the contrary. There is even a link on the online edition of both reviews for you to submit a rapid response pointing out all their flaws which I would encourage you to do.

That’s the real litmus test of scientific debate.

Interestingly some nice fellow DM’d me with a link to “Patient Zero” of the “they dismissed 98% of the data” myth.

twitter.com/benryanwriter/…/1779671152148857212

And of course, everyone has doubled down rather than admit they read the wrong paper. A better “litmus test” of scientific debate is humbly correcting yourself when shown to be wrong.

“They dismissed 98% of the data” remains a lie.

Benjamin Ryan (@benryanwriter) on X

I now understand how activist Alejandra Caraballo @Esqueer_ started the misinformation machine that got so many to believe the false claim that the Cass Review simply discarded non-randomized controlled trials. It's because Caraballo cited the *wrong* systematic lit reviews.🧵⬇️

X (formerly Twitter)
98% of the data could be summarized in one sentence. Trans healthcare and transition medicine works. 98% of the data comes to that conclusion with vast consensus across disciplines and fields. 98% of the data was discarded.

98% of the data was discarded

Liar.

98% of the data was discarded

98% of the data was discarded

It was not. All studies that scored high or moderate quality made it into the synthesis. That’s 60 out of the 103 looked at, that’s not 2%.

No I’m just explaining the process and why it isn’t complete yet. Or even valid yet

You are speculating, based on nothing.

And show me that the Cochrane library ever discarded a study using the criteria even once yet alone with the same level as the Cass report and I’ll write them

Here’s one I found in <7 seconds

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

If you want to find more simply search the Cochrane library for reviews with “Newcastle-Ottawa” in the main body of text. It seems like this is new to you.

For something that illustrates the problem with the Cass report read www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/

The relevance of a paper from 2003 to a systematic review published last week is certainly questionable but it seeks you’re trying to imply that Cass discarded anything except RCT’s. The didn’t and that’'s myth #2 from the original Quackometer article.

Where will the goalposts move next?

Goal pays haven’t moved and I’ve already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.

Goal pays haven’t moved and I’ve already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.

You haven’t pointed out, let alone substantiated, any. If you truly believe you have then I implore to use the rapid response function on the bmj site and communicate these oversights to the editorial team immediately. I’d be eager to know what their reply is.

I don’t need to is already happening within the scientific community of which I am merely a part.

Don’t abdicate responsibility to someone else, you’ve clearly got a firmer grasp of the issue than the editorial board of the British Medical Journal. You would be neglecting your duty as “part of the scientific community” to abdicate responsibility on such an important matter.

Indeed the whole medical established must be told about the critical flaws in the Newcastle-Ottawa scoring system before other medical scandals are allowed to happen. Imagine having that on your conscience.

Lol first sign that you might actually be human.

So strange that everyone waited over 20 years and 100’s of systematic reviews in medicine and science before, serendipitously, discovering that the Newcastle-Ottawa Scale was infact no good during these two particular reviews into trans care in the UK.

Just what are the odds?

No the Cass report is just misusing the scale. It’s not a disqualifying tool and the scale still has uses with which is cause for more analysis

That’s a new goalpost. It’s being used by Cass exactly the way it’s supposed to by scoring studies based on their susceptibility to bias.

If you’d bother to read that similar systematic review on postoperative inflammatory bowel disease you would have seen the exact same usage.

Again that’s a joke to do that.

Damn here’s another “joke” about contraceptives and bone fractures

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And one another yellow fever and HIV

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And influenza vaccines in cancer patients

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And there’s another 96 on the first search tab alone!

Just what are those clowns at the Cochrane Library up to eh?

Relevance?
They all use the same Newcastle-Ottawa system to score studies based on their likelihood of bias in the exact same way the Cass reviews do. The method you described as a joke.
It’s not an indicator of bias, no causal study has been done to show that there is one

Studies that self select their cohort and don’t include adequate controls are more susceptible to bias than those that do otherwise. It’s a vital part of the systematic review process.

You can read more about it here www.ohri.ca/programs/…/oxford.asp

Ottawa Hospital Research Institute

Ottawa Hospital Research Institute