Mastodawn

I argue that there is some understanding in the #psychometrics and professional communities that use #TestScores that evidence of quality is tentative as any statistical result.

And in my personal view, the different methodologies #COSMIN has developed (e.g., https://link.springer.com/article/10.1007/s11136-018-1798-3) or is currently pushing forward (https://doi.org/10.1186/s13643-022-01994-5) are tools to aggregate available evidence exactly for this reason: each individual study and estimate offers an incomplete picture.

COSMIN guideline for systematic reviews of patient-reported outcome measures - Quality of Life Research

Purpose Systematic reviews of patient-reported outcome measures (PROMs) differ from reviews of interventions and diagnostic test accuracy studies and are complex. In fact, conducting a review of one or more PROMs comprises of multiple reviews (i.e., one review for each measurement property of each PROM). In the absence of guidance specifically designed for reviews on measurement properties, our aim was to develop a guideline for conducting systematic reviews of PROMs. Methods Based on literature reviews and expert opinions, and in concordance with existing guidelines, the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) steering committee developed a guideline for systematic reviews of PROMs. Results A consecutive ten-step procedure for conducting a systematic review of PROMs is proposed. Steps 1–4 concern preparing and performing the literature search, and selecting relevant studies. Steps 5–8 concern the evaluation of the quality of the eligible studies, the measurement properties, and the interpretability and feasibility aspects. Steps 9 and 10 concern formulating recommendations and reporting the systematic review. Conclusions The COSMIN guideline for systematic reviews of PROMs includes methodology to combine the methodological quality of studies on measurement properties with the quality of the PROM itself (i.e., its measurement properties). This enables reviewers to draw transparent conclusions and making evidence-based recommendations on the quality of PROMs, and supports the evidence-based selection of PROMs for use in research and in clinical practice.

SpringerLink

To which degree) the term "validity" includes all quality criteria and qualities one can consider to describe the use of #TestScores is a matter of heated discussion in fields related to the philosophy of measurement. But it is probably safe to assume this for the stuff covered in the "Standards".

For a look into the discussion I recommend Kathleen Slaney's book on Construct Validity (https://link.springer.com/book/10.1057/978-1-137-38523-9) and several of her papers deal with this topic.

#Psychometrics

Validating Psychological Constructs

SpringerLink

3) _It's about #TestScores_

Not always discussed explicitly in relation to all quality indicators and most frequently in the relation to "#validity", the process of validation produces evidence for the validity of #TestScores (as opposed to the validity of the test!).

The "Standards" make that very clear for the term "Validity" [2014, p. 11]:
"...the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests"

#Psychometrics