#IRTheory #stats #rstats #scaledevelopment
Continuing the thought from yesterday, after some further thinking.
It makes no sense to divide correlations. It's weird conceptually and also the numbers you get are hard to interpret. Rather it seems most straightforward to use reliability-attenuated correlations and apply a SESOI:
Cor_att = cor*sqrt(rel_a*rel_b)
Apply (crud-corrected) SESOI (Cohen's estimates _might_ work) or whatever is defensible based on the plausible parameter space.
#rstats #stats #IRT #scaledevelopment
Thinking about scale reliability lately.
An obvious thing to remember is that scale reliability (omega) is the upper bound of correlations it can meaningfully (!) have with anything else.
This is highly practical, because, in theory (I haven’t fleshed this out yet), this enables us to test construct validity pretty well by attenuating correlations with reliability. This has been done before, albeit (imho) slightly clumsily by Kristof (1983). They used a Spearman Brown approach based on an ideal correlation to establish thresholds a correlation must test against to conclude discriminant or convergent validity. The formula was threshold = rho_ideal * sqrt(rel_a * rel_b). This is less than ideal, because what an ideal correlation is is up to the researcher.
However, keeping the above fact in mind, we can do better. Kill thresholds and just make it a measure of relatedness altogether that is capped by the lower reliability.
Let’s call this Kappa = rho_real/sqrt(rel_a*rel_b).
This should (again, I haven’t tested this yet) give you two types of information right away:
1)If its >1, then your scale is off.
2)If it’s less than 1 it informs about construct validity. If it’s less than 0.5, probably discriminant, if it’s higher probably convergent. The extreme the score, the higher the confidence.
Please correct me If I'm doing some goofing here.
Publication proposing a 'Comprehensibility Continuum' method to demonstrate #PatientReported outcome measure comprehensibility systematically and consistently from interview data:
https://link.springer.com/article/10.1007/s11136-024-03858-y
Purpose Evidence of comprehensibility is frequently required during the development of patient reported outcome measures (PROMs); the respondent’s interpretation of PROM items needs to align with intended meanings. Cognitive interviews are recommended for investigating PROM comprehensibility, yet guidance for analysis is lacking. Consequently, the quality and trustworthiness of cognitive interview data and analysis is threatened, as there is no clear procedure detailing how analysts can systematically, and consistently, identify evidence that respondent interpretations align/misalign with intended meanings. Methods This paper presents a novel, structured approach to comprehensibility analysis - the ‘Comprehensibility Continuum’ – that builds upon existing cognitive interview guidance. Results The Comprehensibility Continuum comprises a structured rating scale to code depth of alignment between intended item meaning and respondent interpretation and consists of five main stages: before cognitive interviews are conducted, researchers must (1) Define intended meanings of PROM items; and (2) Determine comprehensibility thresholds for both participant- and item-level. After conducting interviews, they (3) Prepare data by transcribing interviews ‘intelligent’ verbatim; (4) Code transcripts using the Comprehensibility Continuum scale in iterative sets, assigning an overall code for each item at participant-level; and (5) Compare participant-level codes across all participants to determine overall item comprehensibility, such that decisions can be made to retain, modify, or remove items. Conclusion Quality in qualitative data analysis is achieved through rigorous methods that are clearly described and justified. Given insufficiency in guidelines, cognitive interviewers must reflect on how best to demonstrate PROM comprehensibility systematically and consistently from interview data, and the Comprehensibility Continuum method offers a potential solution.
Oliver D. Reithmaier
Jan R. Boehnke
Dr. phil. Stefan T. Siegel