Nick Byrd, Ph.D.

I need help computing an #interraterReliability score for a dataset of ratings that had more than one response format and that has some missing data.

Here's a reproducible toy example with more info: "How to clean redundancies and missings in rater dataset and then compute reliability (e.g., Cohen's kappa) using R?"

https://stackoverflow.com/questions/73912754/how-to-clean-redundancies-and-missings-in-rater-dataset-and-then-compute-reliabi

#rStats #dataAnalysis #dataScience #R #Rstudio #stats

How to clean redundancies and missings in rater dataset and then compute reliability (e.g., Cohen's kappa) using R?

I've nearly 10,000 rows of numeric and text ratings about various items from up to 5 raters. I need to 1. Clean the data (particularly redundancies and empty ratings) 2. Compute inter-rater reliabi...

Stack Overflow
Nov 12, 2022 at 4:51pmWeb
Show threadNick Byrd, Ph.D.Nov 16, 2022

The good news:
- Mike Zephyr helped clarify my goal in their office hour tonight! (Find Mike's "free office hours" at https://instats.com/seminars_courses)
- Apparently I need interrater *agreement* (consensus) scores rather than interrater reliability (consistent) scores—two things that were often conflated until [checks notes] THE 70s!!! (https://psycnet.apa.org/doiLanding?doi=10.1037%2F0021-9010.77.2.161, https://psycnet.apa.org/record/2000-16936-008)

The bad news: I still need help computing my raters' CONSENSUS/AGREEMENT.

#stats #rStats #reliability #agreement

Statistics and Data-Science Courses and Seminars | Instats

Learn advanced statistical methods like Longitudinal Data Analysis and SEM using Mplus, Rstat, Stata and more with Instats online courses and seminars.

Instats
Show threadLouis ChartrandNov 16, 2022
@ByrdNick Hi Nick, I'd love to help. I think I could learn a thing or two on interrater metrics, but processing data and computing scores is my bread and butter.
Show threadNick Byrd, Ph.D.Nov 16, 2022
@locha Wow! That’d be great! I’ve sent a message to follow-up.