Steve HarozApr 11, 2022
My review for the #CHI2022 paper: "Aperitif: Scaffolding Preregistrations to Automatically Generate Analysis Code and Methods Descriptions". The survey of preregistration in HCI is great. But it also advertises a very questionable analysis tool. https://osf.io/kc9bv/ 1/

OSF
Show threadSteve HarozApr 11, 2022
I strongly discourage anyone from using the Aperitif preregistration and analysis tool. My review describes serious concerns regarding the the application and the paper that summarizes it that extend beyond simple issues that can be added to a limitations section. 2/
Show threadSteve HarozApr 11, 2022
The paper lacks critical info on the automated analysis, overstates features, and could inhibit users from running appropriate and complete analyses. Example: it can't preregister interaction effects or ANOVAs. @acm_chi decided to accept the submission despite these concerns. /3
Show threadSteve Haroz
I encourage you to read section 3, which was my final summary of my concerns. 4/4 https://osf.io/kc9bv/
OSF

Apr 11, 2022 at 1:29pmWeb
Show threadSteve HarozApr 11, 2022
This whole situation should call into question the revision process at #CHI2022. If actual concerns aren't discussed in the second round, let's not pretend it matters.