Nicola RomanΓ²In the past couple of years I have been in a lot of meetings centered about the topic of "OMG students are using genAI in assessments what do we doooo?"
After marking a lot of assessments of different type from different courses in different years of study and two different undergraduate programmes, here are my conclusions, some of which I have no way of proving, I know, but that's fine.
1. A lot of students use genAI. In many cases I cannot prove that, it's just a feeling (nobody writes like this, especially not Year 1 non-native speakers), but I had students telling me directly and I do believe them.
2. Looking at marks in the cases above where I have that feeling, I see a wide spread from fails to high A.
3. Following from 2, overall marks have not changed significantly and systematically in any of our courses from the past 5 years. There are of course year-to-year fluctuations, but that's cohort-dependent there seems to be no overall trend.
4. The conclusion you might draw from 2 is that we're rewarding students who are good at using genAI. Possibly, and I have not made up my mind up about this. My answer is that we should change our way of assessing and teaching, rather than trying to "catch" students using genAI. We're currently redesigning our Programmes and that's what we're trying to do. I am in the process of designing a new course and I have tried to do that assuming students will use genAI, but making it so that using it will not be an advantage, and might actually make it more cumbersome to do the assessments.
5. We looked at final year dissertations and plotted marks against %AI writing detected by our submission system. Taking this with a pinch of salt, given that AI detectors are biased and unreliable, there is only a small negative correlation but it's such a small effect size as to be essentially negligible. Again, the distribution of marks is the same as in previous years.
So in conclusion, just like anything else AI related, there's a lot of hype on how this is disruptive and it's terrible or game-changing depending on which side you're on. And yet, in practice...
Has anyone had similar observations? I'd love to hear your thoughts.
AstroMancer5G (she/her)
SETI Institute
Benjamin Carr, Ph.D. π¨π»βπ»π§¬
N-gated Hacker News
Carolina
