Cat Hicksoh my fking god. I am TRYING to do research for my book right now but I am getting DISTRACTED by the fact that I followed a citation from a paper into another paper and learned that they're interpreting EEG and eyetracking as a way to sort people into "in the flow" or "not in the flow" I AM GOING TO EFFING LOSE IT
I love that this was published in the same overall goddamn umbrella organization that sent me reviewers that said my work had "too much psychology" cited in it I AM. GOING. TO. LOSE. IT.
eyeblinks as a measure of frustration? I'm going to throw my entire laptop into the ocean
this is UNCRITICALLY cited in a BIG paper with NO reference to it being a biometrics study AT ALL
I mean. Can we measure things with EEG and with eyetracking, yes we can. Can we detect different kinds of waves yes we can. Do blinks correlate with something, I don't know, probably. This is all just so effing strange.
how is engineering the most critical field with the LEAST critical research I've ever seen. Do you guys just think if something is published in Fuck All Journal, 2024, that makes it good?
ML CLASSIFIER ON SIX DATA POINTS WHAT IS HAPPENING
Movement dominates brain activity, and is ALSO a huge artifact in any skin conductance. So like THAT'S a confound in these conditions. Yet no details at all about how this is truly analyzed, because it's an off the shelf EEG "band" someone bought with some research cash I bet. This paper is "we thought this was cool and so we bought a lot of stuff and tried it."
What's the play guys. What's the thing. Are we going to believe developers are human beings if we count enough eyeblinks and say they're systematically different if you focus harder? Are we going to unlock the magical super Computer Brain region if we measure hard enough?
MY WIFE PUBLISHED A PAPER ABOUT HOW MOVEMENT DOMINATES BRAIN ACTIVITY AND NEEDS TO BE ACCOUNTED FOR IN MEASURES AND NONE OF THESE GUYS READ IT
one of the measures is defined as capturing "various emotional states" I am dying
somebody made like twelve tables for this paper. Just like, table after table of everything they could tally up. NOT ONE OF THESE THINGS SHOWS KEY INFORMATION ABOUT THE EEG. This is like the baseball statistics version of analysis. Just tallying up as many numbers as possible. "mean task completion" across like six people which tells us fuck all about fuck all stfu
look, it's not that I wish anything but healing upon these people who wrote this paper and don't know better. it's just that you're so underserved. You're so underserved and you don't even know it. Somebody is serving you up a plate of raw hamburger paired with rat shit in the back alley next to the dumpster and telling you it's fine dining. You (software developers) BUILD THE WORLD. YOU COULD GET A NICE MEAL ONCE IN A WHILE.
You're stuck. We're all stuck. We're stuck in the back alley trying to say please god believe I have emotions look at my eyeballs WHO PUT US HERE
"we deleted all measurements that were marked as invalid by the eye-tracking device"
ahahhahahahahahhahhahahahhahahahahahhaaaaaa
they extracted??????? eyeblink "data"????? FROM THE EEG??????? what
are we treating the artifact AS the measure now
oh, I see, the sensor (purchased from some random website) has "two pre-processed signals" omfffffg YOU GUYS. HOW IT IS PROCESSED IS THE POINT
we're going to say that brain waves are predictive but not say WHICH brain waves we're talking about?
"finding relevant code" vs "not finding relevant code" are two sides of the classification how much do you want to bet this is essentially categorizing scrolling and skimming vs "not scrolling and reading" which is, again, A MOTOR MOVEMENT among many other things
The obsession with the idea that we are detecting "higher order problem-solving" via Secret Signals in the Skin
I forced my neuroscientist wife to look at this paper from a signal processing POV because she's actually published on the complex math of signal processing (and because I am not above outsourcing my complaining) and in her characteristically gentle spirit she said "many things could be extremely misleading, there's no way to know with what is being reported" I am going to translate that for you: GARBAGE IN GARBAGE OUT
Do you all want to know something super funny, the lab I did my PhD in actually did EEG work as well and THIS is why I NEVER talk about it, early in my tech career someone was like "omg you should put on your resume that you can do EEG and eye tracking and your entire career in UX will be set!" And I said no thank you, I will never! And ever since then people have mistakenly said I was "Qualitative" and maybe "not a scientist" but I sleep at night, so
Baffled by the idea that we should divide brain states into two categories that seem to mean "thinking about stuff" and "chilling" and some of these activities under the "chill" category do not sound chill at all unless we're talking about taking a nap
Stress is mentioned as a possible threat to validity when I think it's probably actually the entire ball game if you're forcing people to solve a coding task in a lab setting
Stress+ motor movements, that's what I see
There is something so bleakly funny to me about the fact that they list out the Greek letters for brain waves and then DO NOT show you this data with any specificity because they did not analyze it, their "off the shelf device" gave them two categories which they then tried to map onto coding "states" (I am guessing).
I followed a different citation to a completely different paper which is doing roughly the same thing but with heart rate data
I would like someone to look me in the eyes and explain to me, in physiological detail, the theoretical model under which we can make a causal claim about SPECIFICALLY Flow State SPECIFICALLY because of how the heart is beating
ok, so we have a manipulation that's supposed to create "Flow state" (it's always goddamn flow state) and the manipulation failed the validity checks, so instead of thinking the manipulation maybe doesn't reliably create "Flow state", we're going to exclude the sessions that failed the validity check, mmmmhmmmmmmmmmmmm
"a flow detection model using end-to-end deep learning" I Want. To. Walk. Into. The. Ocean.
adding one of the physiological measures makes one of the classifications worse, and another one better! Instead of concluding "perhaps these physiological measures are WILDLY BIZARRE TO USE FOR THIS" we conclude there is something magic about the second task idk
oh, if you're wondering if there is STRESS + MOTOR MOVEMENT involved in the tasks of this experiment again you would be correct
This paper is *also* being uncritically cited as a proof of the existence of special-brain-states-during-coding by a venue that should know better. I feel like the only human alive who reads something like "physiological measures have shown...." and goes huh, HAVE THEY?
Building a classifier that sorts cardiac activity is not hard. That is because OF THE WAY THAT HEARTS WORK. That doesn't make something a "flow classifier" holy shit this is an entire area
"This could be the basis for an intelligent, automatic controlling of office tasks and workloads"
oh, ok I see
no wonder they had to come up with their own little ML venues, psych and neuro would eat this shit ALIVE people would be meme-ing your paper within thirty seconds
so sad and baffling to me when the lit reviews of software research on developers decides to pick these citations as its ground truth of evidence, not, you know, anyone who studies human beings or has ever experienced the human body for a reflective moment
amazing to me how often people come up with a rote task design (like playing a repetitive game) to get people to zone out, and then measure a bunch of stuff and give it the ol razzle dazzle of analysis (I think some of this math is just fine, it's a perfectly fine way to sort things! It's just that the math doesn't solve the problem here! At all!), and THEN say, "we now take this to be a conclusion about what happens when people face wildly novel and unexpected tasks" I MEAN
look me in the eyes and hold my hand and explain to me why these two different things are actually the same thing and also distinguishable by your heart, and that we didn't just come up with this because measuring heart stuff was super easy and came with a fun wrist monitor and now we're justifying groupings post hoc
Something something something physiology astrology
because I hate myself I'm reading one of the papers THIS paper cites and they are talking about how prefrontal activity "may be necessary" during the performance of skills I mean sometimes you just have to be like, "you're not wrong, but" let's just measure dead people vs not dead people and be done with it
what really gets me is, even in these extremely poorly conceived and baffling studies, most of them have mixed results, and then they are cited as NOT having mixed results but instead providing PROOF of something in the next paper up the chain, leading me to once again conclude I am alone on an island, reading original claims as if there is meaningful information in them, as if the scientific record is an actual thing that we can actually engage in, so silly
Theory is an accident, a trifle, a bonus hobby, to some of this work. Psychological theory in particular is nothing but a shiny curiosity that you ornament your actual study with. Your actual study has no interest in engaging with the discipline, cites controversial and niche claims like they're established and well tested, and you get away with it because nobody cares, I think, to even imagine that the measurement of human beings could be a respected science in its own right
Sparrows@grimalkina I'm watching this thread go down, with the beleaguered expression of someone holding a bucket at a housefire, wanting to do their part but utterly outclassed by the enormity of the disaster and the skill of the professionals on site already.
@sparrows I know we're total strangers but can't express what a big wave of fondness for you this image made me feel. Wherever I have a firetruck I'll be there
@grimalkina Eh, it was the image that popped into my head, and I hoped it would make you smile. Keep on keeping on.
Simon@grimalkina there should be a @hertzpodcast about this
Bynkii@grimalkina this was a wild ride, and I’m glad for it. I’ve had so many people lecture my ADHD ass about flow and I have no fucking idea what they mean because what they call flow is what I call hyperfocus and it’s kind of fucking annoying. Like if it’s on enough, I miss everything. Computer reminders? Nope.
The amount of times I am about *running* to the can because it took actual bladder/kidney pain to snap me out of it is not small.
@bynkii as you well know the attention system is VERY COMPLICATED and I am REALLY TIRED of people acting like we should be Magical Focus Robots Grinding at Code
@grimalkina to me it reeks of 19th century factory owner bullshit.
@bynkii yes it is really really really odd that it has this human-centered veneer over it but ultimately the entire model is "let's optimize all people getting into this state at all times" like the modeling of it is very economic
@grimalkina yep. Then add in how this focus creates people who have depth in a single area but no breadth. So they can code like a MFer but can’t communicate their ideas well, or approach a problems in any way other than “code will fix it”, because that’s the only thing they know.
When I tell people the most useful classes I had in college, career-wise were the “non-technical” ones like writing, comms, public speaking, psych/socio/anthro, phil, they look at me like I grew a third eye.
Honestly, I think taking 39 years to get my degree, and working in so many places even outside of IT is a huge part of my skill set.
Fast food, a newsstand, Mexican place, pizza delivery, flightline avionics, IT in finance, higher ed, advertising, weather science, manf., the breadth, the requirements to use radically different tools, different environments, different audiences, all of it taught me so many useful things & showed me so many things I’d never learn in a server room.
Because of the ADHD, I actually find interruptions somewhat helpful because the “squirrel” part of my brain needs those. It’s not like I’m not working on elebenty things anyway. (This is why I can’t go to casinos. Ever try to track all the light patterns in one? I have, not fun.)
a wandering happenstance@grimalkina I mean, if you ask Denny Borsboom this is true of most studies done by academic psychologists, too (I’m thinking of “Attack of the Psychometricians”, DOI 10.1007/s11336-006-1447-6
emmatonkin@grimalkina
One of my hobbies is checking confident assertions during peer review by looking at the papers cited, then getting unreasonably annoyed when the author's response makes it clear that they never read the paper in the first place. Whole areas of research in comp sci are based on "i reckon three other people claimed this was totally a thing".
One of my hobbies is checking confident assertions during peer review by looking at the papers cited, then getting unreasonably annoyed when the author's response makes it clear that they never read the paper in the first place. Whole areas of research in comp sci are based on "i reckon three other people claimed this was totally a thing".
Martijn Faassen has moved@grimalkina
I think this is unfortunately common in many fields.
I have limited experience but in biblical criticism someone can say that, for instance, Harnack has conclusively shown that the gospel of Marcion is a cut down version of Luke, not the other way around, and then that stands for many decades as the consensus.
And when you dig into it the debate was a lot more complicated and less conclusive. When solid foundations are difficult to establish, people seem to do it anyway.
Colin Gordon@grimalkina I feel this in my bones... This seems to be *A THING* with some SE + ML researchers.... A student and I wrote a paper a few years ago about why we thought using machine learning for a particular task was a poor choice, and half of the citations are lumping us in with a list of papers described as showing the efficacy of ML for the task we said it shouldn't be used for 🙃
@csgordon *endless scream* thank you for your service! Citation by keyword truly
Bill, organizer of stuff@csgordon @grimalkina Oh man, I remember this from 7 or 8 years ago. 🙄
Shannon Quinn 
@grimalkina Science: the world’s most consequential game of telephone
Sasha Akhavi@magsol @grimalkina exactly, the most sequential game of telenovela ever
Morten@grimalkina Yup. One of the reasons I don't want to be in academia. I will say that the fewer citations a paper has, the more reliable it is likely to be. Anything that comes from a medical doctor is complete trash - 500+ citations of things they've barely glanced at (probably copied wholesale from another paper). But journals love it because impact factor goes brrrrrrrr.
T. T. Perry@grimalkina so many examples of this!
From auditory science: https://ttpphd.com/2022/12/11/communicating-complex-research-auditory-training-and-age-related-deficits/
D. G. Fitch@grimalkina dead coders may be better at some coding tasks than alive coders, and I say this as an alive coder
(I'm a support engineer in a neuro lab and this thread gave me many laffs, so thank you for sharing your rage/pain)
Davidi found the paper you haven't named yet and i have to say, this is one of the reasons we shouldn't have conference proceedings.
David Wheeler@grimalkina I have enjoyed this evisceration of poor research. Thank you.
Lorin Hochstein 
@grimalkina Revenge of Ginger2! https://ieeexplore.ieee.org/document/799942
Kathy Reid
Glyph@grimalkina I feel like if you get to a mention of “dopamine” here you’ll win a prize
@glyph there is WAY more language about machine learning than about human biology in this particular paper!
@grimalkina the sense in which I am imagining "dopamine" being invoked isn't *really* about biology, though. it's a vibe. maybe gradient descent optimizations can catch vibes too!
@grimalkina
Why do you think there is such interest in this state in the first place?
Why do you think there is such interest in this state in the first place?
@faassen I genuinely do not know. I feel like it is a self-perpetuating thing because everyone just cites each other on it and not reading these original papers really
Mark Dennehy@grimalkina I keep seeing "deep learning" and remembering that kalman filters do better on very limited resources than most of the ML/AI/LLM nonsense does with near-infinite resources *so long as you have a model for the thing they're predicting*. If the human doesn't grok the system, they can't make a kalman filter work well; the AI/ML/etc selling point is "you don't have to understand the thing you're trying to predict" and that's SO MUCH nonsense yet we spend so much on them.
arosien
John Schrag@grimalkina This brings back a memory. My best friend back in the early 1980s was (as an undergrad) doing a study relating the “T” component of EKG signals (whatever that means) to people’s perceived sense of whether they were bored, engaged, or overwhelmed with tasks in front of them. I think it was a military contract. Sounded like his prof wanted to build something dystopian.