Cat Hicksoh my fking god. I am TRYING to do research for my book right now but I am getting DISTRACTED by the fact that I followed a citation from a paper into another paper and learned that they're interpreting EEG and eyetracking as a way to sort people into "in the flow" or "not in the flow" I AM GOING TO EFFING LOSE IT
I love that this was published in the same overall goddamn umbrella organization that sent me reviewers that said my work had "too much psychology" cited in it I AM. GOING. TO. LOSE. IT.
eyeblinks as a measure of frustration? I'm going to throw my entire laptop into the ocean
this is UNCRITICALLY cited in a BIG paper with NO reference to it being a biometrics study AT ALL
I mean. Can we measure things with EEG and with eyetracking, yes we can. Can we detect different kinds of waves yes we can. Do blinks correlate with something, I don't know, probably. This is all just so effing strange.
how is engineering the most critical field with the LEAST critical research I've ever seen. Do you guys just think if something is published in Fuck All Journal, 2024, that makes it good?
ML CLASSIFIER ON SIX DATA POINTS WHAT IS HAPPENING
Movement dominates brain activity, and is ALSO a huge artifact in any skin conductance. So like THAT'S a confound in these conditions. Yet no details at all about how this is truly analyzed, because it's an off the shelf EEG "band" someone bought with some research cash I bet. This paper is "we thought this was cool and so we bought a lot of stuff and tried it."
What's the play guys. What's the thing. Are we going to believe developers are human beings if we count enough eyeblinks and say they're systematically different if you focus harder? Are we going to unlock the magical super Computer Brain region if we measure hard enough?
MY WIFE PUBLISHED A PAPER ABOUT HOW MOVEMENT DOMINATES BRAIN ACTIVITY AND NEEDS TO BE ACCOUNTED FOR IN MEASURES AND NONE OF THESE GUYS READ IT
one of the measures is defined as capturing "various emotional states" I am dying
somebody made like twelve tables for this paper. Just like, table after table of everything they could tally up. NOT ONE OF THESE THINGS SHOWS KEY INFORMATION ABOUT THE EEG. This is like the baseball statistics version of analysis. Just tallying up as many numbers as possible. "mean task completion" across like six people which tells us fuck all about fuck all stfu
look, it's not that I wish anything but healing upon these people who wrote this paper and don't know better. it's just that you're so underserved. You're so underserved and you don't even know it. Somebody is serving you up a plate of raw hamburger paired with rat shit in the back alley next to the dumpster and telling you it's fine dining. You (software developers) BUILD THE WORLD. YOU COULD GET A NICE MEAL ONCE IN A WHILE.
You're stuck. We're all stuck. We're stuck in the back alley trying to say please god believe I have emotions look at my eyeballs WHO PUT US HERE
"we deleted all measurements that were marked as invalid by the eye-tracking device"
ahahhahahahahahhahhahahahhahahahahahhaaaaaa
they extracted??????? eyeblink "data"????? FROM THE EEG??????? what
are we treating the artifact AS the measure now
oh, I see, the sensor (purchased from some random website) has "two pre-processed signals" omfffffg YOU GUYS. HOW IT IS PROCESSED IS THE POINT
we're going to say that brain waves are predictive but not say WHICH brain waves we're talking about?
"finding relevant code" vs "not finding relevant code" are two sides of the classification how much do you want to bet this is essentially categorizing scrolling and skimming vs "not scrolling and reading" which is, again, A MOTOR MOVEMENT among many other things
The obsession with the idea that we are detecting "higher order problem-solving" via Secret Signals in the Skin
I forced my neuroscientist wife to look at this paper from a signal processing POV because she's actually published on the complex math of signal processing (and because I am not above outsourcing my complaining) and in her characteristically gentle spirit she said "many things could be extremely misleading, there's no way to know with what is being reported" I am going to translate that for you: GARBAGE IN GARBAGE OUT
Do you all want to know something super funny, the lab I did my PhD in actually did EEG work as well and THIS is why I NEVER talk about it, early in my tech career someone was like "omg you should put on your resume that you can do EEG and eye tracking and your entire career in UX will be set!" And I said no thank you, I will never! And ever since then people have mistakenly said I was "Qualitative" and maybe "not a scientist" but I sleep at night, so
Baffled by the idea that we should divide brain states into two categories that seem to mean "thinking about stuff" and "chilling" and some of these activities under the "chill" category do not sound chill at all unless we're talking about taking a nap
Stress is mentioned as a possible threat to validity when I think it's probably actually the entire ball game if you're forcing people to solve a coding task in a lab setting
Stress+ motor movements, that's what I see
There is something so bleakly funny to me about the fact that they list out the Greek letters for brain waves and then DO NOT show you this data with any specificity because they did not analyze it, their "off the shelf device" gave them two categories which they then tried to map onto coding "states" (I am guessing).
I followed a different citation to a completely different paper which is doing roughly the same thing but with heart rate data
I would like someone to look me in the eyes and explain to me, in physiological detail, the theoretical model under which we can make a causal claim about SPECIFICALLY Flow State SPECIFICALLY because of how the heart is beating
ok, so we have a manipulation that's supposed to create "Flow state" (it's always goddamn flow state) and the manipulation failed the validity checks, so instead of thinking the manipulation maybe doesn't reliably create "Flow state", we're going to exclude the sessions that failed the validity check, mmmmhmmmmmmmmmmmm
"a flow detection model using end-to-end deep learning" I Want. To. Walk. Into. The. Ocean.
adding one of the physiological measures makes one of the classifications worse, and another one better! Instead of concluding "perhaps these physiological measures are WILDLY BIZARRE TO USE FOR THIS" we conclude there is something magic about the second task idk
oh, if you're wondering if there is STRESS + MOTOR MOVEMENT involved in the tasks of this experiment again you would be correct
Martijn Faassen has moved@grimalkina
Why do you think there is such interest in this state in the first place?
Why do you think there is such interest in this state in the first place?
Mark Dennehy@grimalkina I keep seeing "deep learning" and remembering that kalman filters do better on very limited resources than most of the ML/AI/LLM nonsense does with near-infinite resources *so long as you have a model for the thing they're predicting*. If the human doesn't grok the system, they can't make a kalman filter work well; the AI/ML/etc selling point is "you don't have to understand the thing you're trying to predict" and that's SO MUCH nonsense yet we spend so much on them.
arosien
John Schrag@grimalkina This brings back a memory. My best friend back in the early 1980s was (as an undergrad) doing a study relating the “T” component of EKG signals (whatever that means) to people’s perceived sense of whether they were bored, engaged, or overwhelmed with tasks in front of them. I think it was a military contract. Sounded like his prof wanted to build something dystopian.
Deborah Apthorp@grimalkina OMG quality rant thank you!
Jazz Rabbit 🐇🏳️⚧️@grimalkina oh no oh no are they using the ThinkGear TGAM chip? It provides two preprocessed values for “attention” and “meditation” from the EEG wave data. These are good enough to make a toy that responds to vague brain activity. But no way no how is it possible to do what they’re saying.
@madrabbit this has a different product name in the paper but yes 😭😭 "attention" and "meditation" lmfao what does it MEAN
@grimalkina I used that chip to make a silly toy, and as far as I could tell, attention is a 1-second average of some block of frequencies, and meditation is a similar average of others.
They weren’t diametrically opposed. I recall that reducing my physical activity and remaining still increased M, and it could be at the same time as a high level of A. Who knows what the signal processing was. But I bet it wasn’t sophisticated.
Obscuredavid@grimalkina
As far as I'm concerned six data points is roughly zero data points.
As far as I'm concerned six data points is roughly zero data points.
@Cameleopard depends on the question at hand (for example "CAN cells in a human brain do this" can be answered by looking at a single brain, but not "how often do cells in human brains in this population do this")
@grimalkina
Fair enough.
Fair enough.
DeterioratedStucco@grimalkina
I am in a bizarre horrified-giggles state.
My son just came down the stairs to check if I was OK...
I am in a bizarre horrified-giggles state.
My son just came down the stairs to check if I was OK...
Glyph@grimalkina I am sitting in a black belt board test for my kids martial arts class and reading this during a break and desperately trying not to loudly lose it. This is “we did an fMRI on a cheeseburger to see if it has the buddha nature” levels of gobbledygook. How did someone write all this down without realizing what they were doing??
0xC0DEC0DE07EA@glyph @grimalkina “Oh, god, I’ve been with this advisor for [however long] and we have nothing, but I’ve been paying for ‘education’ and I need to publish a thesis to graduate, so maybe if I just put a ton of graphs in, I can escape with a Master’s and go DO something and not feel like a complete failure?”
Dan Sugalski@grimalkina this looks uncomfortably like the time I went looking for foundational economics papers. (Would you like to see a list of foundational papers on economics with clear theses, properly collected data, and clean statistical analyses that support the paper’s conclusions? Yeah, well, so would I, that’d be a nice change)
Jacob Something@grimalkina Which journal?
Eaton@grimalkina oh, you know… waves…
emmatonkin@grimalkina
I get the impression it's a NeuroSky? If so that explains the preprocessed signals and the reporting of eye blinks. The MindWave gives, or at any rate used to give last time I played with their original model in like 2014, "attention and meditation" waves and eye blinks. I think the blinks are just freebies because it has one sensor and that happens to be on the forehead.
I get the impression it's a NeuroSky? If so that explains the preprocessed signals and the reporting of eye blinks. The MindWave gives, or at any rate used to give last time I played with their original model in like 2014, "attention and meditation" waves and eye blinks. I think the blinks are just freebies because it has one sensor and that happens to be on the forehead.
@grimalkina (if it is one of those, back in the day we concluded that it would need serious validation to become relevant. If it isn't, worrying that there are multiple devices in this class.)
Samael 
I once had a coworker (we were both software dev) who refused to discuss unions with me. I don't recall if I could coax a reason out of him, but he also complained about feeling overworked and, I just... couldn't. FUCK!!!
Why are humans? (that's it, that's the question)
@Kishi @grimalkina well now you know your mistake. You should have blinked a lot more, with your brainwaves. You’re welcome
Liz Marley@grimalkina Thank you for trying to help us. I am a software developer, not any kind of doctor or research specialist, and you put the problems with these papers in terms I can understand.
grillchen@grimalkina 6 people? wtf
@grillchen we can learn interesting things about small groups of people sometimes but Not Like This
@grimalkina yeah but it sounds like they confused qualitative with quantitative work.
Rich Puchalsky ⩜⃝It's not a real paper unless they ran correlations between all of the variables in those twelve tables and their research result was that 2 of the variables were strongly correlated.
Morten@grimalkina Does this paper have any comments on PubPeer?
The Witch of Crow Briar@grimalkina 🤦🏼♀️🤦🏼♀️🤦🏼♀️🤦🏼♀️
robryk
Single-trial neural dynamics are dominated by richly varied movements - Nature Neuroscience
The authors use a linear model to reveal how neural activity patterns are related to cognition or movements. They find that uninstructed movements dominate single-cell and population activity throughout the brain, outpacing task-related activity.
@robryk yes that's one!
Wharrrrrrgarbl@grimalkina obviously the solution is to implant depth electrodes until the machine learning model we tape to the headbox tells us that it has decoded The 10x Engineer Coding Flow State
i can taste the citations already
they taste like electrodes jammed into the anterior insula
Stephan Eggermont@grimalkina Don't tease me without the link, please :)
robryk (@[email protected])
@[email protected] Do you mean https://www.nature.com/articles/s41593-019-0502-4?
Joseph Kohlmann@grimalkina Makes you wonder if they’ve tried, uh, measuring skulls or some shit 🫠🤬😭
Maya@grimalkina That was what my research advisor did back in the 2000s.
I was super-pissed off because I couldn't replicate the prior grad student's EEG classifier work at all, even with the source code, and I was under so much stress because it felt like I was being pressured into doing academic fraud if I wanted to complete my thesis.
It was a major contributor to why I ended up ABT.
@Mayabotics oh this breaks my heart to hear this. I know exactly what you mean and I've felt these kind of pressures. This is yet another reason it's so important to have a compassionately rigorous science
@grimalkina And I was extremely uncomfortable by the fact that my research advisor wanted me to collect EEG data without human studies review.
Ted Mielczarek@grimalkina lmao that is absurd. They wrote a whole paper with n=6???
@tedmielczarek actually not the worst thing about this paper, there is a lot you can learn from a small group from bio measures and a lot of measures within individuals, but in this particular case, really really not right for this particular question