AkaSci 🛰️

How AI Impacts Skill Formation:

"We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI.

We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library."
🤔
https://arxiv.org/abs/2601.20245
1/n

10:41amWeb
Show threadAkaSci 🛰️13h ago

Related article - "Is AI ruining our skills? Early results are in — and they’re not good"

"Evidence suggests that AI-driven ‘deskilling’ is starting to happen in medicine, computer science and other fields. Researchers are now discussing how to preserve important human expertise in the age of AI."

https://www.nature.com/articles/d41586-026-01947-1

Is AI ruining our skills? Early results are in — and they’re not good

Reliance on artificial-intelligence tools degrades the abilities of physicians and software engineers, studies show.

Show threadAkaSci 🛰️13h ago

"Participants were divided into 3 groups: LLM, Search Engine, and Brain-only (no tools).

EEG revealed significant differences in brain connectivity: Brain-only participants exhibited the strongest, most distributed networks; Search Engine users showed moderate engagement; and LLM users displayed the weakest connectivity.

The LLM group's participants performed worse than their counterparts in the Brain-only group at all levels: neural, linguistic, scoring."

https://arxiv.org/abs/2506.08872
3/n

Show thread⁥jaseg12h ago
@AkaSci I had a look at this study when it came out last year, and it looked very shaky scientifically. I can’t find the thread anymore, but I remember finding that this study doesn’t support any of the conclusions people usually draw from it.
Show thread⁥jaseg12h ago
@AkaSci IIRC the most basic problem is one of people mid-interpreting differences in EEG activity as a good or a bad thing. EEG is a really coarse tool and doesn’t allow for conclusions like that. The premise that EEG allows any measurement of “brain connectivity” is also shaky at best and bullshit at worst, depending on how far you’re willing to stretch that “connectivity” term.
Show thread⁥jaseg12h ago
@AkaSci IIRC (I might be wrong, it has been a while, please read for yourself) they just did a basic correlation analysis between electrode signals which doesn’t really tell you anything of value.
Show thread⁥jaseg12h ago
@AkaSci Like on other social media, over here on masto, we need to be careful not to hype, misinterpret, or overstretch scientific results just because they happen to align with our biases or worldview.
Show thread⁥jaseg11h ago
@AkaSci One more note on that article: The ArXiv version is a preprint that hasn’t passed peer review, and AFAICT today, a year after its initial publication the article still hasn’t been accepted in peer review and properly published anywhere. This suggests to me that they are having trouble convincing peer reviewers of the merits of that article.
Show threadAkaSci 🛰️11h ago

@jaseg
Yes, we should be cautious when drawing strong conclusions from new research in a fast moving field.

However, this is not an isolated study on the issue. There are others, some mentioned in this thread, that are studying these issues.

Also, this paper does not rely solely on EEG data; there are other more traditional metrics that were measured in the study as well.

Nothing wrong with reporting on studies. There are others here with deeper expertise than the provide more insights.

Show threadMiro Collas5h ago
@AkaSci @jaseg This vid mentions the paper, and the problems with it. It also mentions other studies:
https://www.youtube.com/watch?v=52FiVExXfnU
Big Tech Is Speedrunning Human Stupidity

YouTube
Show threadJules 🍺14h ago
@AkaSci
I was going to post this in my hotdesk space Slak group, but I don't think the evangelists are listening and I definately do not want to hear their justifications for grand theft and environmental destruction
Show threadFennix13h ago

@AkaSci

I mean we kinda all know (as in, it's so common I think most / all cultures have a specific saying about it) that if you don't regularly use a skill it starts to erode very quickly.

For people jumping fully into using LLMs to code this must be like quitting cold turkey.

Show threadozeng11h ago

@fennix @AkaSci I think the key ongoing decision is what to gain/maintain skills in (ie what do you spend your time doing).

About 30 years ago, embedded C compilers weren’t perfect. The old school ASM crowd decried and derided C. But time marched on, compilers got better, and I needed to pore through list files less and less. The higher level language meant I could do *way* more complex things without having to shuffle registers all day. Who needs to know ASM now? Some people, absolutely.. but not many. Unless you’re in a niche role, ASM skills are now largely useless.

That’s not a direct analogy with AI by any means, I enjoy coding so repeatedly asking a machine to make something for me until it succeeds is not satisfying. But for many people it may end up being a trade they will willingly make, and their role will morph away from coding and into something else. Some people will always need to know how to write software themselves.

Show thread0x4d6165 (Julie)10h ago
@AkaSci why is this funded by anthropic?

See the asterisk on the first and second author on the preprint paper