Ev (not Eve) Fedorenko

@ev_fedorenko@mstdn.science
724 Followers
78 Following
10 Posts
I study language using tools from cognitive science and neuroscience. I also like snuggles.
Language
Neuroscience
Ev (not Eve) FedorenkoDec 9, 2022
NancyKanwisherDec 7, 2022
@NicoleCRust
@tyrell_turing @fedeadolfi @rachelfheaton @jehummel @Neurograce @ShahabBakht @DrYohanJohn @KathaDobs
6) I think most sensible people agree that standard deep nets are pretty good at pattern classification but not so good at what Josh Tenenbaum calls building models of the world. So as many have already realized we need to think about how to put these different styles of models together if we are to account for anything beyond the most basic perceptual processes.
7) ok back to work.
Ev (not Eve) FedorenkoDec 7, 2022
Anna IvanovaDec 6, 2022

My co-lead Carina Kauf and I present: an in-depth investigation of event knowledge in language models.

Using a controlled minimal pairs paradigm, we find that large language models (LLMs) know that “The teacher bought the laptop” is more likely than “The laptop bought the teacher” (an impossible event), but perform below humans on sentences like “The nanny tutored the boy” vs. “The boy tutored the nanny” (a possible but unlikely event).

A 🧵 1/

https://arxiv.org/abs/2212.01488

Event knowledge in large language models: the gap between the impossible and the unlikely

Word co-occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs' semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pre-trained LLMs (from 2018's BERT to 2023's MPT) assign higher likelihood to plausible descriptions of agent-patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n=1,215), we found that pre-trained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign higher likelihood to possible vs. impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely vs. unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.

arXiv.org
Ev (not Eve) FedorenkoDec 7, 2022
Show threadAnna IvanovaDec 6, 2022

This is an international collaboration brought together by @ev_fedorenko and Alessandro Lenci, with vital contributions from @grambelli and Emmanuele Chersoni (and our undergrads Selena She & Zawad Chowdhury).

It's been a crazy run, with zoom calls during the lockdowns of 2020 & coordinated meetings between Boston, Italy, Hong Kong, and sometimes Germany and Russia. Glad the project has finally come to fruition!

8/

Ev (not Eve) FedorenkoNov 29, 2022
Konrad KordingNov 29, 2022

#introduction I am a professor at Penn and also co-director of the CIFAR Learning in Machines and Brains program. I like to think about neuroscience, AI, and science in general. Neuromatch. Recently, much of my thinking is about Rigor in science and I just started leading a large NIH funded initiative community for rigor (C4R) that aims at teaching scientific rigor.

My interests are broad: Causality, ANNs, Logic of Neuroscience, Neurotech, Data analysis, AI, community, science of science

Ev (not Eve) FedorenkoNov 29, 2022
Marino PaganNov 29, 2022

🚨🚨🚨NEW PREPRINT🚨🚨🚨

A big mystery in brain research is what are the neural mechanisms that drive individual differences in higher order cognitive processes. Here we present a new theoretical and experimental framework, in collaboration with Vincent Tang, Mikio Aoi, Jonathan Pillow, Valerio Mante, @SussilloDavid and Carlos Brody.

https://www.biorxiv.org/content/10.1101/2022.11.28.518207v1

1/16

Ev (not Eve) FedorenkoNov 29, 2022
Mastodon MigrationNov 28, 2022

TIP OF THE DAY

A Pinned Thread on Threading and Pinning

While different, threading in Mastodon is actually pretty good. People are familiar with Twitter threads that link one post to the next in chronological order. You can do this on Mastodon too, but it’s a little less polished.

Read more of this thread on How To Thread by expanding this post…

1 of 8 #twittermigration #mastodonmigration #feditips #mastodontips #NewUser

Ev (not Eve) FedorenkoNov 29, 2022
Prof. Emily M. Bender(she/her)Nov 29, 2022

Had a wonderful conversation with @mmitchell_ai this morning planning how we'll use the time in this event. I'm super excited for this conversation! And also just a little intimidated now that @RuthStarkman has posted that there are 3200+ people registered (online) plus hundreds in person...

https://www.eventbrite.com/e/envisioning-paths-individual-collective-action-for-ethical-technology-tickets-466438639527

Envisioning Paths: Individual & Collective Action for Ethical Technology

Join Professor Emily M. Bender and Dr. Meg Mitchell for an event discussing ethical technology development. Open to the public.

Eventbrite
Ev (not Eve) FedorenkoNov 28, 2022
Tiago PimentelNov 28, 2022

The time we take to read a word depends on its predictability, i.e. its surprisal. However, we only know how surprising a word is after we see it. Our new paper investigates whether we anticipate words' surprisals to allocate reading times in advance :)

Joint work with Clara Meister, Ethan Wilcox, @roger_p_levy , @rdc
Paper: https://arxiv.org/abs/2211.14301
Code: https://github.com/rycolab/anticipation-on-reading-times

On the Effect of Anticipation on Reading Times

Over the past two decades, numerous studies have demonstrated how less predictable (i.e. higher surprisal) words take more time to read. In general, these previous studies implicitly assumed the reading process to be purely responsive: readers observe a new word and allocate time to read it as required. These results, however, are also compatible with a reading time that is anticipatory: readers could, e.g., allocate time to a future word based on their expectation about it. In this work, we examine the anticipatory nature of reading by looking at how people's predictions about upcoming material influence reading times. Specifically, we test anticipation by looking at the effects of surprisal and contextual entropy on four reading-time datasets: two self-paced and two eye-tracking. In three of four datasets tested, we find that the entropy predicts reading times as well as (or better than) the surprisal. We then hypothesise four cognitive mechanisms through which the contextual entropy could impact RTs -- three of which we design experiments to analyse. Overall, our results support a view of reading that is both anticipatory and responsive.

arXiv.org
Ev (not Eve) FedorenkoNov 28, 2022
Katharina DobsNov 28, 2022

New preprint out on "Using deep convolutional neural networks to test why human face recognition works the way it does" with Joanne Yuan, Julio Martinez, @NancyKanwisher:
https://www.biorxiv.org/content/10.1101/2022.11.23.517478v1

We find that classic signatures of human #face #perception emerge in #deep convolutional #neural #networks only if CNNs are optimized for face identification, not object categorization (even if face experience is matched), suggesting that these signatures result from optimization for face recognition.

Ev (not Eve) FedorenkoNov 28, 2022
Francisco PereiraNov 27, 2022

The Machine Learning Team at the National Institute of Mental Health (NIMH) is hiring a research scientist

https://nih-fmrif.github.io/ml/index.html

with an emphasis on deep learning methods for applications in psychology, neuroscience, and psychiatry. We work with brain imaging, text, speech, behaviour, and other data types, coming from our collaborators across tens of labs at NIMH and NIDA.

Please boost or forward, thank you!