Marino PaganNov 29, 2022

🚨🚨🚨NEW PREPRINT🚨🚨🚨

A big mystery in brain research is what are the neural mechanisms that drive individual differences in higher order cognitive processes. Here we present a new theoretical and experimental framework, in collaboration with Vincent Tang, Mikio Aoi, Jonathan Pillow, Valerio Mante, @SussilloDavid and Carlos Brody.

https://www.biorxiv.org/content/10.1101/2022.11.28.518207v1

1/16

Show threadNicole RustNov 29, 2022

@marinopagan @SussilloDavid
This is really exciting. Congratulations!

Here we've been talking a lot about complex dynamical systems, and I have so many questions.

First one: Over on the bird site, @SussilloDavid wrote, "Hell, I’ll go bigger. This work provides strong indirect evidence for the computation thru dynamics framework."

Can you please unpack that just a bit more @SussilloDavid ?

@PessoaBrain @DrYohanJohn @manlius @kordinglab @neuralengine @cogneurophys
@complexsystems

Show threadKonrad Kording
@NicoleCRust @marinopagan @SussilloDavid @PessoaBrain @DrYohanJohn @manlius @neuralengine @cogneurophys @complexsystems yeah. I am quite unclear how else we could compute than through dynamics. So would be very curious about the answer.
Nov 29, 2022 at 9:39pmWeb
Show threadSimon SchultzNov 29, 2022
@kordinglab @NicoleCRust @marinopagan @SussilloDavid @PessoaBrain @DrYohanJohn @manlius @cogneurophys @complexsystems
Yes I agree with you Konrad. I guess one could view computations that used only steady state processes to be not computing "with dynamics". (But they would still have dynamics).
Show threadBen de BivortNov 29, 2022

@neuralengine @kordinglab @NicoleCRust @marinopagan @SussilloDavid @PessoaBrain @DrYohanJohn @manlius @cogneurophys @complexsystems We found evidence that individuality in odor-choice assays can arise in the sensory periphery https://www.biorxiv.org/content/10.1101/2021.12.24.474127

We think of it as differences in odor-coding rather than dynamics, but I suppose some brief integration is needed to realize small individual differences in firing rates.

Show threadNicole RustNov 29, 2022

@neuralengine @kordinglab @marinopagan @SussilloDavid @PessoaBrain @DrYohanJohn @manlius @cogneurophys @complexsystems

I suppose it's only fair for me to copy the second part of that post as well, which elaborates a bit: "By reasoning about input projections onto *putative* line attractors, Marino could explain a surprising amount of behavioral heterogeneity.

Destined to be a classic."

Still looking forward to the unpack now that @SussilloDavid gets at least 500 characters.

Show threadDavid SussilloNov 30, 2022

@kordinglab @NicoleCRust @marinopagan

I’d distinguish computation without dynamics as feed forward computation.
y = f(g(h(x))) isn’t dynamical imo
x_n = f(f(f(…(x_0)))) is imo

As vision has historically been the paradigm, the top equation has been the predominant approximation in neuro.

Recurrence (bottom) brings a different set of formalisms and appears be useful in other, more traditionally ignored areas.

Show threadDavid SussilloNov 30, 2022
@kordinglab @NicoleCRust @marinopagan
We wrote an entire review about what we meant, though no doubt Konrad you already know it and find it all trivial.
https://www.annualreviews.org/doi/abs/10.1146/annurev-neuro-092619-094115
Show threadNicole RustNov 30, 2022
@SussilloDavid @kordinglab @marinopagan Really helpful - thank you!
Show threadNicole RustNov 30, 2022
@SussilloDavid @kordinglab @marinopagan
I’m really excited to dig into this. @marinopagan: as I do, can you help with a TL;DR on the theoretical build on Mante/Sussillo paper that we all know and love? The experiment advanced (eg lots of subjects) is clear to me. I get that there’s a theoretical advance too - what should I be looking for?
Show threadMarino PaganNov 30, 2022

@NicoleCRust @SussilloDavid @kordinglab

In Mante/Sussillo, monkey data suggested that irrelevant info is not gated before reaching frontal cortex. To understand how computations in frontal cortex can selectively accumulate relevant evidence, they trained RNNs and found that most RNNs solved the task using context-dependent recurrent dynamics. This led to the idea that context-dependent recurrent dynamics is a candidate solution for how the brain solves the task.

1/3

Show threadMarino PaganNov 30, 2022

@NicoleCRust @SussilloDavid @kordinglab
In our work, we asked whether other solutions exist. Our key theory result is that, under assumptions supported by both monkey and rat data, all networks solving the task use a combination of three components. This led us to multiple insights: 1) Most RNNs trained with gradient descent converge to a single solution, context-dependent recurrent dynamics (which we call selection vector modulation), but the full space of solutions is larger than that!

2/3

Show threadMarino PaganNov 30, 2022

@NicoleCRust @SussilloDavid @kordinglab
2) Using the math of our theory result (Eq. 1 and Eq. 2 in the paper) we can now engineer RNNs spanning the full space of solutions. 3) Different solutions have anatomical implications (i.e. impose constraints on network architecture), as well as implications on the resulting neural dynamics and behavior. 4) A specific link exists between neural dynamics and behavioral integration, which was crucial to explain individual variability in our rat data.

3/3

Show threadRob MokNov 30, 2022

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Sorry to butt in but maybe this is a good time to raise this point I've wondered for a while. In this task colour and motion are separate and so it might be less surprising that there is no attention gating or suppression to pfc. In tasks where features are more integrated or not orthogonal, it would make more sense to suppress irrelevant info. And of course others have shown enhancement of relevant vs irrelevant in pfc.

Show threadRob MokNov 30, 2022

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

I guess I wonder how general it is to fit to this task? NNs in my experiences don't show "attention" effects automatically just from training.

Show threadMarino PaganNov 30, 2022
@Rob_Mok @NicoleCRust @SussilloDavid @kordinglab
Not sure what you mean by "attention effects". RNNs are trained to form context-dependent decisions so context must somehow influence computation. Here we find that RNNs trained with backprop rely mostly on changes of recurrent dynamics across contexts, so the general cautionary tale is that the space of solutions can be bigger than what is found w/ backprop. However I am super interested in understanding how these findings relate to other tasks!
Show threadRob MokNov 30, 2022

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Attention effects meaning units increase or decrease firing rate or modulate tuning curves,which often also affects the strength of the stimulus rep (eg lower decoding accuracy).

To show increase or decreasing in firing rate some tasks have a neutral condition where both are attended.

Show threadMarino PaganNov 30, 2022
@Rob_Mok @NicoleCRust @SussilloDavid @kordinglab
Two quick points: 1) In our rat task the two features (location and frequency) are orthogonal exactly like motion and color in the original Mante/Sussillo task (see figure). 2) There is an ongoing debate about gating of irrelevant info before reaching frontal cortex, with potentially different results across tasks. We note that, even though we don't observe it experimentally, gating of irrelevant info is also captured by our theoretical framework.
Show threadRob MokNov 30, 2022

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Great stuff thanks. Have to check it out!

Show threadharrison ritzNov 30, 2022

@marinopagan @Rob_Mok @NicoleCRust @SussilloDavid @kordinglab
There’s good work in cog control on this q https://pubmed.ncbi.nlm.nih.gov/18760657/

But, I think generalization is a little overrated. The brain is optimizing for local conditions, so it will use task-appropriate strategies when necessary.

Multiple conflict-driven control mechanisms in the human brain - PubMed

Conflict between competing neural representations is thought to serve as an internal signal for the recruitment of 'cognitive control', which resolves conflict by biasing information processing in line with current task demands. Because conflict can occur at different levels of stimulus and response …

PubMed
Show threadSimon SchultzNov 30, 2022
@SussilloDavid @NicoleCRust @kordinglab @marinopagan it seems to me to be a very strange definition of dynamics. Even a feedforward system has dynamics. As a case in point, some held the sharp overshoot responses of V1 and MT cells to be evidence for some kind of feedback gain control system - but they can be entirely explained by the transient dynamics of linear filters (as described in any undergrad control or circuits textbook).
Show threadSimon SchultzNov 30, 2022
@marinopagan @SussilloDavid @kordinglab @NicoleCRust of course, feedback systems can have much more rich and complex dynamics, and I certainly believe that these are important for neural computation. But if feedback vs feedforward computation is what is meant, would not that be a better term?
Show threadDavid SussilloNov 30, 2022
@neuralengine @marinopagan @kordinglab
@NicoleCRust
In idealized system the top equation doesn’t have dynamics. In brains, where there are complex dynamics in every neuron even for an AP, then I see your point from your first post.
Feedforward works fine, but feedback to me implies controllers, optimal control, etc. CtD was meant rather the pursuit of how a massively recurrent network of simple neurons computes.
Show threadDavid SussilloNov 30, 2022
@neuralengine @marinopagan @kordinglab
@NicoleCRust
Ten years ago, we could barely train RNNs, and nobody understood how an optimized rnn worked on toy problems. Now we can understand trained rnn solutions on toy problems.
Relatedly, outside of single neuron integrators in LIP, or population vectors, few had any clue about dynamics of neurons eg motor or frontal cortex.
We now have sophisticated hypotheses, even if maybe wrong.
Show threadSimon SchultzNov 30, 2022
@SussilloDavid @marinopagan @kordinglab @NicoleCRust I'm definitely not disagreeing that feedback is important (and you are using the word recurrence - to me they are the same thing - we are not talking about linear control systems, optimal or otherwise). I think this is likely just a semantic issue. But I would say that computation can use dynamics even in a feedforward system, just as a logical point (although it is simpler dynamics).