Mastodawn

@PessoaBrain @DrYohanJohn @manlius @kordinglab @neuralengine @cogneurophys
@complexsystems

Nicole Rust Nov 29, 2022

@marinopagan @SussilloDavid
This is really exciting. Congratulations!

Here we've been talking a lot about complex dynamical systems, and I have so many questions.

First one: Over on the bird site, @SussilloDavid wrote, "Hell, I’ll go bigger. This work provides strong indirect evidence for the computation thru dynamics framework."

Can you please unpack that just a bit more @SussilloDavid ?

Konrad Kording Nov 29, 2022

@NicoleCRust @marinopagan @SussilloDavid @PessoaBrain @DrYohanJohn @manlius @neuralengine @cogneurophys @complexsystems yeah. I am quite unclear how else we could compute than through dynamics. So would be very curious about the answer.

David Sussillo Nov 30, 2022

@kordinglab @NicoleCRust @marinopagan

I’d distinguish computation without dynamics as feed forward computation. y = f(g(h(x))) isn’t dynamical imo x_n = f(f(f(…(x_0)))) is imo

As vision has historically been the paradigm, the top equation has been the predominant approximation in neuro.

Recurrence (bottom) brings a different set of formalisms and appears be useful in other, more traditionally ignored areas.

David Sussillo Nov 30, 2022

@kordinglab @NicoleCRust @marinopagan
We wrote an entire review about what we meant, though no doubt Konrad you already know it and find it all trivial.
https://www.annualreviews.org/doi/abs/10.1146/annurev-neuro-092619-094115

Nicole Rust Nov 30, 2022

@SussilloDavid @kordinglab @marinopagan Really helpful - thank you!

Nicole Rust Nov 30, 2022

@SussilloDavid @kordinglab @marinopagan
I’m really excited to dig into this. @marinopagan: as I do, can you help with a TL;DR on the theoretical build on Mante/Sussillo paper that we all know and love? The experiment advanced (eg lots of subjects) is clear to me. I get that there’s a theoretical advance too - what should I be looking for?

@NicoleCRust @SussilloDavid @kordinglab

Marino Pagan

In Mante/Sussillo, monkey data suggested that irrelevant info is not gated before reaching frontal cortex. To understand how computations in frontal cortex can selectively accumulate relevant evidence, they trained RNNs and found that most RNNs solved the task using context-dependent recurrent dynamics. This led to the idea that context-dependent recurrent dynamics is a candidate solution for how the brain solves the task.

1/3

@NicoleCRust @SussilloDavid @kordinglab
In our work, we asked whether other solutions exist. Our key theory result is that, under assumptions supported by both monkey and rat data, all networks solving the task use a combination of three components. This led us to multiple insights: 1) Most RNNs trained with gradient descent converge to a single solution, context-dependent recurrent dynamics (which we call selection vector modulation), but the full space of solutions is larger than that!

2/3

@NicoleCRust @SussilloDavid @kordinglab
2) Using the math of our theory result (Eq. 1 and Eq. 2 in the paper) we can now engineer RNNs spanning the full space of solutions. 3) Different solutions have anatomical implications (i.e. impose constraints on network architecture), as well as implications on the resulting neural dynamics and behavior. 4) A specific link exists between neural dynamics and behavioral integration, which was crucial to explain individual variability in our rat data.

3/3

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Sorry to butt in but maybe this is a good time to raise this point I've wondered for a while. In this task colour and motion are separate and so it might be less surprising that there is no attention gating or suppression to pfc. In tasks where features are more integrated or not orthogonal, it would make more sense to suppress irrelevant info. And of course others have shown enhancement of relevant vs irrelevant in pfc.

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

I guess I wonder how general it is to fit to this task? NNs in my experiences don't show "attention" effects automatically just from training.

@Rob_Mok @NicoleCRust @SussilloDavid @kordinglab
Not sure what you mean by "attention effects". RNNs are trained to form context-dependent decisions so context must somehow influence computation. Here we find that RNNs trained with backprop rely mostly on changes of recurrent dynamics across contexts, so the general cautionary tale is that the space of solutions can be bigger than what is found w/ backprop. However I am super interested in understanding how these findings relate to other tasks!

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Attention effects meaning units increase or decrease firing rate or modulate tuning curves,which often also affects the strength of the stimulus rep (eg lower decoding accuracy).

To show increase or decreasing in firing rate some tasks have a neutral condition where both are attended.

@Rob_Mok @NicoleCRust @SussilloDavid @kordinglab
Two quick points: 1) In our rat task the two features (location and frequency) are orthogonal exactly like motion and color in the original Mante/Sussillo task (see figure). 2) There is an ongoing debate about gating of irrelevant info before reaching frontal cortex, with potentially different results across tasks. We note that, even though we don't observe it experimentally, gating of irrelevant info is also captured by our theoretical framework.

@marinopagan @NicoleCRust @SussilloDavid @kordinglab

Great stuff thanks. Have to check it out!