My self-advice for writing grants is to write each sentence in the voice of David Attenborough.

Good writing is a story. A story has an arc. A story has a theme. A story has characters. A story has resolution. The best stories, though, are also narrated by a kind British man with a passion for nature and education.

I'm crowdsourcing career advice. I want to study ⭐​ What humans find easy or hard to learn ⭐​ Tell me: what does this bring to mind for you? Whose research? What approaches?

I'm open to suggestions spanning all fields, including:
- learning science
- critical period & controlled rearing research
- deep learning theory
- dev. psych

⭐​ What defines the line between easy vs. hard tasks?
⭐​ When can brain areas change specialties (think chess experts, blind individuals), and what determines their new specialty?
⭐​ How do learning biases sculpt the adult brain?

Help me build a reading list or find mentors!

I felt called out by this, as a scientist:

"Our success – my success – is the community's success. Your talent, your skill, I will celebrate it because I also see that as mine – even though you are the one that is performing that song. Because we are so interconnected as a community, I am practicing to see your joy as my joy. So there’s freedom there, there’s a freedom in sharing the happiness. There’s freedom in sharing the success and in the growth also."
– Br. Pháp Hữu

I'd love to feel more of this sentiment in science. What of one's work and ideas is truly and solely one's own?

Interpretable AI really wants to understand what neurons in LLMs are doing. But this effort is very likely to fail – and it's not the right approach to understand what AI is doing and why.

Like, today, there's weirdly a lot of press about how OpenAI just showed that "Language models can explain neurons in language models" (https://openai.com/research/language-models-can-explain-neurons-in-language-models). But look at the metrics – this was a failed effort. GPT-4 *cannot explain* what neurons in GPT-2 are doing.

More importantly, single-unit interpretability in LLMs is not the same as understanding why and what LLMs as a whole are doing. Even if you did understand when a handful of units activate, you will never be able to stitch these together into a general understanding of why an LLM says the words that it does.

LLMs may someday be able to explain themselves in plain language. But describing (in plain language) when each neuron fires is not going to get us there.

#interpretableAI #LLMs #openai

I love this preprint from Tzuhsuan Ma and Ann Hermundstad for its point that, as a theorist, you can't separate "optimal" sensory representations from "optimal" behavior. The optimal action depends the constraints upon a sensory system.(https://www.biorxiv.org/content/10.1101/2022.08.10.503471v1)

For background, there's lots of theory about the optimal way an animal can update its beliefs about the world (a sensory problem) and, separately, the optimal way to act given one's beliefs (an action problem). This separation is fine as long as one has optimal beliefs. But biology is constrained. Sub-optimality means that the action problem is no longer disjoint from the sensory problem – evolution must tailor representations for action.

The analysis is beautiful. One sees first-hand how Bayesian-like behavior does not imply a truly Bayesian program.

@EliasNajarro very interesting! Amazing this works so well.

It also seems like low-rank adaptation might also be useful for theories of learning in neuroscience. Instead of adjusting all synapses in the brain for a new task in a continual learning setting (as implied by some gradient-descent-in-the-brain), it might be better to keep most weights fixed and only optimize a separate low-rank adapting loop.

Kinda reminds me of reinforcement learning in cortex -> BG -> thalamus loops.