Hacker Newshttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011843
#ycombinator #Action_potentials #Neuronal_dendrites #Axons #Neurons #Synapses #Machine_learning #Calcium_channels #Ion_channels
Short-term Hebbian learning can implement transformer-like attention
Author summary Many of the most impressive recent advances in machine learning, from generating images from text to human-like chatbots, are based on a neural network architecture known as the transformer. Transformers are built from so-called attention layers which perform large numbers of comparisons between the vector outputs of the previous layers, allowing information to flow through the network in a more dynamic way than previous designs. This large number of comparisons is computationally expensive and has no known analogue in the brain. Here, we show that a variation on a learning mechanism familiar in neuroscience, Hebbian learning, can implement a transformer-like attention computation if the synaptic weight changes are large and rapidly induced. We call our method the match-and-control principle and it proposes that when presynaptic and postsynaptic spike trains match up, small groups of synapses can be transiently potentiated allowing a few presynaptic axons to control the activity of a neuron. To demonstrate the principle, we build a model of a pyramidal neuron and use it to illustrate the power and limitations of the idea.
CellBioNews