Fabrizio MusacchioMar 4, 2024

Here is a short #tutorial on understanding the #HebbianLearning rule in #HopfieldNetworks (applied to the problem of #PatternRecognition), again accompanied by some #Python code:

🌍 https://www.fabriziomusacchio.com/blog/2024-03-03-hebbian_learning_and_hopfiled_networks/

Feel free to use, share and modify.

#CompNeuro #ComputationalNeuroscience

Understanding Hebbian learning in Hopfield networks

Hopfield networks, a form of recurrent neural network (RNN), serve as a fundamental model for understanding associative memory and pattern recognition in computational neuroscience. Central to the operation of Hopfield networks is the Hebbian learning rule, an idea encapsulated by the maxim ‘neurons that fire together, wire together’. In this post, we explore the mathematical underpinnings of Hebbian learning within Hopfield networks, emphasizing its role in pattern recognition.

Fabrizio Musacchio
Fabrice TshimangaJan 21, 2024

On #biological vs #artificialintelligence and #neuralnetworks
Just skimmed through "Inferring neural activity before plasticity as a foundation for learning beyond backpropagation" by Yuhang Song et al. https://www.nature.com/articles/s41593-023-01514-1

Quite interesting but confusing, as I come from #backpropagation DL.
If I got it right, the authors focus on showing how and why biological neural networks would benefit from being Energy Based Models for Predictive Coding, instead of Feedforward Networks employing backpropagation.
I struggled to reach where they explain how to optimize a ConvNet in PyTorch as an EB model, but they do: there is an algorithm and formulae, but I'm curious about how long and stable training is, and whether all that generalizes to typical computer vision architectures (ResNets, MobileNets, ViTs, ...).
Code is also #opensource at https://github.com/YuhangSong/Prospective-Configuration

I would like to sit a few hours at my laptop and try to better see and understand, but I think in the next days I will go to Modern #HopfieldNetworks. These too are EB and there's an energy function that is optimised by the #transformer 's dot product attention.
I think I got what attention does in Transformers, so I'm quite curious to get in what sense it's equivalent to consolidating/retrieving patterns in a Dense Associative Memory. In general, I think we're treating memory wrong with our deep neural networks. I see most of them as sensory processing, shortcut to "reasoning" without short or long term memory surrogates, but I could see how some current features may serve similar purposes...

Inferring neural activity before plasticity as a foundation for learning beyond backpropagation - Nature Neuroscience

This paper introduces ‘prospective configuration’, a new principle for learning in neural networks, which differs from backpropagation and is more efficient in learning and more consistent with data on neural activity and behavior.

Nature
vintage screwlisp accountApr 21, 2023
#AI #DeepLearning #HopfieldNetworks (My rudimentary package is named #binryhop ; a pun on bunny-hop and binary hopfield ).
So I had an idea. What if I take two source images: A fish and a tree, thresholded. The fish and tree are roughly the same resolution as an image screwtape-fish-tree I will draw. I griddle the high resolution images into 90x90 regions. Then, I make the fish and tree grid pieces my memories and simply run each swatch of my own image through some updates.
Dominic BoutetDec 13, 2022

#arxivfeed :

"Attention in a family of Boltzmann machines emerging from modern Hopfield networks"
https://arxiv.org/abs/2212.04692

#MachineLearning #NeuralNetworks #HopfieldNetworks #BoltzmannMachines

Attention in a family of Boltzmann machines emerging from modern Hopfield networks

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for a special case and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and denoising autoencoder with softmax units. We also investigate BMs introduced by other energy functions, and in particular, observe that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.

arXiv.org