Rylan Schaeffer

Logo
Resume
Research
Learning
Blog
Teaching
Jokes
Kernel Papers


8 September 2022

Paper Summary - "Universal Hopfield Networks"

by Rylan Schaeffer

The below are my notes on Millidge et al. 2022’s Universal Hopfield Networks.

Summary

Universal Hopfield Networks

Hopfield Networks

\[z = sign(M^T identity(M q))\]

Sparse Distributed Memories

\[z = P thresh(hamming(M, q))\]

Modern Continuous Hopfield Networks

\[z = W^T softmax (W q)\]

Continuous Sparse Distributed Memories

\[z = P softmax (A q)\]

Auto-associative vs Hetero-associative Memories

If the project matrix \(P\) is the same as \(M\), then a memory model is called auto-associative; if the two are different, then the memory model is called hetero-associative.

Relation to Transformer Networks

A hetero-associative Modern Continuous Hopfield Network is equivalent to a self-attention layer:

\[z = V softmax(K q)\]

Neural Dynamics

Experiments

Using the dot product for similarity performs more poorly than using a Manhattan or Euclidean distance.

tags: machine-learning - neuro-ai - memory