Every layer in a transformer is a memory store, and the capacity math shows why attention wins

Capacity as an interference budget

The post argues that linear layers, MLPs and self-attention are all associative memory stores: each saves key-value pairs and recalls values by similarity, and their capacities differ in how they handle interference. The metric is the interference-to-signal ratio; capacity is the number of memories at which interference equals signal.

An outer-product store has capacity of roughly the key dimension d_k. Fitting all memories jointly by least squares whitens the keys and roughly doubles it to 2d_k. Passing similarities through a nonlinearity suppresses weak matches: ReLU reaches about 2d_k, squared ReLU about 2d_k^2/3, cubed ReLU about 2d_k^3/15. With the exponential, softmax self-attention capacity grows exponentially with sharpness; at sharpness sqrt(d_k) and d_k=64 it is about 1.2 million memories, and since each context token adds a memory, attention is non-parametric. Scaling sharpness with log M keeps interference low as memories grow.

Readers brought their own fields: neuroscience and signal processing

The thread split between readers who found the math impenetrable and readers who recognized it. A neuroimaging scientist whose bio lists memory function and brain development pointed the author to biology:

"Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you."

A self-described electrical engineer said they asked a chatbot to translate the post into DSP terms, and it helped. The vocabulary of interference, signal and capacity is the one a communications engineer already uses.

The post's estimates assume random keys and values, so they are upper-bound intuitions for why attention scales with context, not measured limits; studies that count bits per parameter reach different numbers.