Every layer in a transformer is a memory store, and the capacity math shows why attention wins (alexwang.ai)
A linear layer holds about as many memories as its key dimension. Fitting them jointly doubles that, ReLU MLPs reach about 2d, and softmax attention with sharpness sqrt(d) and d=64 stores around 1.2 million.
Sign in to save