Is Attention a Hopfield Network? The 2020 Equivalence, Stated Precisely

TL;DR

• Yes, transformer attention is equivalent to one update step of a modern continuous Hopfield network, under the substitutions given by Ramsauer et al. 2020.
• The identity is narrow: it is not a claim that transformers were historically derived from Hopfield networks, and it does not cover every detail of trained transformer systems.
• Classical Hopfield capacity has three different numbers in circulation: about 0.15N from Hopfield 1982, αc≈0.138 often rounded to 0.14 from Amit, Gutfreund, and Sompolinsky, and n/(2 ln n) for exact recall from McEliece et al. 1987.
• The useful engineering vocabulary is retrieval quality: single-pattern retrieval, metastable subset averaging, or global averaging.

Transformer attention really is a Hopfield-network update, but only in the precise 2020 sense: one update step of a modern continuous Hopfield network equals scaled dot-product attention after a query, key, value identification. Read that way, the attention operation is a fast associative-retrieval step: it returns a similarity-weighted blend of stored patterns in a single pass. The identity covers that operation alone, not the residual connections, layer norm, MLP blocks, masking, or training dynamics around it. That is the useful statement. The broader folklore version loses the boundary conditions.

For the historical and capacity background, see the companion deepdive on Hopfield associative memory. This article narrows the question to the equivalence itself.

A modern Hopfield network is a continuous-state associative memory whose update rule maps a query state toward stored patterns through an energy function and a softmax-weighted retrieval step.

Associative memory is a memory model that retrieves stored patterns from partial, noisy, or related cues by pattern completion rather than by exact address lookup.

The Ramsauer 2020 Hopfield attention equivalence

Ramsauer et al. state the core claim directly in "Hopfield Networks is All You Need": "The new update rule is equivalent to the attention mechanism used in transformers." The paper was posted to arXiv in 2020 and appeared at ICLR 2021, with companion PyTorch layers published at ml-jku.github.io/hopfield-layers.

About that title. It is a wink at "Attention Is All You Need", the 2017 transformer paper, and the homage doubles as the thesis: the result is that the Hopfield update is attention. The wink also belongs to a genre. A 2025 study, itself titled "'All You Need' is Not All You Need for a Paper Title", counted 717 "All You Need" papers between 2009 and 2025, 200 of them in 2025 alone, and concluded the format rewards "memorability over precision". This one is a rare case where the memorable title is also the theorem.

The modern Hopfield energy in Eq. 2 is:

E = -lse(β, Xᵀξ) + ½ξᵀξ + β⁻¹log N + ½M²

Here lse(β,x) = β⁻¹log Σᵢ exp(βxᵢ), X stores key patterns, ξ is the query state, and M = maxᵢ‖xᵢ‖. The paper states the bound 0 ≤ E ≤ 2M² and proves global convergence in Theorem 1 through the Concave-Convex Procedure.

The update rule in Eq. 3 is the part engineers recognize:

ξ_new = X softmax(β Xᵀξ)

This says: compare the query state with stored patterns, turn similarities into softmax weights, then return the weighted pattern sum.

The transformer mapping appears in Eq. 10. Set xᵢ = W_Kᵀyᵢ and ξᵢ = W_Qᵀrᵢ. Define Xᵀ = K = Y W_K, Ξᵀ = Q = R W_Q, and V = Y W_K W_V = Xᵀ W_V. Take β = 1/√d_k. After multiplying the Hopfield update by W_V, the expression becomes:

Z = softmax(1/√d_k · QKᵀ)V

That is scaled dot-product attention. In self-attention, the paper states R = Y and replaces W_K W_V with just W_V.

The equivalence is therefore exact under those substitutions, at the level of the operation. A query attends over keys and returns a value-weighted sum. A modern Hopfield state updates by attending over stored patterns and returning their weighted sum.

What the Hopfield attention identity does not say

The identity has sharp edges.

First, it is one update step. The update rule typically retrieves patterns after one update, and the paper notes that "retrieval with one update is compatible with activating the layers of deep networks." Theorems 4 and 5 quantify one-step retrieval: after one update, distance to the fixed point is exponentially small in the separation Δᵢ = xᵢᵀxᵢ - max_{j≠i} xᵢᵀxⱼ, under the paper's stated error bound.

Second, the equality sign in the paper's Figure 1 is not a historical claim. The caption says the "=-sign means 'keeps the properties'." Transformer attention was introduced by Vaswani et al. in 2017 (arXiv:1706.03762), three years before this identity was shown. The correct reading is post-hoc mathematical equivalence, not derivation.

Third, trained transformers include details that the base identity does not cover. Millidge's 2024 blog notes that transformer "stored patterns" are learned input-dependent projections, that causal masking is not considered there, and that the correspondence covers a single retrieval step (beren.io). Treat that as a blog interpretation, not a peer-reviewed replacement for the Ramsauer result.

Fourth, later literature refines the scope rather than refuting the identity. Krotov and Hopfield 2021 give an alternative derivation in a microscopic two-body theory with hidden neurons (arXiv:2008.06996). Millidge et al. 2022 frame single-shot associative memories through similarity, separation, and projection, with modern Hopfield networks as one instance (arXiv:2202.04557). Bricken and Pehlevan 2021 relate attention to Kanerva's Sparse Distributed Memory under certain data conditions (arXiv:2111.05498). Hoover et al. 2023 design Energy Transformer layers to minimize an engineered energy function (arXiv:2302.07253). Masumura and Taki 2025 state that agreement holds in the adiabatic approximation and generalize the model with hidden states (arXiv:2511.20698). Among the papers we reviewed, none overturns the identity; each narrows the conditions under which it holds.

So the safe sentence is short: attention is equivalent to one modern Hopfield update under the Ramsauer substitutions.

Hopfield network capacity: 0.15N, 0.138N, and n/(2 ln n)

Capacity folklore often compresses three different questions into one number.

Hopfield's 1982 PNAS paper reported an empirical rule of thumb: "About 0.15 N states can be simultaneously remembered before error in recall is severe." That result came from simulations at N=30 and N=100, and Hopfield also noted overload: adding memories beyond capacity can make all memory states irretrievable unless the system can forget old memories (DOI:10.1073/pnas.79.8.2554).

Amit, Gutfreund, and Sompolinsky gave the statistical-mechanics capacity result. Their 1985 PRL states that associative memory exists only below αc≈0.14 for α=p/N (DOI:10.1103/PhysRevLett.55.1530). Their 1987 Annals of Physics treatment states that the network can retrieve patterns at T=0 with error below 1.5% for α < αc = 0.14, and that retrieval states disappear discontinuously at αc (DOI:10.1016/0003-4916(87)90092-3). The three-decimal value is commonly given as αc≈0.138, often rounded to 0.14.

McEliece, Posner, Rodemich, and Venkatesh asked a stricter question in 1987: exact recall. They showed that if m random fundamental memories are stored, the maximum asymptotic m for most memories to be exactly recoverable is n/(2 log n), with natural logs. For every memory to be exactly recoverable, the bound is n/(4 log n). Above the bound, memories almost surely are not even fixed points (DOI:10.1109/TIT.1987.1057328).

There is no contradiction. 0.138N permits a small error rate. n/(2 ln n) demands exact recovery.

Modern variants changed the scaling question. Krotov and Hopfield 2016 introduced Dense Associative Memory with polynomial interactions and capacity K_max = αₙN^(n−1) (arXiv:1606.01164). Demircigil et al. 2017 used an exponential interaction and stored M = exp(αN)+1 patterns under the paper's conditions (DOI:10.1007/s10955-017-1806-y). Ramsauer et al. 2020 prove exponential capacity in dimension d for continuous states in Theorem 3 (arXiv:2008.02217).

The whole ladder in one view:

Model	Capacity bound	Error tolerance	Setting
Hopfield 1982	about 0.15N	"before error in recall is severe"	binary, empirical (N=30, 100 simulations)
Amit, Gutfreund, Sompolinsky 1985/1987	αc ≈ 0.138 (often rounded to 0.14), so about 0.138N patterns	under 1.5% bit errors at T=0	binary, replica theory
McEliece et al. 1987	n/(2 ln n), or n/(4 ln n) for every memory	zero errors, exact recall	binary, asymptotic
Krotov and Hopfield 2016	αₙN^(n−1)	error threshold stated in the paper	binary patterns, polynomial interaction xⁿ
Demircigil et al. 2017	exp(αN), exponential in N	exact fixed points with probability tending to 1	binary patterns, exponential interaction
Ramsauer et al. 2020	N ≥ √p · c^((d−1)/4), exponential in dimension d	exponentially small retrieval error	continuous states

In the last row, d is the dimension of the associative space, 1−p the success probability, and c a constant the paper derives from β, the pattern radius, and p (Theorem 3; worked examples give c ≥ 3.1546 at d = 20 and c ≥ 1.3718 at d = 75).

Fixed points as a retrieval-quality vocabulary

Ramsauer et al. give three fixed-point types: global fixed points that average over all patterns, metastable states that average over a subset of patterns, and fixed points that store a single pattern.

This taxonomy maps onto retrieval quality.

A single-pattern fixed point is the clean case. The query is close enough to one stored pattern, or the pattern is well separated enough, that the update retrieves one dominant item.

A metastable state is a subset blend. Similar stored patterns compete, so the update returns a weighted average over that subset. The authors' blog describes this as a superposition of multiple stored patterns and notes that lower β, meaning higher temperature, makes metastable states more likely (Hopfield layers blog).

A global average happens when no pattern is well separated. The softmax becomes close to uniform, with pᵢ = 1/N, so the retrieved state averages broadly.

Ramsauer et al. also analyze trained transformer heads through this lens. They write that heads in first layers preferably perform global averaging, while higher layers perform partial averaging via metastable states. That matters: metastability is an operating regime, not a pathology. It gives engineers a precise way to discuss blended retrieval without turning it into an unsupported claim about hallucination.

What this means for AI agent memory

For agents, the equivalence gives a useful boundary between attention and memory.

Attention can act as associative retrieval over the live context. Google Research's Titans paper says attention, because of limited context and accurate dependency modeling, performs as short-term memory, while neural memory acts as longer-term persistent memory (arXiv:2501.00663). The same paper says the key and value matrices act as the model's memory and are updated by appending key and value pairs without compression.

That statement fits the Hopfield reading: keys are stored patterns for the current computation, and attention retrieves by similarity. It also shows the limit. The context window is a short-term workspace. It does not become durable agent memory by itself.

Other work places memory elsewhere in the transformer. Bietti et al. describe transformer weight matrices as associative memories (arXiv:2306.00802). Geva et al. describe feed-forward layers as key-value memories over training patterns (arXiv:2012.14913). MemGPT frames long-running agents as constrained by limited context windows and moves data between fast and slow memory tiers through virtual context management (arXiv:2310.08560).

At Mnemoverse, we treat retrieval as closer to pattern completion than to flat lookup. The Hopfield-attention equivalence gives that design vocabulary a formal anchor, without implying that live attention is durable memory.

Common questions

Is transformer attention really a Hopfield network?

Mathematically, yes: Ramsauer et al. showed that one update step of a modern continuous Hopfield network equals transformer attention under a specific Q/K/V identification.

What is the storage capacity of a Hopfield network?

For classical binary Hopfield networks with Hebbian storage, AGS found αc≈0.138, often rounded to 0.14, for retrieval with under 1.5% errors; exact recall is a stricter n/(2 ln n) result from McEliece et al. 1987.

What did Ramsauer et al. 2020 actually prove?

They proved that the modern Hopfield update ξ_new = X softmax(β Xᵀξ) becomes scaled dot-product attention when X, ξ, the output projection, and β are identified with keys, queries, values, and 1/√d_k.

Were transformers derived from Hopfield networks?

No. Transformer attention was introduced in 2017, and the Hopfield equivalence was shown afterward by Ramsauer et al. in 2020. It is an identity of operation, not a lineage claim.

What are metastable states in modern Hopfield networks?

Metastable states are fixed points that average over a subset of similar stored patterns rather than retrieving one clean pattern or averaging over all patterns.

Sources

Primary papers

• Ramsauer et al., "Hopfield Networks is All You Need", arXiv:2008.02217, ICLR 2021: arXiv, ar5iv, Hopfield layers
• Krotov and Hopfield, "Dense Associative Memory for Pattern Recognition", NeurIPS 2016, arXiv:1606.01164: arXiv
• Demircigil et al., "On a Model of Associative Memory with Huge Storage Capacity", Journal of Statistical Physics 168, 2017, DOI: 10.1007/s10955-017-1806-y

Classical capacity

• Hopfield, "Neural networks and physical systems with emergent collective computational abilities", PNAS 79(8), 1982, DOI: 10.1073/pnas.79.8.2554
• Amit, Gutfreund, and Sompolinsky, "Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks", Physical Review Letters 55(14), 1985, DOI: 10.1103/PhysRevLett.55.1530
• Amit, Gutfreund, and Sompolinsky, "Statistical mechanics of neural networks near saturation", Annals of Physics 173(1), 1987, DOI: 10.1016/0003-4916(87)90092-3
• McEliece, Posner, Rodemich, and Venkatesh, "The capacity of the Hopfield associative memory", IEEE Transactions on Information Theory IT-33(4), 1987, DOI: 10.1109/TIT.1987.1057328

Follow-ups and scope

• Krotov and Hopfield, "Large Associative Memory Problem in Neurobiology and Machine Learning", ICLR 2021, arXiv:2008.06996: arXiv
• Millidge et al., "Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models", arXiv:2202.04557: arXiv
• Bricken and Pehlevan, "Attention Approximates Sparse Distributed Memory", NeurIPS 2021, arXiv:2111.05498: arXiv
• Hoover et al., "Energy Transformer", NeurIPS 2023, arXiv:2302.07253: arXiv
• Masumura and Taki, "On the Role of Hidden States of Modern Hopfield Network in Transformer", NeurIPS 2025, arXiv:2511.20698: arXiv
• Behrouz et al., "Titans: Learning to Memorize at Test Time", Google Research, arXiv:2501.00663: arXiv
• Bietti et al., "Birth of a Transformer: A Memory Viewpoint", arXiv:2306.00802: arXiv
• Geva et al., "Transformer Feed-Forward Layers Are Key-Value Memories", arXiv:2012.14913: arXiv
• Packer et al., "MemGPT: Towards LLMs as Operating Systems", arXiv:2310.08560: arXiv
• Vaswani et al., "Attention Is All You Need", arXiv:1706.03762: arXiv
• Krotov, "A new frontier for Hopfield networks", Nature Reviews Physics, 2023, DOI: 10.1038/s42254-023-00595-y

Blogs

• Ramsauer et al., Hopfield layers project blog: ml-jku.github.io/hopfield-layers
• Millidge, "Linear Attention as Iterated Hopfield Networks", 2024 blog: beren.io

Related

• Hopfield Networks: The Memory Model That Became Attention
• Hebbian Memory for AI Agents
• Multimodal Memory Integration

— Edward Izgorodin · Last updated 2026-08-16

— Mnemoverse is a persistent-memory API for AI agents. Free key: console.mnemoverse.com · Docs: Getting Started