Cross-attention (encoder-decoder attention)

Cross-attention (encoder-decoder attention) is the attention sub-layer in each decoder layer that takes its queries from the decoder and its keys and values from the encoder's output, so every target position can look at the whole source sentence.

Last updated: 07 Oct, 2026 · NumPy

The Transformer decoder has three sub-layers. Masked self-attention lets the target tokens look at each other. Cross-attention is the only place where the decoder looks at the input sentence.

Taking queries from the decoder and keys and values from the encoder

In the paper's figure, two of the three arrows into the decoder's second multi-head attention come from the encoder output, and one comes from the decoder's own Add & Norm. The video names them:

• Encoder output → a set of attention vectors K and V, the keys and values.
• Masked multi-head attention output → the attention vector Q, the query, built from the shifted output embeddings and positional encoding.

These keys and values are used by each decoder in its encoder-decoder attention layer, and they help the decoder focus on the appropriate places in the input sequence. Inside, the computation is the one from Self-attention: multiply the queries with the keys, apply softmax, multiply with the values. Only the sources of Q, K and V differ.

More precisely, the encoder outputs one matrix H, the top encoder layer's vectors, one row per source token. Each decoder layer projects H with its own W_K and W_V (one pair per head) and its own decoder states Y with W_Q. The same H is used by all six decoder layers.

The weight matrix has one row per target position and one column per source token, so its shape is (target length × source length), and the output has one vector per target position. This is the transformer's version of the encoder-decoder attention in Attention mechanism (Bahdanau and Luong), with the RNN replaced by attention layers on both sides.

Masking in cross-attention

Cross-attention has no look-ahead mask. The source sentence is complete before decoding starts, so every target position may look at every source token. The only mask is the source padding mask, which blocks the columns of padded source tokens.

Cross-attention between two sentences in NumPy

The source “Je suis étudiant” is padded to four tokens; the decoder input is “<s> I am a student”. H and Y are random stand-ins for the encoder output and for the decoder states after masked self-attention, with d = 6 so the shapes are easy to tell apart.

Projecting Q from the decoder, K and V from the encoder

Q = Y @ WQ          # queries from the decoder   (target length, d)
K = H @ WK          # keys from the encoder      (source length, d)
V = H @ WV          # values from the encoder    (source length, d)

import numpy as np

rng = np.random.default_rng(7)
d = 6
src = ["Je", "suis", "étudiant", "<pad>"]       # source sentence, padded to 4
tgt = ["<s>", "I", "am", "a", "student"]        # decoder input (shifted right)

H = rng.normal(0, 1, (len(src), d))             # top encoder output, one row per source token
Y = rng.normal(0, 1, (len(tgt), d))             # decoder states after masked self-attention + Add & Norm
WQ, WK, WV = (rng.normal(0, 0.5, (d, d)) for _ in range(3))

Q = Y @ WQ                                      # queries from the decoder
K = H @ WK                                      # keys from the encoder
V = H @ WV                                      # values from the encoder

scores = Q @ K.T / np.sqrt(d)                   # (target length, source length)
src_keep = np.array([1, 1, 1, 0])               # source padding mask, no causal mask
scores = np.where(src_keep == 1, scores, -np.inf)
w = np.exp(scores - scores.max(axis=-1, keepdims=True))
w = w / w.sum(axis=-1, keepdims=True)
out = w @ V

print("Q", Q.shape, " K", K.shape, " V", V.shape)
print("attention weights", w.shape)
print(np.round(w, 3))
print("rows sum to", np.round(w.sum(axis=1), 6))
print("output", out.shape, "= one vector per target position")

Q (5, 6)  K (4, 6)  V (4, 6)
attention weights (5, 4)
[[0.135 0.072 0.793 0.   ]
 [0.197 0.27  0.532 0.   ]
 [0.245 0.451 0.304 0.   ]
 [0.144 0.834 0.021 0.   ]
 [0.057 0.933 0.01  0.   ]]
rows sum to [1. 1. 1. 1. 1.]
output (5, 6) = one vector per target position

What the shapes and weights show

• Q is (5, 6), K and V are (4, 6): five target positions ask, four source tokens answer.
• The weights are (5, 4), target length × source length, not square as in self-attention.
• The last column is 0 on every row: the source padding mask hides <pad>. There is no triangle of zeros, because nothing is hidden from the future.
• Each row sums to 1, and the output is (5, 6), one vector per target position. The weights themselves are random, since nothing is trained here.

Self-attention vs masked self-attention vs cross-attention

Where you use cross-attention

• Translation and summarization: each output word looks back at the source sentence.
• Image captioning and speech recognition: the decoder attends to image patches or audio frames encoded by another network.
• Text-to-image models: the image generator attends to the encoded text prompt.

Related

• Previous: Transformer decoder
• Next: Linear and softmax output layer
• See also: Attention mechanism (Bahdanau and Luong)