Transformer decoder

The transformer decoder is the stack of layers that generates the output sequence one token at a time, using the encoder's output and the tokens it has generated so far.

Last updated: 07 Oct, 2026 · NumPy

The Transformer encoder reads the whole input at once. The decoder's job is different: it writes, and writing has an order. That order is why the decoder has a mask, a shifted input and two ways of running, one for training and one for inference.

Listing the three sub-layers of a decoder layer

One decoder layer has three main components, each followed by Add & Norm:

1. Masked multi-head self-attention over the output tokens (Masked self-attention).
2. Multi-head attention over the encoder output, called encoder-decoder attention or Cross-attention (encoder-decoder attention). The output of the encoder stack goes into this sub-layer of every decoder.
3. A feed-forward neural network, the same position-wise network as in the encoder.

The encoder takes x1, x2, x3 all at once. The decoder produces y1 at time step t = 1, y2 at t = 2 and y3 at t = 3. The video splits the decoder's behaviour into a training mechanism, where the real outputs y1, y2, y3 are given to the decoder together with padding and masking, and an inference mechanism, where a new input is translated with no outputs given.

Shifting the target right

The paper's figure labels the decoder's input “Outputs (shifted right)”. Shifting right means putting a start token in front of the target and dropping its last token. For the target “I am a student <eos>” the decoder input is “<s> I am a student”. Position 1 sees <s> and must predict “I”, position 2 sees <s> I and must predict “am”, and so on. Each position predicts the next token from the tokens before it, which is what it will have to do at inference. Padding to a common length is a separate step that happens after the shift.

Training with teacher forcing

In training the whole shifted target goes in at once and all positions are computed in parallel. The real previous tokens are fed in, not the model's own guesses; this is called teacher forcing. The look-ahead mask is what makes the parallel pass honest: row t can only see positions 1 to t, so its output is the same as if the sentence had been fed one token at a time. The loss is computed at every position in the same pass. The video's y1, y2, y3 given in training are these real target tokens, fed one position later than they are predicted.

Generating at inference time

At inference there is no target. The decoder starts with <s> alone, predicts the first word, appends it to its input and runs again, until it produces <eos> or reaches a length limit. The encoder runs once; the decoder runs once per generated token.

Checking that the mask makes one pass equal step by step

The claim above can be tested. Run masked self-attention on all five positions at once, then again on each prefix alone (position 1 with only <s>, position 2 with <s> I, ...) and keep the last row. With the causal mask the two must agree. Without it, they must not.

The shifted decoder input

target = ["I", "am", "a", "student", "<eos>"]
dec_in = ["<s>"] + target[:-1]                   # start token in, last token out

import numpy as np

target = ["I", "am", "a", "student", "<eos>"]
dec_in = ["<s>"] + target[:-1]                   # shifted right: start token in, last token out
print("decoder input :", dec_in)
print("labels        :", target)

rng = np.random.default_rng(42)
d = 4
X = rng.normal(0, 1, (len(dec_in), d))            # embeddings + positions of the decoder input
WQ, WK, WV = (rng.normal(0, 0.5, (d, d)) for _ in range(3))

def self_attention(X, causal=True):
    Q, K, V = X @ WQ, X @ WK, X @ WV
    s = Q @ K.T / np.sqrt(d)
    if causal:
        s = np.where(np.tril(np.ones((len(X), len(X)))) == 1, s, -np.inf)
    w = np.exp(s - s.max(axis=-1, keepdims=True))
    return (w / w.sum(axis=-1, keepdims=True)) @ V

for causal in (True, False):
    parallel = self_attention(X, causal)                                   # training: one pass
    step_by_step = np.array([self_attention(X[: t + 1], causal)[-1] for t in range(len(X))])
    print(f"causal mask {causal}: parallel pass == one position at a time:",
          np.allclose(parallel, step_by_step))

decoder input : ['<s>', 'I', 'am', 'a', 'student']
labels        : ['I', 'am', 'a', 'student', '<eos>']
causal mask True: parallel pass == one position at a time: True
causal mask False: parallel pass == one position at a time: False

What the comparison shows

• The decoder input is ['<s>', 'I', 'am', 'a', 'student'] and the labels are the target itself: input and label differ by one position.
• With the causal mask, the parallel pass equals the step-by-step run (True). Training all five positions at once teaches the same thing as generating word by word.
• Without the mask they differ (False): in the parallel pass, early positions would see later tokens they will never have at inference.

Teacher forcing vs autoregressive inference

Where you use the transformer decoder

• Machine translation (the paper's task): an encoder reads French, the decoder writes English.
• Text generation: GPT models are decoder stacks without the cross-attention sub-layer.
• Summarization and question answering in encoder-decoder models such as T5 and BART.

Related

• Previous: Masked self-attention
• Next: Cross-attention (encoder-decoder attention)
• See also: Encoder-decoder (seq2seq) models