The causal mask and position

The causal mask

A text generator learns to predict the next token. If, during training, it could read the answer, it would just copy it. So each token is only allowed to attend to earlier tokens. The code does this by setting the scores of later positions to minus infinity, so that softmax gives them weight 0. This is a causal mask.

Read the picture: each row is a token that is reading ("the", "agent", "called", "the", "tool") and each column is a token it looks at. Row 2 ("agent") is 0.84 on "the" and 0.16 on itself, and 0 on the three later words. The weights in this picture come from random, untrained matrices, so they show the mechanism, not learned behaviour.

Two tests make the effect concrete. If you change only the last token of a sequence, every earlier output is unchanged under the causal mask. Without the mask, the earlier outputs change. That also means a model can keep its earlier work while it generates, which is what saves time in section 4.

Position

Attention treats its input as a set. A test shows it: shuffle the input tokens and the outputs are simply shuffled in the same way, so "dog bites man" and "man bites dog" would look the same. So models add position information to each token's vector. The original method adds a fixed pattern of sines and cosines (the example's sinusoidal_positions). Here are the first three positions for four numbers (real output):

position 0   [ 0.00   1.00   0.00   1.00 ]
position 1   [ 0.84   0.54   0.01   1.00 ]
position 2   [ 0.91  -0.42   0.02   1.00 ]

Each position gets its own pattern. Other models learn positions, or, in many modern models, rotate the queries and keys by an amount that depends on position (covered in the deep-dive module). Tests confirm that adding positions breaks the shuffling symmetry and that every position gets a unique signal.