Self-Attention

The core innovation. Each token computes weighted sum of values from all tokens.

For each token, generate three vectors via learned projections:

• Query (Q): "what am I looking for?"
• Key (K): "what do I represent?"
• Value (V): "what do I contribute?"

Where X is (T, D) for T tokens, D embedding dim.

Attention scores: dot product of each query with each key.

Softmax over keys (per query):

Weighted sum of values:

Each output token = weighted average of all V's, weighted by Q-K similarity.

Putting together:

Why dot product? Measures similarity in embedding space.

Why scale by sqrt(d_k)? Without it, dot products grow large with dimension → softmax saturates → gradients vanish.

Causal mask (for GPT-style):

• Each token can only attend to itself + earlier tokens.
• Mask future positions: scores[i, j] = -inf if j > i.
• After softmax: 0 weight on future.

In encoder models (BERT): no mask, attend bidirectionally. In decoder (GPT): causal mask.