Embeddings

A token embedding maps a discrete token (word or character) to a dense vector.

For vocab_size=10000 and embedding_dim=256: 10000×256 = 2.56M parameters.

The model learns these during training. Tokens that appear in similar contexts get similar embeddings.

Famous example: King − Man + Woman ≈ Queen (in Word2Vec, GloVe).

Modern models use byte-pair encoding (BPE) for tokens, not words:

• Vocab ~50000 BPE tokens.
• "unhappiness" → ["un", "happ", "iness"].
• Common words = 1 token. Rare words split.

Position encoding:

• Transformer is permutation-invariant by default.
• Need positional info: which word came first?

Four approaches:

1. Absolute position embeddings: learned per position.

   • position_embeddings[i] for i = 0..max_length.
   • Limit: max_length is fixed.

2. Sinusoidal (original Transformer):

   • PE(pos, 2i) = sin(pos / 10000^(2i/dim))
   • PE(pos, 2i+1) = cos(pos / 10000^(2i/dim))
   • No params; works for any length.

3. Rotary Position Embedding (RoPE) (modern, Llama):

   • Apply rotation matrix during attention.
   • Better extrapolation to longer sequences.

4. ALiBi:

   • Add positional bias directly to attention scores.
   • Used by some recent LLMs.

Sum is the standard combine. Position info lives alongside content.

Embedding dimension scales with model:

• GPT-2 small: 768
• GPT-3: 12288
• Llama 70B: 8192

The first thing each layer sees is the embedding sequence.