From token IDs to vectors

Neural networks operate on continuous vectors, but text is made of discrete tokens (word pieces, characters, etc.), each represented by an integer ID. The embedding table bridges this gap: it's simply a matrix of shape [vocab_size, d_model] where row i is the learned vector representation of token i. Looking up a token's embedding is just indexing into this matrix — no computation, just a row fetch:

embedding(token_id) = EmbeddingTable[token_id]

During training, these rows are ordinary learnable parameters updated by backprop like any weight matrix — tokens that behave similarly in context tend to converge toward similar vectors, which is where the well-known "semantic geometry" of embeddings (king − man + woman ≈ queen) comes from.

The problem embeddings alone can't solve: order

A raw sequence of embeddings is a set of vectors, not an ordered sequence — nothing in the embedding lookup itself encodes "this token came before that one." Since attention (used throughout Transformers) treats all positions symmetrically, the model needs some explicit signal to know where in the sequence a token sits.

Sinusoidal positional encoding

The original Transformer paper's solution: add a fixed (not learned) vector to each position's embedding, built entirely out of sine and cosine waves at different frequencies:

PE[pos, 2i]   = sin(pos / 10000^(2i/d))
PE[pos, 2i+1) = cos(pos / 10000^(2i/d))

Key structural details:

• Even indices get sine, odd indices get cosine, both driven by the same frequency for a given pair (2i, 2i+1). Index k determines its pair number via k // 2, and that pair number feeds into the frequency term as 2i — so indices 0,1 share i=0, indices 2,3 share i=1, and so on.
• The frequency decreases as i grows — low dimensions oscillate rapidly with position (good for distinguishing nearby positions), high dimensions oscillate slowly (good for distinguishing far-apart positions). The 10000 base is an arbitrary but conventional wavelength scale.
• At pos = 0, every sin term is sin(0) = 0 and every cos term is cos(0) = 1 — a useful sanity check when debugging: position 0's encoding should always be an alternating 0, 1, 0, 1, ... pattern regardless of d.

This scheme was chosen (over, say, learned positional embeddings) partly because it lets the model generalize to sequence lengths longer than anything seen in training, and because relative offsets between positions can be expressed as a linear function of the encoding — useful properties for attention to exploit.

Combining the two

The standard Transformer input is simply:

input_repr[pos] = TokenEmbedding[token[pos]] + PositionalEncoding[pos]

Element-wise addition — the two signals share the same dimensionality d so they can be summed directly, blending "what token is this" and "where is it in the sequence" into a single vector per position. Everything downstream (attention, feed-forward layers) then operates on these combined vectors, with the network free to learn to disentangle the two signals as needed.

Building the lookup and positional encoding independently, then summing them, mirrors exactly how a Transformer's input embedding layer works in practice.