BM25 Scoring

BM25 (Okapi BM25) is the modern refinement of TF-IDF. Used by Lucene/Elasticsearch as the default scorer; tuned and benchmark-leading for ~30 years.

                                     tf(t, d) * (k1 + 1)
score(d, q) = sum_t  idf(t) * ----------------------------------------
                              tf(t, d) + k1 * (1 - b + b * |d| / avg_dl)

Parameters:

• k1: term frequency saturation. Typical 1.2-2.0. Higher = TF matters more linearly.
• b: length normalization. 0 = ignore length, 1 = full normalization. Typical 0.75.

Why BM25 beats TF-IDF:

1. TF saturation: with TF-IDF, doubling the term count doubles the score. With BM25, saturating at high TF is more realistic (a doc with cat 100 times isn't 100x more relevant than one with cat once).

2. Length normalization: long docs naturally have higher TF. The |d| / avg_dl ratio penalizes verbose docs and rewards concise ones.

3. Robust idf: BM25 uses a smoothed idf:

idf(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)

This avoids negative values for very common terms.

For most retrieval tasks, BM25 with default params (k1=1.2, b=0.75) outperforms TF-IDF. Modern neural rankers beat BM25 but typically use BM25 as a first-stage retriever (re-rank the top 100 with the heavy model).

For this lesson: implement BM25 with k1=1.2, b=0.75.