TF-IDF Scoring

Boolean queries return a SET; users want a RANKING. The classic ranking:

Term Frequency (TF): how often the term appears in the doc.

tf(term, doc) = count of term in doc

Inverse Document Frequency (IDF): how rare the term is across ALL docs.

idf(term) = log(N / df(term))
  where N = total docs, df = number of docs containing term

A common term like 'the' is in every doc, so idf ≈ 0 — it shouldn't score highly. A rare term like 'sphagnum' is in 1 doc, so idf ≈ log(N) — strong signal.

TF-IDF score: tf(term, doc) * idf(term).

For multi-term queries: sum the per-term scores.

query: "quick fox"
doc1: tf(quick, doc1) * idf(quick) + tf(fox, doc1) * idf(fox)
doc2: tf(quick, doc2) * idf(quick) + tf(fox, doc2) * idf(fox)

Sort docs by total score, return top N.

Refinements:

• TF normalization: long docs naturally have more occurrences. Common fix: 1 + log(tf) (sublinear) or tf / doc_length.
• Cosine similarity: treat docs and queries as TF-IDF vectors; rank by cosine angle.

TF-IDF was state-of-the-art ranking from the 1970s through ~2000s. BM25 (next lesson) is a refinement that's still SOTA for many tasks. Modern neural rankers (ColBERT, BERT cross-encoders) beat it but cost 100x more compute.