Step 1 of 5 · Reading · ~2 min
Read
Production Concerns
Putting It All Together
You've built a real search engine:
| Layer | What it gave you |
|---|---|
| Tokenization | one normalization applied to documents AND queries |
| Porter stemmer | morphological variants collapse to one index entry |
| Inverted index | term -> sorted posting list, the core lookup |
| Boolean queries | AND / OR / NOT as set algebra over postings |
| Phrase queries | positions, so adjacency becomes checkable |
| Proximity queries | the same positions, relaxed to a window |
| TF-IDF | a set of matches becomes a ranking |
| BM25 | saturation and length normalization on top of it |
| Relevance feedback | the query itself learns from what the user marked |
| Field boosts | structure: a title match outweighs a body match |
| Posting compression | delta + VByte, the reason an index fits on disk |
| Spell correction | the query survives a typo |
What we didn't cover:
- Synonyms: query expansion ("car" -> {"car", "automobile", "vehicle"}) — cheap to add, surprisingly hard to curate.
- Faceted search: structured filters beside the text query — category, price band, date — usually with a result count per facet.
- Highlighting: show snippets with the query terms marked. Needs the positions you already store.
- Pagination cursors: "page 2 of 1000" without recomputing the whole ranking.
- Sharding and distributed merging: split the index across machines, scatter the query to every shard, merge each shard's top-K into one list. Watch out: df and avg_dl are per-shard, so BM25 scores are only approximately comparable across them.
- Vector search: dense embeddings for semantic matching. Hybrid systems run sparse (BM25) and dense retrieval together and fuse the two result lists, because they fail in different ways — BM25 is unbeatable on a rare exact token and blind to paraphrase.
- Incremental indexing and deletes: everything here assumed a corpus that only grows. Real engines write immutable segments and merge them in the background.
Real systems you should look at:
- Lucene (Java) — the workhorse. Used by Elasticsearch, Solr, OpenSearch.
- Tantivy (Rust) — Lucene-inspired, much faster on modern hardware.
- Bleve (Go) — pure-Go full-text engine.
- Whoosh (Python) — pure-Python full-text engine. Excellent for learning.
- Meilisearch / Typesense — newer "instant" engines focused on UX.
You've now built the foundations of every one of them.
Discussion
Ask a question, share an insight, or help someone who’s stuck.
Sign in to post a comment or reply.
Loading…