Tokenization

A search engine starts by tokenizing documents into searchable units (terms).

A naive tokenizer:

"The QUICK brown fox JUMPS over."
  -> split on whitespace
  -> ["The", "QUICK", "brown", "fox", "JUMPS", "over."]

Real tokenizers do more:

• Lowercase all terms (The -> the)
• Strip punctuation (over. -> over)
• Remove stop words (the, a, is, ...) — they're too common to be useful
• Stem to root forms (jumps -> jump, running -> run) — Porter stemmer is the classic
• Handle Unicode — café should match cafe or not, depending on language

For non-English text:

• CJK (Chinese/Japanese/Korean): no whitespace; use n-gram tokenization or a dictionary
• Arabic / Hebrew: right-to-left, but tokenization same as English after normalization
• German: compound words (Donaudampfschifffahrt) need decomposition

For this course, we'll use a simple tokenizer: lowercase, strip non-alphanumeric, remove stop words, no stemming (yet).

The tokens are what you index. Querying applies the SAME tokenization to the query, so 'JUMP' and 'jump' both find documents containing 'jumps' (with stemming).