Porter Stemmer

Stemming reduces inflectional forms to a common root so that searching for jump matches jumps, jumped, jumping.

The classical algorithm is Martin Porter's stemmer (1980) — Manning describes it in IIR Ch.2. It's a deterministic suffix-stripping algorithm: no dictionaries, no statistics, just five passes of rules.

Key concepts

Measure m(stem) — count VC alternations in a stem (after collapsing runs):

• m("tree") = 0 (one VC of "tr"+"ee" -> after consonant block, no VC)
• m("troubles") = 2
• m("private") = 2

Vowel rule for 'y' — y is a vowel iff the preceding character is a consonant. So sky has no vowel; happy has one.

The 5 steps (simplified)

• Step 1a: sses -> ss, ies -> i, ss -> ss, s -> ''
• Step 1b: eed -> ee (if m(stem)>0); ed / ing removed if stem has a vowel; post-rules for doubled consonants and CVC patterns
• Step 1c: (*v*) y -> i (replace y with i if stem has a vowel)
• Step 2/3/4: long list of suffix transforms gated by m(stem) > 0 (e.g. ization -> ize, ational -> ate, ful -> '')
• Step 5: trailing-e removal, doubled-l reduction

Examples (canonical)

When to use

• Always apply the same stemming to docs AND queries.
• English-only. For other languages, use Snowball stemmers (Porter's later project) or language-specific morphological analyzers.
• Modern alternative: lemmatization (dictionary-based; smarter but slower and language-specific). Trade-off: recall vs precision.

Porter is approximate — it can over-stem ("operate" -> "oper") or under-stem ("photography" / "photographer" / "photograph" all land at "photograph") — but for free-text search it's an enormous recall win.