Designing Token Types

From Characters to Categories

With single characters and simple operators handled, the scanner now needs to recognize the three richer token categories every language has: keywords, identifiers, and literals — plus it needs to behave well when the input is wrong.

Keywords vs. Identifiers: the Maximal Munch Rule

Here's a subtlety: print is a keyword, but printer is a perfectly valid identifier. A scanner can't decide "this is the keyword print" after only seeing p-r-i-n-t, because the next character might continue the identifier.

The rule that resolves this is called maximal munch: when scanning an identifier-like token, consume the longest possible run of valid identifier characters (letters, digits, underscore) first, and only then check whether the resulting string matches a reserved keyword.

This is why keywords are just a lookup table checked after the greedy scan, not a separate code path — print, printer, and printed all go through identical scanning logic; only the final table lookup differs.

Escape Sequences in Strings

Real string literals support escapes: \n for newline, \t for tab, \\ for a literal backslash, \" for an embedded quote. The scanner must distinguish the lexeme (what's literally in the source, backslashes and all) from the literal value (what the string means after escapes are resolved):

• Source text: "line1\nline2"
• Lexeme: "line1\nline2" (16 characters, backslash-n as two chars)
• Literal value: line1 + newline + line2 (an actual newline character)

A simple approach is to do the replacements after the closing quote is found, being careful to replace \\ in a way that doesn't corrupt a \n that came from an original literal backslash-n... in practice, scan character-by-character and build the decoded string incrementally rather than doing blind str.replace calls, which can double-decode.

Error Recovery: Don't Stop at the First Mistake

A production scanner doesn't crash or halt on the first bad character — it reports and keeps going, so a single compile pass can surface every problem in a file at once. Two errors you must handle:

• Unterminated string — you hit end-of-input while still inside a "...". Report [line N] Error: Unterminated string. using the line the string started on, and stop trying to add that broken token, then continue scanning after it.
• Unexpected character — something like @ or # that isn't part of any valid token. Report [line N] Error: Unexpected character: @ and simply skip that one character, then resume scanning normally.

Crucially, errors go to a separate error channel (stderr, or an error list), never into the token stream — a malformed string shouldn't produce a phantom STRING token, and one bad character shouldn't derail the rest of the file. This separation of "valid tokens produced so far" from "errors accumulated so far" is a pattern you'll reuse in the parser and beyond: always try to make forward progress and collect diagnostics rather than aborting.