The Lexer

The lexer (or scanner, tokenizer) reads source characters and emits a stream of tokens.

Token types in C:

• Keywords: int, char, if, else, while, return, etc.
• Identifiers: variable/function names.
• Literals: integers (123, 0x7F, 017), floats (3.14), strings ("hello"), chars ('A').
• Operators: + - * / % == != < > <= >= && || ! ~ & | ^ << >> = += -= ...
• Punctuation: ( ) { } [ ] ; , . -> ?:

Tokens drop whitespace + comments — they're not semantically meaningful for parsing (except as separators).

Edge cases:

• Maximal munch: == should be ONE token, not two =. Always try longest match.
• Hex/octal/binary: 0x1F, 017 (octal is a LEADING ZERO, not 0o), 0b1010 (C23).
• String escapes: "\n", "\\", "\"".
• Numeric suffixes: 42L, 3.14f, 1e10.

Real lexers may be hand-written (chibicc, Clang) or generated (lex/flex). Hand-written is more flexible; generated is more declarative.