Tokenization

The lexer converts characters → tokens. Token types:

• KEYWORD (if, def, for, class, ...)
• IDENT (foo, _bar, x1)
• NUMBER (42, 3.14, 0xff)
• STRING ("hi", 'hi', """...""")
• OP (+, ==, ->, :=)
• INDENT / DEDENT (Python's indentation)
• NEWLINE
• COMMENT (#...)

For real linters: Python ships with tokenize module — don't reinvent. But understanding it matters for token-based rules (e.g. "no print() statement", "no tabs after spaces").

Some rules work at TOKEN level (faster, simpler):

• Line length (count characters before NEWLINE).
• Mixed tabs/spaces (look at indent token bytes).
• Magic numbers (NUMBER tokens not in 0,1,-1,...).

Other rules need full AST (semantic understanding).

The first 80% of lint rules are token-level. AST is needed for the powerful 20% (data flow, scope analysis).