The Scanner — Breaking Source into Tokens

Every programming language implementation starts with a scanner (also called a lexer or tokenizer). The scanner reads raw source code character-by-character and groups those characters into meaningful chunks called tokens.

Why Scanning Matters

Consider this line of code:

var x = 42;

To you, that is five distinct elements: a keyword, a name, an equals sign, a number, and a semicolon. But to the computer, it is just a stream of 13 characters. The scanner bridges this gap by producing:

[VAR "var"] [IDENTIFIER "x"] [EQUAL "="] [NUMBER "42"] [SEMICOLON ";"] [EOF]

The Scanning Loop

A scanner works by maintaining a current position in the source string and advancing through it:

1. Skip whitespace and comments
2. Read the next character
3. Based on that character, decide which token type we are scanning
4. Consume additional characters as needed (e.g., != is two characters)
5. Emit the token

This is essentially a finite state machine. In Crafting Interpreters, Robert Nystrom calls this the "heart of the scanner" — a single switch on the current character that dispatches to specific scanning routines.

Handling Different Character Classes

How Real Languages Do It

Python's tokenizer (tokenize.py) is more complex because Python is whitespace-sensitive — it must track indentation levels and emit INDENT/DEDENT tokens. Lua's scanner (llex.c) is very similar to what we will build: a hand-written loop with a switch statement. V8's scanner for JavaScript handles Unicode identifiers and template literals, adding significant complexity.

Your Task

Build a scanner that reads Lox source code from stdin and outputs tokens in TYPE lexeme literal format. This scanner will be the foundation for the parser, compiler, and VM you build throughout this course.