How Scanners Work

Why a Scanner Comes First

Every interpreter or compiler starts the same way: raw source code is just a string of characters. Before you can understand var x = 42; as a program, you need to understand it as a sequence of meaningful chunks — var, x, =, 42, ;. This chunking process is called lexical analysis, or scanning, and the chunks it produces are called tokens.

A token is more than a substring. It typically carries:

• type — what kind of thing this is (NUMBER, IDENTIFIER, PLUS, keyword like VAR, etc.)
• lexeme — the exact source text that produced it ("42", "x", "+")
• literal — for literals, the actual decoded value (the string "42" becomes the number 42.0)
• line — where it appeared, for error messages

The Scanning Loop

The core of a scanner is a single loop over the source string with a cursor (an index) that only ever moves forward:

At each position you're asking: "does a token start here, and if so, what kind?" A few rules make this tractable:

• Whitespace and newlines don't produce tokens — you skip them, but newlines still increment a line counter so later errors can point at the right line.
• Comments (// ... to end of line) are skipped entirely — never turned into tokens.
• Single-character tokens like (, ), ,, +, -, * are the easy case: look up the character in a table and emit immediately.
• Punctuation that might be two characters — !, =, <, > — requires a one-character lookahead: if ! is followed by =, that's a single != token, not two separate tokens.

The One-Character Lookahead Pattern

This is the recurring trick in scanning: peek at the next character without consuming it yet, decide, then consume the right number of characters.

Get this wrong and <= scans as LESS followed by EQUAL — which will silently break your parser's precedence logic much later, far from the real bug. Always test two-character operators adjacent to other tokens (x<=5, no spaces) since that's the case naive scanners miss.

What You're Building

Your scanner is a standalone pass: it reads the entire source text and produces a flat list of tokens, ending with a synthetic EOF token so the parser always knows when input is exhausted. Nothing here understands meaning yet — if, else, and x are all just tokens at this stage. That understanding comes later, in parsing. For this exercise, focus on getting single-character tokens, two-character operators, and skipping whitespace/comments exactly right, and always terminate with the EOF line.