Parsing Literal Values

Numbers: Simpler Than They Look, Until the Edge Cases

Number scanning seems trivial — consume digits — but two edge cases trip people up:

Trailing dot without a fractional part isn't a decimal. .5 is not a valid number literal in Lox-style grammars; a leading . with no digit before it is just the DOT token followed by a NUMBER. This matters because .5 could plausibly be intended as a method-call-like syntax (object.5 isn't meaningful, but the grammar still parses . as its own token for consistency with member access like a.b).

The fix is a two-character lookahead before committing to consuming a decimal point:

Without the second lookahead (checking the digit after the dot), 1. alone would incorrectly consume the trailing dot as part of the number, and 1.foo() (method call on a number-like expression) would scan wrong.

Every number, whether 42 or 3.14, becomes a NUMBER token whose literal value is a float (42 → 42.0) — the language has one numeric type at this stage, so there's no ambiguity between "integer" and "float" tokens to worry about yet.

Multi-Line Strings

Unlike some languages, a string literal here is allowed to span multiple physical lines — the scanner just keeps consuming characters (including embedded newlines) until it finds the closing ". The important detail: increment the line counter every time you pass a \n inside the string, or every subsequent error message will report the wrong line for the rest of the file.

Two Kinds of Comments

You've already handled // line comments — they run to end of line. Block comments (/* ... */) are a different beast:

• They can span multiple lines (track line the same way as multi-line strings).
• They end at the first */, not the last — comments do not nest by default in most C-like grammars, so /* a /* b */ c */ ends after b */, leaving c */ as regular (broken) code. Some languages choose to support nesting, but the simplest and most common design does not.
• An unterminated block comment (reaches EOF while still inside /* ... */) is an error, just like an unterminated string — report it using the line the comment started on.

Why This Lesson Matters for Everything Downstream

Every token your scanner emits from here forward feeds directly into the parser. A scanner bug — an off-by-one in line tracking, a mis-scanned .5, an unterminated comment silently eating the rest of the file — shows up as a confusing parse error far from its real cause. Get literals and comments airtight now, because debugging them will only get harder once there's a parser and interpreter layered on top.