String & Number Literals

Literal tokens are special because they carry a runtime value in addition to their lexeme. Parsing them correctly requires careful handling of edge cases.

Number Literals

Our language supports integers and floating-point numbers:

42      → NUMBER, literal = 42.0
3.14    → NUMBER, literal = 3.14
0       → NUMBER, literal = 0.0

The scanning algorithm: when you see a digit, keep consuming digits. If you encounter a . followed by more digits, consume those too. But .5 is NOT a valid number — a leading dot is just a dot token followed by a number.

Important: we store all numbers as double-precision floating-point internally, just like JavaScript and Lua. This simplifies the value representation at the cost of integer precision beyond 2^53.

String Literals

Strings are delimited by double quotes:

"hello world"  → STRING, literal = hello world
"line1\nline2" → STRING, literal = line1<newline>line2

The scanner must handle:

• Escape sequences: \n (newline), \t (tab), \\ (backslash), \" (quote)
• Multi-line strings: strings can span multiple lines — track line count
• Unterminated strings: reaching EOF inside a string is an error

In Crafting Interpreters, Nystrom points out that real language implementations often handle string interning at the scanner level — storing each unique string once and sharing references. V8's scanner does this aggressively because JavaScript programs create many identical strings.

Comments

Comments are lexically similar to division:

/ → SLASH token (division operator)
// → comment, skip to end of line
/* → block comment, skip to closing */

Block comments can nest in some languages (Rust allows /* /* nested */ */) but we will keep ours non-nesting for simplicity.

Error Reporting

Good error messages include the line number and a description:

[line 5] Error: Unterminated string.
[line 12] Error: Unexpected character: @

Track the current line by incrementing a counter every time you see a newline character, both in whitespace and inside multi-line strings.

Your Task

Handle all literal types, comments (line and block), escape sequences, and error reporting with line numbers.