Token Types — Keywords, Literals & Operators

A well-designed set of token types is the contract between your scanner and parser. Every token carries a type, a lexeme (the raw text), and optionally a literal value (the parsed representation).

Categories of Tokens

Single-Character Tokens

These are unambiguous — one character maps to exactly one token:

(  )  {  }  ,  .  -  +  ;  *

One-or-Two Character Tokens

These require a lookahead of one character to disambiguate:

!  vs  !=
=  vs  ==
<  vs  <=
>  vs  >=

The scanner peeks at the next character. If it matches, consume both and emit the two-character token. Otherwise, emit the single-character token. This is called a maximal munch strategy — always consume the longest possible token.

Keywords vs Identifiers

Keywords like var, fun, class, if, while look syntactically identical to identifiers. The standard approach: scan the entire word as an IDENTIFIER, then check it against a keyword table. If it matches, change the token type.

In Crafting Interpreters, Nystrom uses a trie-based approach for keyword lookup in the C implementation (clox), which avoids hash table overhead. For a tree-walk interpreter, a simple hash set works well.

Literals

Literal tokens carry a runtime value:

• NUMBER: 42 has lexeme "42" and literal 42.0
• STRING: "hello" has lexeme "\"hello\"" and literal "hello" (quotes stripped)

The Token Data Structure

Token {
    type:    TokenType   // e.g., IDENTIFIER, NUMBER, PLUS
    lexeme:  String      // the raw text: "foo", "42", "+"
    literal: Value?      // parsed value: 42.0, "hello", or null
    line:    Int          // source line for error reporting
}

Line tracking is critical for error messages. Python reports SyntaxError: invalid syntax with a line number and caret pointing to the exact character — you should aim for similar quality.

How Languages Differ

Lua has very few token types (under 30) because its syntax is minimal. JavaScript has far more because of features like template literals, regex literals, and optional chaining (?.). Our language, Lox, sits in between with roughly 40 token types — enough to be interesting without being overwhelming.