Tokenizing JSON

Every serious parser is layered. The lexer (tokenizer) turns a character stream into a token stream; the parser turns the token stream into a value. You could parse JSON straight from characters — but then every grammar rule would drown in "skip whitespace, check for a quote, scan digits" noise. Splitting the job means the parser reasons about token kinds ("is the next token a STRING?") while the lexer owns the messy character-level rules. CPython's json, Java's Jackson, and Rust's serde_json are all built this way, and so is the parser you are about to write.

The token vocabulary

Your lexer emits seven kinds:

Notice what is NOT here: there is no OBJECT or ARRAY token. Objects and arrays are parser concepts assembled out of PUNCT tokens — the lexer never knows it is inside one. The grammar the parser will later build on (RFC 8259):

value  = object | array | string | number | "true" | "false" | "null"
object = "{" [ member *( "," member ) ] "}"
member = string ":" value
array  = "[" [ value *( "," value ) ] "]"

Whitespace: between tokens, never inside

RFC 8259 allows exactly four whitespace characters — space, tab, line feed, carriage return — and only BETWEEN tokens. [ 1 , 2 ] and [1,2] produce the same token stream. But whitespace never splits a token: a raw line break inside a string is illegal, and 1 2 is two NUMBER tokens, not one.

The scanning idiom

One index, one dispatch on the current character:

Two rules keep it honest:

• Numbers are scanned maximally. Consume the optional -, the digits, an optional . plus digits, and an optional exponent: e or E, optional +/-, digits. Emit the raw text — converting to int/float is a later lesson's job. JSON's number grammar is stricter than Python's float() (01, +5, .5, 5. are all invalid JSON), so keeping the text preserves your ability to validate it properly.
• Keywords match exactly, lowercase only. true, false, null. True dies at the T with an unexpected-character error — which is correct behavior.

Your exercise

Build the tokenizer. The starter's number scanner handles digits and a fraction but not exponents — run visible test 1.5e10: the grader expects exactly two lines, NUMBER 1.5e10 then EOF, but the shipped scanner stops after 1.5, hits the e, falls through every branch, and prints ERR unexpected character 'e' at position 3. Extend it to consume e/E, an optional sign, and digits. Second trap: STRING values are printed decoded — a hidden test feeds "line1\nline2" and expects the token's value to span two physical lines (STRING line1 on one line, line2 on the next); the starter's quote-to-quote scan leaves the backslash in. Finally, keep the error format character-for-character: input @ must print ERR unexpected character '@' at position 0, quotes included — the grader compares the exact string. Keep the blank line the starter prints after each document's EOF.