Tokenizing a Command Line

Every shell begins here: a human types a line of text, and before anything can run, that text must be split into tokens — the words, operators, and quoted chunks the rest of the shell operates on. Get tokenizing wrong and every later stage inherits the damage; get it right and pipelines, redirects, and expansion all snap onto a clean foundation.

Why not just split on spaces?

Because of quotes. The command line's oldest bargain is that whitespace separates arguments unless the user says otherwise:

echo hello world          → [echo] [hello] [world]        3 args
echo "hello world"        → [echo] [hello world]          2 args — ONE token
echo 'it is'" here"       → [echo] [it is here]           adjacent quotes JOIN

A tokenizer that naively splits on spaces turns "hello world" into two arguments — precisely the bug you learned to avoid triggering in the bash course by quoting variables. Now you're on the other side of the counter: you're the one who must honor the quotes.

The state machine

Tokenization is a character-by-character walk with a small amount of state — the canonical introduction to state machines, which is why every systems course does it:

• Default state: whitespace ends the current token (if one is open); an operator character (|, <, >) ends the token and emits an operator token; a quote switches state; anything else appends to the current token.
• In single quotes: everything is literal until the closing '. No escapes, no specials — bash's actual rule, and gloriously simple to implement.
• In double quotes: literal until the closing ", except (in later lessons) $ stays meaningful. For tokenizing, the key point is that whitespace inside doesn't split.

Two subtleties that separate working tokenizers from almost-working ones:

1. Quotes join, not create. it' is' is ONE token (it is) — the quoted region glues onto the token in progress. Your tokenizer tracks "am I inside a token?" separately from "am I inside quotes?" An empty quoted string "" still creates a token (an empty argument — legal and occasionally useful).
2. Operators end words without needing spaces. ls>out is three tokens: ls, >, out. Users type it both ways; the grammar treats them identically.

And the error case your tests will include: an unclosed quote. A real interactive shell prompts for more input (> continuation); a batch tokenizer reports the error. Either way, "end of input while in quote state" must be detected, not ignored.

Tokens are not yet meaning

Resist the urge to interpret while tokenizing. > becomes an operator token — what it redirects is the parser's job (next lessons). $HOME stays literal text for now — expansion is its own stage. The Unix shell's actual pipeline is a textbook compiler front-end: tokenize → parse → expand → execute, each stage simple because it trusts the previous one. You're building stage one, and building it pure.

Your exercise: Tokenize Command Lines

Lines in, token lists out, per the spec's format. Work the gauntlet in this order: bare words → double quotes with spaces → single quotes → adjacent-quote joining → operators with and without surrounding spaces → empty quotes → unclosed quote error. That's the complete behavior space of a POSIX-style tokenizer's core, and each case is a line or two in the state machine — the whole thing fits in a screen, which is the point: parsing feels like magic until you've written one, and then it's a switch statement.