Tokenizing a Command Line

After this lesson your shell turns the text a user typed into the exact words and operators the next stage will parse, and it tells the user when a quote was never closed. Every later stage, from pipelines to redirects, reads what this one produces.

Stage one of four

A shell is a small compiler front end: tokenize, parse, expand, execute. The tokenizer sees only characters. It does not know that > means redirection or that $HOME is a variable; it only decides where one token ends and the next begins. POSIX calls this "Token Recognition" (Shell Command Language, section 2.3).

The hard part is quoting. The same bytes mean different things depending on which quote you are inside:

Inside double quotes any other backslash stays a backslash: "C:\temp" is the seven characters C:\temp, not a tab. Single quotes cannot contain a single quote, which is why 'it'\''s' exists.

Words are glued from pieces

A word is not one quoted thing. It is everything between separators, with quoted and unquoted pieces concatenated. Take a"b c"'d e'f: the walk meets a, then a double-quoted piece, then a single-quoted piece, then f, and no separator in between, so the result is one word, ab cd ef. Two flags carry this: "a word is open" and "I am inside a quote". An empty pair of quotes opens a word with no characters, so "" is a real empty argument.

Operators end words without spaces

Outside quotes some characters are operators and always end the word before them. Match the longest operator at each position: <<-, &&, ||, >>, <<, >&, <&, then the single characters |, &, ;, <, >. So a||b is a, ||, b, never two pipes. Inside quotes the same characters are ordinary: "a|b" is one word.

One more rule, and it has bitten real scripts. A word made only of unquoted digits that sits directly against a redirection operator is not an argument but an IO number: it names the file descriptor being redirected. echo 2>err redirects descriptor 2 and echoes nothing, while echo 2 >err echoes a 2 into the file. Your tokenizer must emit those two differently.

Comments

An unquoted # at the start of a word begins a comment that runs to the end of the line. In the middle of a word it is just a character, so a#b stays whole.

Common mistakes

• Splitting on spaces first and handling quotes afterwards. "a b" has already been torn apart by then.
• Treating a backslash inside double quotes as an escape for everything. Only four characters are escapable there.
• Dropping "". An empty quoted string is a word and must survive.
• Forgetting that the error depends on state: reaching the end of the line while a quote is still open (or with a lone trailing backslash) is a syntax error, not a short word.

Your exercise

Read command lines from stdin and print the tokens of each. The starter has the loop, the operator table and the output format; it splits on spaces and nothing else. Replace that with the character walk. The most likely mistake is deciding about an IO number too early: you only know a digit word is one when you reach the operator after it.