Here-Documents

After this lesson your shell reads a script line by line and, whenever a command line contains <<WORD, gathers the lines that follow it as that command's input, stopping at the delimiter. It records whether the body will be expanded and whether leading tabs are stripped.

The body is not on the command line

A here-document lets a script embed a block of text as the standard input of one command:

cat <<EOF
first line
second line
EOF

The command line ends at the end of its own line. The body is read from the same input stream, starting with the next line, up to a line that is exactly the delimiter. This is why the tokenizer never sees the body: the shell reads the command line, tokenizes it, notices a << operator, and only then goes back to the input for more lines. Whatever a body contains, quotes, pipes, a stray ', is data, not syntax. A body line such as rm -rf / is only text.

Finding the delimiter

The word after << or <<- is the delimiter, and the tokenizer has already removed its quotes. What survives is a flag: if any part of the delimiter word was quoted ('EOF', "EOF", \EOF, even E"O"F), the body is literal, with no $ expansion of any kind. If it was written bare, a later stage will expand variables and command substitutions in the body. For this lesson you only record the flag.

A line matches the delimiter only if it is the whole line. EOF with a trailing space, or EOF with leading spaces, is body text.

<<- and tabs

With the dash form the shell strips leading tab characters from every body line and from the line it compares against the delimiter. That lets a script indent a here-document inside an if and still terminate it. Spaces are never stripped, and without the dash a tab-indented delimiter does not terminate the body.

More than one, and the end of input

A command line may hold several here-documents (diff <<A <<B). Their bodies follow the line one after the other, in the order the operators appear, each with its own delimiter. A here-document can sit anywhere on the line, including before a pipe:

cat <<EOF | sort

If the input ends before the delimiter, real shells warn and use what they have. Yours reports it instead.

A worked trace

Script lines: cat <<-STOP, a line with two leading tabs and the text x, a line with a single tab then STOP, then ls. The first line has a heredoc with delimiter STOP, not quoted, with tab stripping. The next line becomes the body line x once the tabs are removed. The line after it strips to STOP, which equals the delimiter, so the body ends. ls is a new command.

Common mistakes

• Running the tokenizer over body lines. A body line with an unbalanced quote must not produce an error.
• Matching the delimiter against the raw line with <<-. The tabs must be stripped before comparing.
• Treating << inside quotes as a heredoc. A quoted << is an ordinary word.
• Stripping spaces as well as tabs, or stripping tabs when the operator was plain <<.

Your exercise

Implement the script walker. The starter reads every input line, skips blank lines and prints each command line. You add the heredoc detection and the body reading. The most likely mistake is forgetting that a quoted delimiter and its unquoted twin end the body in the same place and differ only in the expand flag.