Command Substitution: $(...)

After this lesson your shell can run a command in the middle of another one: echo today is $(date) becomes echo today is Fri, because the text $(date) is replaced by what the inner command printed. Substitutions nest, they survive parentheses inside quotes, and trailing newlines vanish exactly as in a real shell.

What the shell does

Command substitution is part of the expansion stage (POSIX 2.6.3). When the shell meets $(, it:

1. finds the ) that closes it,
2. runs the text between as a command, capturing its standard output through a pipe,
3. removes all trailing newline characters from the captured text,
4. splices the rest into the word where the $(...) stood.

Step 3 is why $(printf 'a\n\n\n') is just a, while a newline inside the output, as in a\nb, stays. Programs print a final newline so that terminals look right; the shell removes it so you can build words from output.

Finding the end

Counting parentheses is not enough, because quoted ones do not count:

echo $(echo ")")       the first ) is inside quotes
echo $(echo $(echo a)) two levels: the middle ) closes the inner one

Scan forward from the $( with a depth of 1. A ( outside quotes adds one, a ) outside quotes subtracts one, and when the depth reaches zero you have the end. Inside single quotes nothing is special until the next single quote; inside double quotes only the closing double quote ends the region; a backslash skips the next character. If you run off the end of the line the substitution is unterminated, which is a syntax error.

Nesting and order

The text inside is a command line like any other, so it can contain variables and further substitutions. Expand it first, then run the result. That is why $(echo $(upper hi)) works from the inside out: the inner upper hi runs, its output HI is spliced into the outer text, and only then does the outer echo HI run.

Real programs are not available in the grader, so this lesson uses five toy commands: echo [-n] words (words joined by one space, plus a newline unless -n), printf text (text with \n, \t and \\ turned into the characters they name, and no newline added), upper words, lower words and len words (the number of characters). Any other command, or an empty one, prints nothing. Words are split with the usual quoting: single quotes and double quotes group, a backslash escapes.

What stays untouched

A single-quoted region is copied as typed, so '$(echo no)' is not run. A backslash protects the next character, so \$(echo no) is not run either. Inside double quotes the substitution still runs. A lone $ that does not start a name, ( or { is just a dollar sign. The quotes themselves stay in the text in this lesson: it substitutes, it does not remove quotes.

Limits of this lesson

A real shell would now split the result into words at spaces and expand wildcards in it. That comes later. Here the output of EXPAND is the substituted text, newlines included.

Common mistakes

• Matching the first ) instead of the right one, which breaks nesting and quoted parentheses.
• Stripping only one trailing newline. The rule is all of them.
• Expanding variables after running the command, so the command sees a literal $NAME.
• Running substitutions that sit inside single quotes.

Your exercise

The starter reads SET and EXPAND lines and prints the result; you write the function that expands one text. Most likely mistake: forgetting that the text inside $( ) must itself be expanded before it is run.