Pipes and Redirection

By the end of this lesson you can send a program's output to a file or into another program, keep errors out of your results, and chain small tools into a pipeline that answers a question about a pile of text. This is the idea bash exists for: every program reads a stream in and writes streams out, and you can plug them into each other.

The three streams

Every process starts with three open channels:

• stdin (descriptor 0): data in
• stdout (descriptor 1): results out
• stderr (descriptor 2): errors and chatter out, kept separate on purpose so that problems never contaminate results

Redirection: streams to and from files

sort < unsorted.txt           # stdin comes from a file
ls > listing.txt              # stdout goes to a file (truncating it first!)
ls >> listing.txt             # append instead
ls /nope 2> errors.log        # stderr only, to a file
ls /nope 2> /dev/null         # stderr into the void: silence the complaints
make > build.log 2>&1         # both streams into one file

> file empties the file before the command runs, which is the classic self-inflicted wound: sort data > data leaves data empty before sort reads a byte. In 2>&1, &1 means "wherever stdout points right now", so it has to come after the > file part. /dev/null is a file that swallows everything written to it.

Pipes: streams into programs

A | B connects A's stdout to B's stdin. No temporary file is involved, and both programs run at the same time. Each stage is small and dumb; the pipeline is smart. The stages worth knowing cold:

grep PATTERN        keep lines that match (grep -E for extended patterns, -v to invert)
cut -d, -f2         pick a column: delimiter, then field number
sort                order lines (-r reverse, -n numeric, -u drop duplicates)
uniq -c             collapse ADJACENT identical lines and prefix each with its count
head -3 / tail -3   first / last lines
wc -l               count lines (-w words)
tr 'a-z' 'A-Z'      translate characters
awk '{print $2, $1}'  split each line on whitespace and print fields in any order

Nearly every "quick question about some data" has the same shape: select, normalise, group, rank. Here it is on a list of colors, one stage at a time:

$ cat colors.txt
red
blue
red
green
blue
red

$ sort < colors.txt | uniq -c
      2 blue
      1 green
      3 red

$ sort < colors.txt | uniq -c | sort -k1,1nr
      3 red
      2 blue
      1 green

uniq only collapses lines that sit next to each other, so the sort in front is not decoration: without it red would be counted three separate times. Everyone gets a wrong count from this exactly once. sort -k1,1nr sorts by the first field, numerically, largest first. A second key such as -k2,2 states the tie-break (alphabetical) explicitly instead of relying on sort's default.

uniq -c pads its counts with spaces. To get clean output, awk '{print $2 ": " $1}' prints the second field first, which puts the name before the number.

When you cannot tell which stage misbehaves, build the pipeline one stage at a time and look at the output of each.

One footnote that pays off later: each stage runs in its own subshell, so ... | while read x; do total=$((total + 1)); done loses total when the pipe ends. Feeding the loop with < file instead keeps the loop in the main shell.

Common mistakes

• uniq without sort. Equal lines that are not adjacent are counted separately.
• Sorting numbers as text. sort alone puts 10 before 9. Add -n.
• Redirecting to the file you are reading. sort data > data empties it; write to a new file and move it into place.
• Redirection order. cmd 2>&1 > file sends stderr to the old stdout (the terminal). Write cmd > file 2>&1.
• Forgetting that errors skip the pipe. Only stdout goes through |; stderr still reaches the screen unless you redirect it.

Your exercise

Summarise a log file. Several kinds of line appear in it and only some of them count. Build a single pipeline that keeps the right lines, reduces each to its level word, groups and counts the levels, ranks them, and prints each result in the requested format. Remember what uniq -c needs from its input, and decide what to do with ties.