Executing a Pipeline

cat log | grep 404 | wc -l — three programs, running simultaneously, bytes flowing between them with no temp files and no orchestrator shoveling data. The shell's job is to build that machine and then get out of the way. This lesson is the wiring diagram.

The pipe primitive

pipe() asks the kernel for a one-way byte channel and hands back two file descriptors: a read end and a write end. Bytes written to one appear at the other, with the kernel buffering in between (and — crucial for flow control — blocking writers when the buffer fills and readers when it's empty; that back-pressure is why cat hugefile | head -1 doesn't read the whole huge file).

The trick that makes pipes compose with programs: file descriptors survive fork and exec. Set up fd 1 (stdout) to be a pipe's write end before exec, and the program — which knows nothing about pipes — writes to "stdout" as always, and the bytes enter the pipe. Programs stay innocent; the shell does the plumbing. (This is the fork/exec gap from last lesson, cashing its check.)

The wiring plan

For an N-command pipeline: N−1 pipes, N forks, and per child, the right dup'ing:

for each command i in pipeline:
    child i:
        if not first:  stdin  ← pipe[i-1].read
        if not last:   stdout → pipe[i].write
        close ALL pipe fds (both ends, all pipes)
        exec command i
parent: close all pipe fds; wait for all children

The dup2(fd, 0) / dup2(fd, 1) calls do the redirecting — "make fd 0 be a copy of this pipe end." Middle commands get both treatments; the first and last keep the terminal at one end. All children are forked and running concurrently — the shell does not run stage 1 to completion and then stage 2; the pipes' back-pressure coordinates everything.

The bug every implementation has once

Unclosed pipe ends hang pipelines. A reader gets EOF only when every write end of its pipe is closed. Fork N children and each inherits every pipe fd — so if child 3 (or the parent!) still holds pipe 1's write end open, then even after child 1 exits, child 2 never sees EOF and waits forever. grep finishes, wc hangs, and you learn the rule everyone learns the hard way: every process closes every pipe fd it isn't actively using — and the parent closes all of them after forking. When your pipeline hangs (it will, once), audit the closes first.

Status of a pipeline

The parent waits for all N children. Whose exit status is "the pipeline's"? Default POSIX rule: the last command's — false | true succeeds (!), which surprises people and is exactly the behavior bash's pipefail option (Script Hygiene lesson, other course) exists to change. Your shell implements the default; knowing both is what makes the hygiene flag meaningful.

Your exercise: Pipeline I/O Plan

Given a parsed pipeline, emit the wiring plan — which fds each process dups where, what gets closed, per the spec's format. It's the diagram above made mechanical: first/middle/last classification, one dup rule each side, closes for everything else. The graded traps are the two off-by-ones (command i uses pipe i−1 for stdin and pipe i for stdout) and the completeness of the close list — the exercise checks it precisely because real shells that miss one close don't fail loudly; they hang quietly, which is the crueler bug.