Layered Images

An image is not a tarball of a filesystem. It is a stack of tarballs, each holding only what changed, and the stack is the reason docker pull on your second image of the day takes two seconds instead of two minutes.

One instruction, one layer

FROM ubuntu:22.04                    # several layers, already on your disk
RUN apt-get update                   # + a layer: the package indexes
RUN apt-get install -y nginx         # + a layer: nginx and its dependencies
COPY ./site /var/www/html            # + a layer: your files
CMD ["nginx", "-g", "daemon off;"]   # no layer — metadata only

RUN, COPY and ADD produce layers; CMD, ENV, EXPOSE and friends only write fields into the image config. Each layer is a tar of the differences its instruction made, addressed by the SHA-256 of its own content.

Content addressing is what makes sharing automatic rather than clever. Two images built FROM ubuntu:22.04 do not have similar base layers, they have the same base layers — identical bytes, identical digest, stored once and pulled once. That is the entire argument for small bases: alpine is about 5 MB against Ubuntu's 80, so the first pull is cheaper and every image that shares it is free after the first.

Two consequences people learn the hard way

Order your Dockerfile by rate of change. A layer's cache is invalidated when its input changes, and every layer below an invalidated one is rebuilt too. COPY . . before RUN npm install means every source edit reinstalls your dependencies; the other order means it almost never does.

Deleting in a later layer does not shrink the image. RUN rm secrets.pem adds a layer that hides the file; the layer that contains it is still in the stack, still shipped, still readable by anyone who pulls the image and unpacks it by hand. A secret in any layer is a secret in the image.

The stack at runtime

lowerdir=<base>:<apt indexes>:<nginx>   read-only, top-of-list first
upperdir=<this container's writes>      read-write
workdir=<scratch space overlayfs needs>
mount -t overlay overlay -o lowerdir=...,upperdir=...,workdir=... /merged

The container sees one directory, /merged. A read is served by the topmost layer that holds the path; a write always goes to the upper. That single rule — topmost wins — is what your exercise implements, and it is also why the same image can back a thousand containers without a thousand copies of nginx.

Why the exercise is a simulator

Mounting an overlay needs CAP_SYS_ADMIN. The lookup rule needs nothing but a stack and a map, and the lookup rule is the whole of what a union filesystem is.

Your exercise: Layer Lookup Order

Search the stack from the top down for LOOKUP, and for LIST walk it from the bottom up letting later layers overwrite the entry — the two directions must agree on the same answer for every path. A layer declared with no files still counts as a layer; it simply never wins. Nothing here deletes: hiding a lower file needs a whiteout, which is the next lesson but one.

Reference: kernel Documentation/filesystems/overlayfs.rst.