cgroups: CPU Limits

Memory has one obvious question — how many bytes? CPU has three, because "half a CPU" can mean three different things, and Docker exposes a flag for each. Getting them confused is how a service ends up mysteriously slow on a host that shows 60% idle.

Shares: a relative weight

cpu.shares = 1024        # the default

Two groups with 1024 each split a contended core 50/50. At 2048 against 1024 it is 2/3 against 1/3. With no competitor, shares do nothing at all — a group with cpu.shares=2 uses the whole machine if nothing else wants it. Shares express priority, never a cap, and a load test on an idle host will never reveal them.

Quota: an absolute cap

cpu.cfs_period_us = 100000    # a 100 ms accounting period
cpu.cfs_quota_us  =  50000    # 50 ms of CPU time allowed per period

That is 0.5 CPUs, enforced whether or not anyone else is running. Quota may exceed the period: 200000 over 100000 means two cores' worth, which is how --cpus 2 is implemented. In cgroup v2 the same pair is one file, cpu.max, written as 50000 100000.

The failure mode here is the one people actually hit. Throttling is invisible to the process — no signal, no error, no log line; the threads are simply not scheduled until the period rolls over. A latency-sensitive service with a low quota shows p99 spikes shaped like the period, and the only direct evidence is throttled_time climbing in cpu.stat:

nr_periods 4211
nr_throttled 1180        # throttled in 28% of periods
throttled_time 9223114000

Cpuset: which cores

cpuset.cpus = 0-3        # run only on these cores
cpuset.mems = 0          # allocate only from this NUMA node

A different kind of limit — placement, not bandwidth. It buys cache locality and NUMA-local memory for workloads that care, and it is the flag people reach for last and should sometimes reach for first.

Docker's spellings: --cpu-shares 512, --cpus 0.5 (quota + period), --cpuset-cpus 0,2.

Why the exercise is a simulator

The scheduler is not something a sandboxed submission can drive, and the useful part is not the scheduling anyway — it is the accounting: a budget per period, consumed by running, refilled on the period boundary. That loop is what cpu.cfs_quota_us really is.

Your exercise: CPU Quota Tracker

The rule worth stating out loud, because it is where a plausible implementation goes wrong: when a RUN exceeds the budget, used is clamped to the quota, not left where it was and not allowed past it. The group really did burn the rest of its budget before being stopped; it just did not get more. And throttled is a property of the current period only — one TICK and the group is clean, however hard it was throttled a moment ago.

Reference: kernel Documentation/scheduler/sched-bwc.rst; cgroup v2 cpu.max.