Concurrent Connections

The loop you have been building — accept, parse, handle, respond — serves exactly one client at a time. That is fine until a handler blocks. Suppose each request spends 50 ms waiting on a database: your server handles 20 requests per second and the CPU is idle for 99% of that time. The bottleneck is not compute, it is waiting, and every concurrency model in this lesson is a different answer to one question: what does the server do while a connection waits?

Thread per connection

Spawn an OS thread on each accept() and let it block as much as it likes; the kernel scheduler runs the others. It is the easiest model to reason about because each thread reads like the single-connection code you already wrote — the request's whole life is one call stack.

The cost is per-thread memory. A default stack reservation is measured in megabytes, so a few thousand simultaneous connections is the practical ceiling before the box is out of RAM. Above that, context switching starts to dominate: the scheduler spends real time moving between threads that are all blocked anyway. Fine for an internal service with dozens of clients; wrong for anything facing the open internet, where most connections are idle most of the time.

Thread (or process) pool

Fix the worker count in advance and feed the workers from a queue of accepted connections. Memory is now bounded by a number you chose rather than by how many clients showed up, which is the entire point: a load spike turns into a longer queue instead of an out-of-memory kill. Apache's prefork and worker MPMs and most Java servlet containers work this way.

The queue is where the tuning lives, and it is genuinely a trade-off rather than a knob to max out:

• Too few workers and the queue grows; latency climbs even though the CPU is idle, because requests are waiting for a worker rather than for work.
• Too many workers and you are back to the memory and context-switch problems of thread-per-connection.
• An unbounded queue is the subtle failure. Under overload it accepts everything and every request eventually completes — after the client gave up 30 seconds ago. Work you do for a disconnected client is pure waste. A bounded queue that rejects with 503 Service Unavailable when full sheds load honestly, and that is why the depth is a deliberate configuration value and not infinity.

Event loop

One thread, non-blocking sockets, and a readiness API (epoll, kqueue, IOCP) that answers "which of these ten thousand sockets can I move right now?". Each connection becomes an explicit state machine — reading headers, reading body, writing response — and the loop advances whichever ones are ready. Memory per connection drops from a thread stack to a small struct, so ~100k concurrent connections on one process is ordinary. nginx and Node.js are built this way.

The tax is inversion of control: you no longer have a call stack per request, so the request's state must be stored somewhere you can pick back up. And the loop is cooperative — one blocking call anywhere stalls every connection in the process. A synchronous file read, a DNS lookup, a tight CPU loop in a handler: any of them freezes ten thousand clients at once. Go blurs the distinction with a hybrid, giving you blocking-style code on goroutines that its runtime multiplexes over an event loop underneath; that is why net/http feels like thread-per-connection and scales like an event loop.

What the exercise models

Strip the three models down and the same machine is underneath all of them: a fixed number of workers, a FIFO queue, and time passing. That machine is what you are about to build.

ARRIVE c  ->  free worker?  yes: worker takes c, starts counting down
                            no:  c joins the back of the queue
TICK      ->  every busy worker's remaining time drops by 1
              any that reach 0 finish, and free workers pull from the queue

The ordering inside a TICK is the whole lesson, and it is the bug real dispatchers actually ship. Completions must be processed before dispatch, because a worker that finishes this tick is available this tick — drain first, then dispatch, and you get the queue moving with no idle worker. Dispatch first and a connection waits a full tick for a worker that was already free, which is exactly how a pool with plenty of capacity develops mysterious latency.

Your exercise: Worker Pool Simulation

Commands in, an event log out: STARTED when a worker picks up a connection, DONE when it finishes, and a STATUS line reporting free, busy, queued and done counts. The ladder runs from one worker and one connection, through more arrivals than workers so the queue actually fills, to a long job holding a single worker while two short ones wait their turn behind it. Get the drain-then-dispatch order right and every case falls out; get it backwards and the counts stay correct while the event order quietly lies.