Concurrent Connections

A single-threaded accept -> handle -> respond loop blocks on each request. With realistic latency, a server handles less than 100 req/s.

Three approaches to concurrency:

Thread per connection — spawn an OS thread on accept. Simple, scales to a few thousand concurrent connections. Memory overhead (~1 MB stack each) caps the count.

Thread pool — fixed-size worker pool pulling connections from a queue. Bounded memory; queue depth becomes a tuning knob. Used by Apache, Tomcat.

Event loop / async — single thread + non-blocking I/O. Each connection is a state machine. Scales to ~100k connections per process. Used by nginx, node.js, and most modern servers (Go's runtime is a hybrid).

For this lesson we model the queue and dispatcher: connections arrive, workers pick them up, completion frees the worker. This is what every implementation does at heart.