Concurrent Connections

The loop you have been building (accept, parse, handle, respond) serves exactly one client at a time. That is fine until a handler blocks. Suppose each request spends 50 ms waiting on a database: your server handles 20 requests per second and the CPU is idle for 99% of that time. The bottleneck is not compute, it is waiting, and every concurrency model in this lesson is a different answer to one question: what does the server do while a connection waits?

Thread per connection

Spawn an OS thread on each accept() and let it block as much as it likes; the kernel scheduler runs the others. Each thread reads like the single-connection code you already wrote.

The cost is per-thread memory. A default stack reservation is measured in megabytes, so a few thousand simultaneous connections is the practical ceiling before the box is out of RAM. Fine for an internal service, wrong for the open internet, where most connections are idle most of the time.

Thread (or process) pool

Fix the worker count in advance and feed the workers from a queue of accepted connections. Memory is now bounded by a number you chose rather than by how many clients showed up: a load spike becomes a longer queue instead of an out-of-memory kill. Apache's prefork and worker MPMs and most Java servlet containers work this way.

The queue is where the tuning lives:

• Too few workers and the queue grows; latency climbs although the CPU is idle, because requests wait for a worker rather than for work.
• Too many workers and you are back to the memory and context-switch problems.
• An unbounded queue is the subtle failure. Under overload it accepts everything, and every request eventually completes after the client gave up 30 seconds ago. A bounded queue that answers 503 Service Unavailable when it is full sheds load honestly. That is why the depth is a deliberate configuration value and not infinity.

Event loop

One thread, non-blocking sockets, and a readiness API (epoll, kqueue, IOCP) that answers "which of these ten thousand sockets can I move right now?". Each connection becomes an explicit state machine (reading headers, reading body, writing response) and the loop advances whichever ones are ready. nginx and Node.js are built this way.

The tax is inversion of control: no call stack per request, so the state must be stored somewhere you can pick back up. And the loop is cooperative: one blocking call anywhere stalls every connection in the process. A synchronous file read, a DNS lookup or a CPU-heavy handler freezes ten thousand clients at once.

The machine under all three

Strip the models down and the same machine remains: a fixed number of workers, a FIFO queue with a limit, and time passing. That is what you build, as a simulation driven by commands.

ARRIVE c  ->  a worker is free?      yes: it starts c now
              else the queue has room?  yes: c joins the back of the queue
              else                        c is rejected with 503
TICK      ->  1. every busy worker's remaining time drops by 1
              2. every worker that reached 0 finishes (DONE)
              3. free workers pull from the queue (STARTED)

The order inside a tick is the whole lesson, and it is the bug real dispatchers ship. Completions are processed before dispatch, because a worker that finishes in this tick is available in this tick. Dispatch first and a queued connection waits an extra tick for a worker that was already free: the pool has capacity and still shows mysterious latency. Within step 2, workers finish in the order they started.

The contract

Commands, one per line: WORKERS n and optionally QUEUE n (the queue limit, unlimited if absent) come first; then ARRIVE id ms, TICK or TICK k (k ticks in a row), and STATUS.

Output, in the order things happen: STARTED id, DONE id, REJECTED id 503, and for STATUS the line free=F busy=B queued=Q done=D rejected=R. A job needs ms ticks of service.

Common mistakes

• Dispatching before draining, which leaves a free worker idle for a tick.
• Letting a rejected job enter the queue anyway, or counting it as done.
• Treating QUEUE 0 as unlimited. Zero means no waiting room at all.
• Reporting finished jobs in id order instead of start order.

Your exercise

The starter reads the commands. You keep the busy workers and the queue, and apply a tick exactly as described.