Click run in a generative-AI tool and your request joins a line that every other user of that endpoint is already standing in. The scheduler treats each prompt as a job against a fixed pool of accelerators, so once the arrival rate approaches the rate at which jobs finish, requests simply wait. What makes this deceptive is the shape of the delay: while the pool has slack, an extra request costs almost nothing, but as the pool nears saturation the wait per job climbs steeply, and the last increment of load buys the worst response times of the day. Queueing theory described that curve long before computers existed, and nothing about a language model exempts it.

A product team at a mid-sized software company ran into the curve without recognizing it. They scheduled a nightly batch of content drafts, fired every job at once because the API accepted them all, and watched a run that used to finish before midnight spill into the following afternoon. The natural response made it worse: engineers assumed jobs had silently died, resubmitted them, and doubled the arrival rate against the same fixed capacity. Each resubmission consumed real tokens on work that was already queued, so the bill rose while throughput fell. The release train slipped twice before anyone thought to count how many requests were in flight at the same moment.

The lever is concurrency, not volume. A cap on simultaneous in-flight requests, even a crude one, keeps the pool below saturation, and total wall-clock time for the same set of jobs usually falls, because nothing sits behind work it does not depend on. Providers publish rate limits precisely because they expect callers to shape their own traffic, but the interfaces built on top of those APIs rarely show how many of your own jobs are still pending. Until they do, the number of requests you have outstanding is a metric you have to keep yourself.