Queue-Based Load Leveling — Separating Response from Processing to Absorb Bursts of Requests

Definition

When a job looks like "receive a request and record it in another system", such as a form submission, this pattern puts a queue (a place where pending jobs wait in line) between the side that receives requests (the API) and the side that actually records them. The API validates the request, puts it on the queue, and returns success right away. A separate program called a worker then takes items off the queue one by one and does the real work.

"Load leveling" in the name means the queue flattens the sharp peak of a sudden burst of requests, so they reach the other system only at a pace it can handle. It does not mean the queue automatically guarantees "one at a time" processing. We come back to that difference below.

Why it matters

If the API calls an external system directly and waits until it finishes, these problems appear.

• When the external system locks everything while it writes (a script server that appends rows to a spreadsheet is a typical case), ten simultaneous requests let only one in, and the rest fail with "busy right now".
• When the external system limits call frequency (rate limit), errors show up only during a burst. The bug stays invisible on normal days and blows up right after an event.
• Users wait until the slow call finishes, and if it fails, they lose what they entered.

With a queue, the user's response gets faster by the time it takes to put the item on the queue, and the worker can absorb processing failures through retries.

How it works

Browser → API: validate → assign a unique ID → publish to queue → return { ok: true } immediately
                                         ↓
                       Queue (durably stored) → Worker: one external call
                                         ├─ success → ack (removed from the queue)
                                         ├─ transient failure → short jittered retry inside the worker
                                         └─ still failing → throw an exception → the queue redelivers after a delay

1. Accept (publish): The API only validates the input, generates a unique ID for deduplication, and publishes to the queue before responding. From this moment it has promised "accepted".
2. Delivery: The queue hands the message to the worker. Until the worker sends an ack (acknowledgment) to report success, the queue does not delete the message and delivers it again.
3. Two layers of retry: The short retries inside the worker resolve momentary conflicts such as lock contention, while the queue's long-interval redelivery handles failures that last, such as an outage on the other side (see retry-backoff-with-jitter).
4. Exhaustion handling: Once the maximum delivery count is exceeded, the worker reports the failure to an error-tracking tool along with an identifier, then acks so the queue does not get stuck.

Note: a queue may not reduce concurrency. Many managed queues have no FIFO or "deliver one at a time" option, so ten messages can reach workers at the same moment, and their redeliveries also pile up at the same time. Sorting out the order is the job of randomized retries inside the worker.

In practice

// API: accept the request and respond
export async function POST(req: Request) {
  const values = validate(await req.json());
  const submissionId = crypto.randomUUID();
  try {
    await queue.send("contact", { submissionId, values }, { idempotencyKey: submissionId });
  } catch {
    await callBackendDirectly(submissionId, values); // fall back to a direct call only when the queue publish fails
  }
  return Response.json({ ok: true });
}

// Worker: return normally (= ack) only on success
export async function handle(message: { submissionId: string; values: Values }) {
  const result = await callBackendWithJitterRetry(message); // includes a short jittered retry
  if (!result.ok) throw new Error(result.code); // throwing makes the queue redeliver
}

• The fallback path and the worker should use the same call and failure-classification code, so the two paths do not drift apart.
• In local development, the queue path often fails because there is no auth token. Check that local tests are not passing only through the fallback path, and verify the queue path at least once with production logs.

Trade-offs

When not to use it

• For requests where the user needs the result right away (such as the outcome of a payment approval), an acceptance response alone is not enough.
• If the other system is fast enough and has no concurrency limit, a queue only adds complexity.
• If the queue does not guarantee order or concurrency, do not assume that "having a queue makes the contention go away".

Common mistakes

• Acking an unexpected exception and silently dropping it. Since you have already answered "accepted", the request is lost without anyone knowing. Ack only on success.
• Relying only on the queue's redelivery. Redeliveries that pile up at the same moment collide in the same place again. Retry first inside the worker at random intervals.
• Not handling publish failure. A queue outage becomes an intake outage. Provide a direct-call fallback, but limit how many times it is used.
• Never running the queue path even once. Locally, only the fallback may be running.

Related concepts

• retry-backoff-with-jitter — how to split retries between inside the worker and the queue level
• idempotent-consumer-ack-timing — why redelivery creates duplicates, and the ack rules
• optimistic-outbox-rebase — a client-side structure that stacks up outgoing work and delivers it later
• serverless-stateless-execution — why retry state cannot be kept in memory when the worker is a stateless process started fresh for each request