What I wanted to do
A website's contact form API was calling a script server directly that appends a row to a spreadsheet. While writing, this script server holds a lock over the whole thing, so when several inquiries came in at once, only one got in and the rest failed with a "busy right now" error. Users saw the error and lost what they had typed.
Here are a few terms that are good to know in advance.
- Queue and worker: A queue is a place where tasks wait in line, and a worker is a program that takes them out one by one and does the actual work. Think of order tickets and the kitchen in a restaurant.
- ack (acknowledgement): A signal the worker sends to the queue to say "this task is done, you can delete it."
- Jitter: A random value mixed into the retry interval. It keeps failed callers from coming back at the same moment.
What I did
I changed the flow from "the API calls directly" to "the API puts the request in a queue and immediately returns success, and a worker slowly writes it in."
- The API validates the input, creates a unique ID (UUID) for each inquiry, attaches it as an idempotency key, puts it in the queue, and returns success right away. Only when putting it in the queue itself fails does it fall back to a direct call within the same request.
- The worker calls the script server once. If the failure is "worth retrying" (busy, 5xx, timeout), it tries a few more times within the same delivery at random intervals of 1 to 4 seconds.
- If it still fails, it deliberately throws an exception so the queue redelivers at increasingly longer intervals. When the maximum number of attempts is exceeded, it reports to the error tracking tool together with the inquiry ID, and only then acks.
- The script server now only waits for the lock, checks for duplicates by inquiry ID, appends the row, and responds immediately. Notifications are separated out and sent by a periodic trigger.
What I learned
Adding a queue does not resolve contention. The queue I was using had no "deliver only one at a time" option. So ten requests were delivered to the worker at the same moment, and the redeliveries also arrived together and hit the lock again. I let random-interval retries inside the worker take care of sorting out the order, and kept the queue's long backoff only for when the other side is broken for a long time. → Retries, exponential backoff, and jitter
Acking an exception makes it vanish silently. Since the user had already been told "received," if the worker swallows an unexpected exception and exits normally, an inquiry can disappear without anyone knowing. I made it ack only on success and tolerate duplicates from redelivery with the idempotency key. → At-least-once delivery, ack timing, and idempotency keys
For a runtime where deployment and triggers are separate, the entry point has to guarantee the trigger. The script server's time-based trigger does not appear by itself when you deploy the code. If you only deploy and never create the trigger, notifications stay in "pending" forever. Once I made the entry point check for its own trigger and create it if missing, the manual step went away.
There are also things I have not been able to confirm. The queue path cannot be run locally, so the local tests passed only through the fallback path, and I confirmed the queue path only through production logs. The reason the retry-interval option was not applied as expected is still only a guess.