All posts
Queues
Durable queues: retries without the mystery cover

Engineering note

Durable queues: retries without the mystery

Priya Nair3 min readQueues

Retries are easy to add and hard to make honest. A look at leases, idempotency keys and dead-letter replay in the queue that powers background work on weond.

A background job that fails once is an incident. A background job that fails forever is a data problem. Most of the work in a queue system is not delivering messages — it is deciding what to do when delivery goes wrong.

weond queues are signed HTTP delivery queues. Producers enqueue a JSON payload, consumers receive a signed POST, and the platform tracks the result. That sounds simple until you ask three questions: who owns a message while it is being processed, how many times may it be delivered, and where does it go when nobody can process it?

Leases, not locks

When a consumer receives a message, the platform grants a lease. The message stays reserved for the duration of the lease and becomes visible again if the consumer does not acknowledge it in time.

This is deliberately not a lock. A crashed consumer should not block a queue forever, and a slow consumer should not be able to hold a message indefinitely without saying so. Consumers can extend a lease while they are still working, which makes long-running jobs explicit instead of accidental.

  • Visibility timeout controls how long a message stays reserved.
  • Heartbeat extension keeps a healthy long job alive.
  • Max attempts decides when a message stops being retried.

Idempotency is the contract

At-least-once delivery means your handler will occasionally run twice. There is no clever queue configuration that removes this requirement; there is only a handler that can tolerate it.

Every delivery includes a stable message ID and an attempt number. We recommend writing the ID to your database inside the same transaction as the side effect:

insert into processed_messages (message_id)
values ($1)
on conflict do nothing;

If the insert reports zero rows, the work was already done. Acknowledge the message and move on.

Queue packets moving through parallel channels

Dead letters are a feature

When a message reaches its maximum attempts, it moves to the dead-letter queue. The payload, headers, attempt history and last response are preserved. From the dashboard you can inspect the last error and replay the message with a fresh attempt budget.

Replay is intentionally manual. Automatic replay loops hide bugs and create traffic storms. A human decision is cheap; a queue that retries a broken job for three days is not.

The goal is not zero failures. The goal is failures that are visible, bounded and reversible.

Operating a queue you can trust

The most useful metrics are not throughput. They are age of the oldest visible message, lease expiration rate and dead-letter growth. Those three numbers tell you whether the queue is keeping up, whether consumers are healthy and whether something is quietly broken.

You can create a queue from the project dashboard and inspect its deliveries through the CLI reference. Start with a small max-attempt count, make your handler idempotent, and let the dead-letter queue be boring.