A signal arcing toward an endpoint, a faint duplicate fading harmlessly, over an immutable event log
EngineeringApr 30, 20266 min

Webhooks you can actually trust

Exactly-once delivery is a lie. Here is the contract we offer instead, and why it holds.

By Deniz Aydın · Events & delivery

Every payment platform promises reliable webhooks. Then your endpoint has a bad deploy, a retry storm double-ships an order, and you learn what the promise was worth. Exactly-once delivery is impossible — so we offer a different contract.

The exactly-once lie

We POST an event to your endpoint. You process it, then the connection drops before your 200 reaches us. From our side this is indistinguishable from your server crashing before processing. We must choose: retry (risking a duplicate) or don’t (risking a loss). We retry. Lost events are silent and catastrophic; duplicate events are noisy and — with the right contract — harmless.

Our contract instead

Lumina’s delivery contract has three clauses, each checkable from your side: at-least-once with a stable event.id; HMAC-signed and versioned deliveries; and nothing that exists only as a webhook — every event is also in the GET /v1/events log, forever. Webhooks are a latency optimization over the log, not the source of truth.

idempotent consumerts
const ev = lumina.webhooks.verify(req); // throws on bad sig
if (await redis.setnx(`seen:${ev.id}`, 1)) {
  await handle(ev);   // first time: process
}
res.sendStatus(200);  // duplicate: ack and drop

Ordering per object

Global ordering across a distributed system is a tarpit, and you don’t need it. The guarantee is that events are delivered in order per object, serialized on the object’s own timeline. Two different charges may interleave arbitrarily; one charge never contradicts itself.

Recovery as a feature

Failures are normal: deploys, cert expiries, that one proxy timeout no one can explain. Recovery ships as a first-class workflow — adaptive backoff over a 36-hour schedule, replay from the CLI against any endpoint including localhost, and full delivery introspection for every attempt. Design for the Tuesday your load balancer ate every third request.

More reading