Skip to content
ENNO
← Back to blog
·4 min read·

Idempotency Is the Integration Feature You Actually Need

Webhooks arrive twice and people click twice. How I make a second delivery harmless: a key derived from the content, a partial unique index, and the endpoint I forgot to give one.

Idempotency Is the Integration Feature You Actually Need

The integration bug that costs real money is almost never a parsing error. It's the same invoice going out twice because a webhook arrived twice, or the same message landing on someone's phone twice because a person clicked a button twice.

Payment providers, CRMs, message queues and my own retry loops all promise delivery. What they mean is that they'll keep trying until I say yes, and if my yes gets lost on the way back, the message comes again. That's not a bug on their side, it's the only guarantee a network can give. So the job on my side is making a second delivery harmless.

A receipt for every send

Every operation that does something in the outside world gets a key, and the key goes in a table with a unique index, in the same transaction as the work. If the insert hits the index, it's already been done, so I return what happened the first time instead of doing it again.

Here's a real one, from a task app I built for a client that sends field workers their jobs on WhatsApp. When a manager moves a job's due time, the app can send the worker a short "your job moved" message. It builds the key from what the message says:

csharp
private static string ComputeNotifyUpdateKey(Task task) { var raw = $"notify-update|{task.Id}|{task.WorkerId ?? Guid.Empty}|{task.DueDatetime:O}"; var hash = SHA256.HashData(Encoding.UTF8.GetBytes(raw)); return Convert.ToHexString(hash).ToLowerInvariant(); }

Same task, same worker, same new time gives the same key. A different time gives a new key, which is right, because that's a new message the worker needs. The index is partial:

sql
CREATE UNIQUE INDEX ux_dispatches_request_key ON dispatches (request_key) WHERE request_key IS NOT NULL AND status NOT IN ('failed', 'undelivered');

That WHERE matters. A send that failed shouldn't block the retry, so failed rows drop out of the index and the next attempt can claim the key. A second click while the first send is queued or done hits the index, Postgres raises 23505, and the API answers 409 with the dispatch that already exists:

csharp
catch (DbUpdateException ex) when (ex.InnerException is PostgresException { SqlState: "23505" }) { var existing = await db.Dispatches.AsNoTracking() .Where(d => d.RequestKey == requestKey) .OrderByDescending(d => d.CreatedAt) .FirstOrDefaultAsync(ct); return Conflict(new { error = "notify_update_exists", dispatch = existing }); }

The endpoint next to it

That endpoint was fine. The bug came from the one next to it.

Two columns comparing two buttons on the same task. Notify worker (keyed): key from sha256 of task, worker and due time; insert with a unique index; second click fails with 23505 and returns 409 with the message that already went out; worker gets one WhatsApp. Dispatch now (no key): due date edited but the page still shows Dispatch now; manager clicks it; nothing says already sent; worker gets two WhatsApp messages.
Same task, same worker, two buttons.

In a round of testing with the client, a manager edited a job's due date and the screen kept showing "Dispatch now" instead of "Notify worker". "Dispatch now" goes to a different endpoint, the one that sends the original assignment, and that one had no key for this case. So the worker got the whole job again as a fresh message. Making the screen refresh after an edit was a small fix. The lesson was on the backend. Every endpoint that sends something needs its own answer to "what if this runs twice", not just the one I was thinking about at the time.

Two doors into the same record

The other place it bites is when the same fact can arrive by two routes. My own CRM, which tracks the freelance work I bid on, learns about status changes from a poller and from a webhook, and both can see the same change. Both paths go through the same command, and the event's timestamp comes from the source data rather than from when my code saw it. A poll and a webhook for the same change produce the same row, and the second one is a no-op.

If I'd stamped it with DateTime.UtcNow instead, a change seen by both would show up twice in the history, and it would look like the client had done something twice.

The outbound side

When I'm the caller it's the same problem from the other end. If I post a charge and the connection drops, I don't know whether it went through. Most serious APIs take an idempotency key on the request and hand back the original result on a repeat. Where an API doesn't, I need a way to ask it what it did, which usually means searching by my own reference before retrying.

Retries get a budget

Retrying straight away hammers a service that's already struggling, and retrying forever lets one poisoned message block everything behind it. I back off exponentially with a cap, give up after a fixed number of tries, and put what I gave up on somewhere a person will see it. In the task app that's the dispatch row itself, with its status, attempt count and the provider's error.

That row is also the answer when someone asks "did the worker get the message". It's a query, not an afternoon of reading logs.

#integrations#APIs#webhooks#idempotency#reliability