Idempotency is a contract, not a property you hope for

#Retries are not optional

A client sends a request. The response is lost in the network. The client does not know whether the server processed it, so it retries. This is not an edge case, it is the normal behaviour of every network client, and any write path without an answer for it will eventually charge somebody twice.

The general form of the problem has been well described for a long time [1]. Once the acknowledgement can be lost independently of the effect, at-least-once is the strongest delivery guarantee available end to end, and exactly-once has to be built on top of it by the receiver.

#Where the key comes from

The client generates it, and this is the part people resist. A server-generated key cannot help, because the client needs the key before it makes the first call in order to use it on the retry.

The key must be stable across retries of the same logical operation and different across different operations. A UUID generated when the user presses the button satisfies both. A hash of the request body does not, because two genuinely distinct identical operations, such as the same customer buying the same item twice, would collide.

#What to store

Storing the key alone is not enough. The second request needs a response, and returning an empty success is wrong: the client asked for the result of the operation and is entitled to the same result it would have received the first time.

1type IdempotencyRecord struct {
2    Key         string
3    RequestHash string    // guards against key reuse with a different body
4    Status      int
5    Response    []byte
6    CreatedAt   time.Time
7}
Copy

The request hash is the guard that catches a client bug. If the same key arrives with a different body, that is not a retry, and returning the first response would be wrong. A 422 with a clear message surfaces the bug instead of hiding it.

#The concurrent case

The retry may arrive while the first request is still running, which is common when the original is slow enough that the client timed out. Two handlers now execute the same operation concurrently.

The record has to be created before the work starts, in the same transaction that does the work, so that the second arrival finds a row in progress rather than no row at all. What it does then is a policy choice: wait briefly for the first to finish, or return 409 and let the client retry again. Waiting is friendlier and holds a connection, and I have shipped both.

#How long to keep them

Long enough to cover the longest retry window any client uses, plus a margin. Twenty-four hours is a common answer and it is a guess unless you have measured your clients.

The records accumulate, and pruning them is a real operational task. A table of idempotency keys that nobody prunes will eventually be the largest table in the database, and its index will be the reason writes got slower.

#Where it does not reach

Idempotency at the API boundary does not make downstream effects idempotent. If handling a request sends an email, the retry must not send a second email, and that requires the same discipline one layer down. Systems that get this right at the edge and wrong internally are common, and the symptom is a customer receiving four identical notifications [3] [2].