Webhook delivery: signatures, retries and safe replays

A webhook is a promise with a failure mode built in: the platform sends an HTTP request to your endpoint, and your endpoint might be deploying, rate-limited, or just down for thirty seconds. The design questions that matter are what happens next, and how the receiving side tells a genuine event from a forged or replayed one.

Verifying a delivery actually came from AACSearch

Every outbound webhook carries an HMAC-SHA256 signature computed over the timestamp and body, using a secret only your organization and AACSearch know. A receiving endpoint recomputes the same signature from the raw body and compares it before trusting the payload — not after parsing it into your own types, since a payload that fails signature verification shouldn't be acted on regardless of how well-formed its JSON looks. The timestamp in the signed payload also lets a receiver reject an old, replayed request outside a reasonable clock-skew window, independent of TLS.

What happens when your endpoint is down

A delivery that gets a network error or a 5xx response retries automatically with backoff rather than being dropped after one attempt. Every attempt — the eventual success, or the final failure if retries are exhausted — is recorded in a delivery log you can inspect from the console. If your endpoint was down for an incident and came back an hour later, you don't have to wait for the event to happen again to confirm the fix: replay any recorded delivery by hand and watch it succeed.

This matters more than it sounds: a webhook consumer that silently drops missed events during a deploy window has a data-completeness bug that only shows up when someone asks why a downstream system disagrees with AACSearch's own state. A delivery log with manual replay turns that into an operational non-event.

Idempotency: the other half of the problem

Retries solve delivery; they create a new problem on the receiving side, because the same event can now arrive more than once — your endpoint answered 200 but the response was lost, or a replay was triggered by hand after an incident. Every webhook payload carries a stable delivery id; a handler that records processed delivery ids before acting on the event, and skips ones it's already seen, gets exactly-once processing semantics on top of at-least-once delivery.

The same idea protects writes on the way in. Any call the tenant API accepts can carry an Idempotency-Key header: retry the same write with the same key and the API returns the original result instead of repeating the side effect, and reusing the key with a different body is rejected outright rather than silently accepted. Between an idempotency key on writes and a delivery id check on webhook handlers, a retry — yours or AACSearch's — stops being something your integration has to design defensively around on a case-by-case basis.