Design webhooks as a delivery protocol

A webhook is an asynchronous delivery protocol, not an HTTP callback sprinkled onto an API. Providers should define how messages are authenticated, identified, retried, ordered, retained, replayed, and observed. Consumers should acknowledge quickly, persist first, process later, and assume duplicates and reordering will occur.

Sign the message that was actually sent

Use HTTPS and sign a well-defined set of request components. At minimum bind the method, target, timestamp, unique delivery ID, and exact body bytes. Verification must operate on the raw body before JSON parsing or normalization. Include a key identifier and publish a rotation procedure with overlap. RFC 9421 defines HTTP Message Signatures and the covered-component model; even a provider-specific scheme should be equally explicit about canonicalization.

Reject timestamps outside a narrow tolerance and remember delivery IDs long enough to block replay. A timestamp alone does not prevent two uses inside the allowed window. Compare signatures in constant time, authenticate before expensive processing, and never fetch a verification key from an arbitrary URL supplied by the event.

Make at-least-once behavior visible

Give every logical event a stable event ID and every delivery attempt a distinct attempt ID. The event ID is the consumer’s idempotency key; the attempt ID is for transport debugging. If a response is lost after the consumer commits, the provider will retry. Calling that an error would sacrifice reliability.

GitHub’s webhook best practices recommends validating deliveries, responding promptly, subscribing only to needed events, and handling redelivery. A consumer endpoint should authenticate, enforce size limits, durably enqueue the event, and return success. Business processing belongs behind the queue so provider timeouts do not become consumer data loss.

Publish the retry schedule: which status codes retry, how long attempts continue, whether Retry-After is honored, and when an endpoint is disabled. Use exponential backoff with jitter and a maximum age. Treat redirects cautiously; following an attacker-controlled redirect can leak signed content. DNS changes and private-address resolution require outbound request controls on the provider side.

Define ordering narrowly

Global event order usually creates a bottleneck. State the useful boundary, such as events for one repository or subscription. Include an aggregate version or monotonic sequence. Consumers can ignore an older state snapshot, park a gap, or fetch current state. If events are facts rather than snapshots, replay order may be required, so partition and document it.

Version the event envelope separately from domain payloads. Additive fields should be tolerated; semantic changes need a new event type or version. Include occurrence time and delivery time, but do not make wall-clock order the only sequencing mechanism. Clocks and retries make it ambiguous.

Build replay and diagnosis into the product

Retain redeliverable events for a stated period. Let authorized operators replay by event, endpoint, or time range without changing the original event ID. Mark replay attempts and avoid creating a second logical event. Consumer dashboards should show signature result, endpoint response class, latency, attempt count, next retry, and a redacted response excerpt. Never store secrets or full sensitive payloads merely for convenience.

The contract review should answer: What bytes are signed? How are keys rotated? What identifies an event? Which responses retry? What order is promised? How long can events replay? How does a consumer recover after downtime? What payload fields are sensitive?

Implement the receiver’s durable boundary first. Verify the signature over raw bytes, insert the event ID with a unique constraint, enqueue processing in the same local transaction, and return. Then test the uncertain case by dropping the success response and proving the provider’s retry does not repeat the business effect.

Acknowledgement means accepted for processing, not that every downstream side effect has completed.