Webhook retry strategies compared
The retry strategy that holds up in production is two-phase: a few retries seconds apart to absorb transient failures, then delays that grow to hours to ride out real outages, with random jitter added so that deliveries which failed together do not retry together. Fixed-interval retries synchronize subscribers and hammer a recovering endpoint; pure exponential backoff spreads them out but quickly leaves gaps of many hours. Hook0 uses a two-phase schedule with jitter, described in How Hook0 retries.
Some share of webhook deliveries always fails on the first attempt: network blips, rolling deploys, load balancer drains, brief DNS hiccups. These are transient failures, and the endpoint is fine a few seconds later.
Without retries, those events are gone. With naive retries, you risk making things worse. Your strategy determines whether you recover quietly or trigger a cascading outage.
Strategy comparison
| Strategy | Covers long outages | Thundering herd risk | Complexity | When to use |
|---|---|---|---|---|
| Fixed interval | Only with many attempts | High | Low | Dev/test environments only |
| Exponential backoff | Yes, with growing gaps | Medium | Low | Single-consumer systems |
| Exponential + jitter | Yes, with growing gaps | Low | Medium | Multi-consumer systems |
| Two-phase (fast then slow) | Yes, with bounded gaps | Low | Medium | Production webhook infrastructure |
| Circuit breaker | Depends on the reset policy | None | High | When you control both sides |
Fixed interval: why it breaks at scale
Say you have a 10-second fixed retry and 1,000 subscribers hitting a recovering endpoint:
- t=0s: 1,000 deliveries fail (endpoint down)
- t=10s: 1,000 retries hit at the same time
- t=20s: 1,000 retries again -- the endpoint just came back but gets 1,000 concurrent requests
The endpoint either recovers and immediately gets hammered, or stays down because the retry storm prevents recovery. A transient failure becomes a sustained outage.
Exponential backoff: better but not enough
Exponential backoff (1s, 2s, 4s, 8s, 16s...) spreads load over time. But when subscribers are synchronized, every retry wave still lands at the same instant, just less often.
The fix is jitter: add randomness to each delay. Instead of delay = base * 2^attempt, use delay = random(0, base * 2^attempt). This breaks the synchronization across subscribers.
Without jitter, 1,000 subscribers still slam the endpoint at each backoff step. With jitter, those 1,000 retries spread across the full interval.
Two-phase retry: the production pattern
Pure exponential backoff has a problem: delays grow too fast. After 10 attempts at base 2s, you are waiting 17 minutes between retries. After 15 attempts, 9 hours. You either cap the delay (reinventing two-phase) or accept huge gaps.
Two-phase retry splits the schedule into two stages. The fast phase uses short, increasing delays to catch transient failures within seconds. The slow phase switches to fixed-interval retries over hours or days for longer outages.
This maps to how failures actually play out: most resolve in seconds (a restart, a deploy rollout), while some take hours (provider outage, DNS propagation).
How Hook0 retries
Hook0 does not expose per-phase or per-subscription retry tuning. It applies one schedule of increasing delays to every failed delivery:
| Failed attempts so far | Base delay before next attempt |
|---|---|
| 1 | 3 seconds |
| 2 | 10 seconds |
| 3 | 3 minutes |
| 4 | 30 minutes |
| 5 | 1 hour |
| 6 | 3 hours |
| 7 | 5 hours |
| 8 and beyond | 10 hours |
The seconds-apart attempts at the start catch the transient failures that resolve in moments, like container restarts, deploy rollouts, or brief network partitions. The hours-apart attempts later cover longer outages, such as provider downtime or DNS propagation, without hammering an endpoint that is coming back. From the eighth retry on, the delay holds at 10 hours.
Hook0 also applies jitter on top of this schedule, for the reason described above: a small random amount is added to every delay, so deliveries that failed together do not retry together. The amount is only ever added, never subtracted, so a retry never fires earlier than its base delay -- see Why delays are not exact.
Two limits bound the schedule, whichever is reached first: MAX_RETRIES (default 24) and MAX_RETRY_WINDOW (default 8 days). Both are set on the output worker, not per subscription. With the defaults, a failing delivery is retried up to 24 times across roughly 8 days. The full reference is in Webhook retry logic.
Further reading
- Webhook retry logic -- how Hook0 schedules and bounds retries
- Request attempts -- how each delivery attempt is tracked
- Monitor webhook performance -- delivery rates and retry patterns