Skip to main content

Webhook retry logic

When a webhook delivery fails (network timeout, 5xx response, DNS error, connection refused), Hook0 retries with increasing delays. Each failed attempt creates a new request attempt scheduled for later, until the retry limit is reached.

Why retries matter​

Most webhook delivery failures are transient. The receiving server was restarting, a load balancer was draining connections, or a brief network partition occurred. A retry a few seconds later usually succeeds.

Without retries, every transient failure becomes a lost event. With naive retries (fixed interval, no limit), you risk overwhelming a recovering server. Hook0 uses a fixed escalating schedule. Short delays come first, to recover from brief outages quickly; longer delays follow, so a struggling endpoint is not hammered.

The retry schedule​

Hook0 retries on a fixed schedule of increasing delays. The delay before each attempt depends on how many times the delivery has already failed, plus a small random amount (see below):

Failed attempts so farBase delay before next attempt
13 seconds
210 seconds
33 minutes
430 minutes
51 hour
63 hours
75 hours
8 and beyond10 hours

The first attempts are seconds apart to recover from brief outages quickly. Later attempts stretch to hours so a long outage does not turn into a retry storm against an endpoint that is just coming back. From the eighth retry on, the delay holds steady at 10 hours.

This schedule is the same for every application and subscription. It is not tuned per subscription; the two limits that bound it are set on the output worker (see Configuration).

Why delays are not exact​

Hook0 adds a small random amount on top of each delay. It is only ever added, never subtracted, so a retry never fires earlier than the base delay in the table above.

This matters because failures arrive in groups. When one endpoint goes down, every subscription pointing at it fails at almost the same instant. Without randomness they would all retry at exactly the same instant too, and keep doing so at every step of the schedule -- a wave that stresses the queue and keeps the same deliveries locked together for as long as the outage lasts. Spreading them out breaks that lockstep and keeps one struggling endpoint from delaying deliveries for everyone else.

By default the random amount is up to 10% of the base delay, with a minimum of 2 seconds and a maximum of 15 minutes. A first retry therefore fires between 3 and 5 seconds after the failure, and a 10-hour retry between 10h00 and 10h15. Self-hosted deployments can tune this with RETRY_JITTER_RATIO and RETRY_JITTER_MAX_SPREAD (see Configuration); setting either to 0 restores strictly deterministic delays.

The 2-second minimum is not configurable on its own: lowering RETRY_JITTER_RATIO never takes the random amount below it.

How far retries go​

Two limits decide when Hook0 stops retrying, whichever is reached first:

  • MAX_RETRIES (default 24): the maximum number of retry attempts.
  • MAX_RETRY_WINDOW (default 8 days): the maximum total time spent retrying. Hook0 schedules the next attempt only if it still fits inside this window.

With the defaults, a failing delivery is retried up to 24 times over roughly 8 days before Hook0 gives up.

What happens on failure​

When a delivery attempt fails, Hook0 follows this decision process:

Non-retryable errors​

Some errors are never retried because retrying would produce the same result:

  • Invalid header: the webhook signature could not be constructed (e.g., event type contains characters that are invalid in HTTP headers).

Subscription and application checks​

Before scheduling a retry, Hook0 checks that the subscription is still enabled, has not been soft-deleted, and that the parent application still exists. If any of these fail, the retry is skipped.

If the subscription is degraded or recovering, the next retry is scheduled at least 1 hour later (DEGRADED_SUBSCRIPTION_MIN_RETRY_DELAY), or later if the normal delay (including Retry-After and jitter) is longer. This never changes how many retries a delivery gets.

Subscription health​

When most deliveries of a subscription fail, retrying them at the normal pace only adds load on an endpoint that is already struggling. Hook0 can track the health of each subscription and slow down deliveries of unhealthy ones until their endpoint recovers.

How health is computed​

Every 5 minutes (SUBSCRIPTION_HEALTH_PROBE_PERIOD), Hook0 looks at the delivery attempts of each active subscription that completed during the last 15 minutes (SUBSCRIPTION_HEALTH_PROBE_WINDOW):

  • Unhealthy: at least 10 completed attempts (SUBSCRIPTION_HEALTH_UNHEALTHY_MIN_ATTEMPTS) and more than 30% of them failed (SUBSCRIPTION_HEALTH_MAX_FAILURE_RATIO).
  • Healthy: at least 1 completed attempt and at most 30% of them failed.
  • Otherwise (for example, no completed attempt), nothing changes.

Failures caused by Hook0 itself (E_INTERNAL, E_INVALID_HEADER) are not counted at all, so an incident on Hook0's side does not make subscriptions unhealthy. Cancelled attempts (subscription disabled or deleted) are not counted either.

A subscription then moves between three states:

A recovering subscription goes back to healthy after 1 hour (SUBSCRIPTION_HEALTH_RECOVERING_MAX_DURATION), or earlier, after 10 minutes (SUBSCRIPTION_HEALTH_RECOVERING_MIN_DURATION), if at least 10 attempts (SUBSCRIPTION_HEALTH_RECOVERING_MIN_ATTEMPTS) completed since it started recovering. It goes back to degraded if any check in the meantime finds it unhealthy.

Effects on deliveries​

While a subscription is degraded or recovering:

  • New request attempts (including the ones created by replaying an event) are created paused: they are not sent right away.
  • Each output worker regularly releases paused attempts, oldest first, so that the subscription gets at most 50 (degraded) or 500 (recovering) request attempts per minute, counting the ones that were just processed and the ones already waiting. Once the subscription is healthy again, the remaining paused attempts are released at up to 5000 per minute. See the PAUSED_REQUEST_ATTEMPTS_RELEASE_* settings of the output worker.
  • Retries are scheduled at least 1 hour later (see Subscription and application checks).

A paused attempt is never dropped: it is eventually sent, or cancelled if the subscription is disabled or deleted.

Modes​

Subscription health is controlled by SUBSCRIPTION_HEALTH_PROBE_MODE on the API:

ModeBehavior
off (default)Subscriptions are not classified and health has no effect
shadowHealth is computed and recorded, but has no effect on deliveries
enforceHealth pauses and slows down deliveries of unhealthy subscriptions; SUBSCRIPTION_HEALTH_ENFORCE_ONLY_FOR can restrict this to some applications

Switching from enforce back to off or shadow (and restarting the API) stops pausing new attempts; attempts that are still paused are released at the healthy rate.

Delivery status flow​

Each webhook delivery attempt goes through these states:

More precisely, Hook0 tracks six statuses (the API value is the status.type field of a request attempt):

StatusAPI valueMeaning
WaitingwaitingScheduled for future delivery (delay_until has not elapsed yet)
PendingpendingReady to be picked up by a worker
PausedpausedHeld back because the subscription is degraded or recovering; it will be released gradually
In Progressin_progressCurrently being delivered (picked by a worker)
SuccessfulsuccessfulDelivery succeeded (2xx HTTP response)
FailedfailedDelivery failed

The request_attempt table stores every attempt with timestamps (created_at, picked_at, succeeded_at, failed_at, delay_until), so you can calculate:

  • Time to first delivery: picked_at - created_at
  • Delivery latency: succeeded_at - picked_at
  • Total time to success: succeeded_at - created_at (including retries)

Each retry creates a new row in the request_attempt table with an incremented retry_count and a delay_until set to the scheduled retry time.

When all retries are exhausted​

When the maximum number of retries is reached (or the retry window expires), Hook0 does not create another attempt. The last attempt stays in failed status.

Failed deliveries are not lost. You can:

  1. Inspect all delivery attempts and their responses via the API or dashboard
  2. Replay the event via the API to re-trigger delivery to all matching subscriptions

Replaying an event resets its dispatched_at field. The dispatch trigger then creates new request attempts for all active subscriptions that match the event's type and labels.

Idempotency​

Every event in Hook0 has a unique event_id. Consumers should use this as an idempotency key to handle duplicate deliveries.

Duplicates happen when:

  • The consumer processed the event but returned a non-2xx response (e.g., crashed after processing but before responding)
  • Network issues caused the response to be lost
  • Manual replay of an event

Example implementation​

-- PostgreSQL example
CREATE TABLE processed_webhooks (
    event_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Before processing:
INSERT INTO processed_webhooks (event_id)
VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

-- If no row returned, event was already processed -- skip it.

Configuration​

The output worker's retry and delivery behavior is configured via environment variables:

ParameterDefaultDescription
MAX_RETRIES24Maximum delivery attempts before giving up
MAX_RETRY_WINDOW8 daysMaximum time window for retries
CONNECT_TIMEOUT5 secondsTimeout for establishing a TCP connection
TIMEOUT15 secondsTotal HTTP request timeout (including connect)
CONCURRENT1Number of request attempts handled concurrently
DEGRADED_SUBSCRIPTION_MIN_RETRY_DELAY1 hourMinimum retry delay while a subscription is degraded or recovering

Subscription health settings are listed in the configuration reference.

Error types​

When a delivery fails, Hook0 records one of these error codes:

Error codeMeaning
E_TIMEOUTThe HTTP request timed out
E_CONNECTIONCould not establish a connection to the target
E_DNSThe target's hostname could not be resolved: either no name server could be reached, or one answered with an error (for example SERVFAIL or REFUSED). This does not mean the target URL is wrong
E_HTTPThe server responded with a non-2xx status code
E_INVALID_TARGETThe target URL is invalid, does not exist (NXDOMAIN), or resolves to a forbidden IP
E_INVALID_HEADERA required header value could not be constructed (non-retryable)
E_INTERNALA failure on Hook0's side that says nothing about the target (for example, the event's payload could not be found); it does not count against the subscription's health
E_UNKNOWNAn unexpected error occurred

SSRF protection​

Hook0 blocks webhook deliveries to IP addresses that are not globally reachable by default (loopback, RFC 1918 private ranges, carrier-grade NAT, link-local -- which covers cloud metadata endpoints such as 169.254.169.254 -- unique-local, multicast and reserved ranges). This prevents Server-Side Request Forgery attacks.

IPv6 targets are checked too. The whole IETF-reserved block ::/8 (RFC 4291 section 4) is refused: nothing routable was ever allocated there, but every way of writing an IPv4 address inside an IPv6 one lives there -- IPv4-mapped (::ffff:0:0/96), IPv4-compatible (::/96), IPv4-translated (::ffff:0:0:0/96, RFC 2765 SIIT) and ISATAP (::5efe:0:0/96, RFC 5214). 6to4 (2002::/16, together with its IPv4 relay anycast prefix 192.88.99.0/24) is refused outright too, because its bits 16 to 47 are an arbitrary IPv4 address and that is where the traffic ends up. The one carve-out inside ::/8 is the NAT64 well-known prefix (64:ff9b::/96), which is refused only when the IPv4 address it carries is itself forbidden -- so Hook0 keeps working behind NAT64/DNS64 without the prefix becoming a way to spell 127.0.0.1 in IPv6.

The check applies to hostnames and to URLs written with an IP literal, and the addresses it vetted are pinned into the connection, so a second DNS answer cannot redirect the request afterwards (DNS rebinding).

This check can be disabled with the DISABLE_TARGET_IP_CHECK flag for development environments.

Further reading​