Skip to main content

Debugging failed webhook deliveries

To debug a failed webhook in Hook0, open the event's request attempts, read the HTTP status and error name of the last one, then fetch its response to see the body and headers your endpoint sent back. Any non-2xx response or network error is retried automatically on a fixed schedule, so a 4xx caused by a bug in your handler keeps failing until you fix the handler or disable the subscription. Once the endpoint is fixed, replay the event with POST /events/{event_id}/replay.

The rest of this guide covers each step: reading the dashboard and the API, the five failure scenarios that cause most incidents, local debugging, alerting on failure rates, and bulk recovery.

What the delivery log records​

Every delivery try is a request attempt. Its stored response holds what your endpoint answered.

FieldStored onWhat it tells you
status.typeRequest attemptpending, paused, in_progress, waiting, successful or failed
retry_countRequest attempt0 for the first try, then one more for each retry
http_response_statusRequest attemptHTTP status your endpoint returned, empty when the request never got an answer
created_at, picked_at, failed_at, succeeded_atRequest attemptWhen the attempt was queued, sent, and how it ended
delay_untilRequest attemptWhen the next retry is due
response_idRequest attemptPoints to the stored response, fetched with GET /responses/{response_id}
response_error_nameResponseE_CONNECTION, E_TIMEOUT, E_DNS, E_HTTP, E_INVALID_TARGET or E_INVALID_HEADER
http_code, headers, bodyResponseThe status, headers and body your endpoint sent back
elapsed_time_msResponseHow long your endpoint took to answer
Try it on your own events

The free Hook0 Cloud plan includes delivery monitoring for 100 events a day, with no credit card. Create an account, or send a first request to Hook0 Play to see what a signed delivery looks like.

General Troubleshooting

For API errors, connection issues, and authentication problems, see Troubleshooting Guide. This guide focuses specifically on webhook delivery failures.

Quick diagnosis checklist​

Before diving deep, run through this quick checklist:

  • Is the webhook endpoint accessible?
  • Are you returning the correct HTTP status codes?
  • Is the endpoint processing requests within timeout limits?
  • Have you verified the webhook signature?
  • Is your subscription properly configured?
  • Is your subscription enabled? (see Troubleshooting Guide - Events Not Being Delivered)

Understanding webhook failures​

Hook0 categorizes delivery failures by root cause:

HTTP status code categories​

2xx - Success

Request processed successfully. No retry needed.

4xx - Client Errors

Retried like any other failure, so a 4xx from a bug in your handler fails again on every retry until you fix it. For 429 (Rate Limited), Hook0 waits at least as long as the Retry-After header asks, up to a cap.

5xx - Server Errors

Retried on the retry schedule. Points to a problem in your webhook endpoint.

Network errors
  • Connection timeouts
  • DNS resolution failures
  • Connection refused

Set up environment variables​

# Set your service token (from dashboard)
export HOOK0_TOKEN="YOUR_TOKEN_HERE"
export HOOK0_API="https://app.hook0.com/api/v1" # Replace by your domain (or http://localhost:8081 locally)

# Set your application ID (shown in dashboard URL or application details)
export APP_ID="YOUR_APPLICATION_ID_HERE"

Save these values:

# Save to .env file for later use
cat > .env <<EOF
HOOK0_TOKEN=$HOOK0_TOKEN
HOOK0_API=$HOOK0_API
APP_ID=$APP_ID
EOF

Step 1: Access Hook0 dashboard diagnostics​

  1. Go to your Hook0 Dashboard
  2. Select your Application
  3. Click on "Events"
  4. Find the failed event and click "View Details"
  5. Review "Request Attempts" section

Analyze delivery attempts​

A request attempt records where a delivery stands and how it got there:

{
  "request_attempt_id": "b3f1c2a4-5d6e-47f8-9a0b-1c2d3e4f5a6b",
  "event_id": "9f8e7d6c-5b4a-4938-8271-6a5b4c3d2e1f",
  "event": {
    "event_id": "9f8e7d6c-5b4a-4938-8271-6a5b4c3d2e1f",
    "event_type_name": "billing.invoice.paid"
  },
  "subscription": {
    "subscription_id": "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d",
    "description": "Billing service webhook"
  },
  "created_at": "2024-01-15T10:30:00Z",
  "picked_at": "2024-01-15T10:30:01Z",
  "failed_at": "2024-01-15T10:30:06Z",
  "succeeded_at": null,
  "delay_until": "2024-01-15T10:34:00Z",
  "response_id": "7c6b5a49-3827-4160-9e5d-4c3b2a1f0e9d",
  "retry_count": 3,
  "http_response_status": 500,
  "status": {
    "type": "waiting",
    "since": "2024-01-15T10:30:06Z",
    "until": "2024-01-15T10:34:00Z"
  }
}

http_response_status and retry_count tell you what happened and how many times Hook0 has tried. The response body, headers, and error name are not on the attempt: it links to them through response_id. Follow that id to GET /responses/{response_id} (Step 2 below) to read what your endpoint actually returned.

Step 2: Using the API for analysis​

Get request attempts via API​

# Get all request attempts for an application
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Filter by event
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&event_id={EVENT_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Filter by subscription
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&subscription_id={SUBSCRIPTION_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

Get response details​

# Get the response body and headers for a failed attempt
curl "$HOOK0_API/responses/{RESPONSE_ID}?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

Step 3: Common failure scenarios​

Scenario 1: Connection timeouts​

Symptoms:

  • Error message: "Connection timeout"
  • No HTTP status code
  • High duration_ms values

Diagnosis:

# Test endpoint connectivity
curl -v --max-time 30 https://your-webhook-endpoint.com/webhook

# Check DNS resolution
nslookup your-webhook-endpoint.com

# Test from different networks
curl -v https://your-webhook-endpoint.com/webhook

Solutions:

  • Optimize webhook processing to respond faster
  • Increase server resources
  • Check network connectivity
  • Verify DNS configuration

Scenario 2: SSL/TLS certificate issues​

Symptoms:

  • Error message: "SSL certificate verification failed"
  • Connection errors for HTTPS endpoints

Diagnosis:

# Check SSL certificate
openssl s_client -connect your-domain.com:443 -servername your-domain.com

# Verify certificate chain
curl -v https://your-webhook-endpoint.com/webhook

Solutions:

# Renew expired certificates
certbot renew

# Fix certificate chain issues
# Ensure intermediate certificates are included

# Test with SSL Labs
# https://www.ssllabs.com/ssltest/

Scenario 3: Webhook signature verification failures​

Symptoms:

  • 401 Unauthorized responses
  • Error messages about invalid signatures
  • Webhook endpoint rejecting Hook0 requests

Diagnosis:

Add logging to compare expected vs received signature:

// Capture raw body for signature verification
app.post('/webhook', express.json({
  verify: (req, res, buf) => { req.rawBody = buf; }
}), (req, res) => {
  const signature = req.headers['x-hook0-signature'];
  // Use raw body for signature verification, not JSON.stringify(req.body)
  const rawBodyString = req.rawBody.toString('utf8');

  console.log('Received signature:', signature);
  console.log('Body length:', rawBodyString.length);
  console.log('Body preview:', rawBodyString.slice(0, 100));

  // Your verification logic here
  const isValid = verifySignature(rawBodyString, signature, secret);
  console.log('Signature valid:', isValid);

  if (!isValid) {
    console.error('Signature verification failed');
    return res.status(401).json({ error: 'Invalid signature' });
  }

  res.json({ status: 'processed' });
});

Common mistakes:

// ❌ Wrong - Using parsed body instead of raw
const computed = crypto.createHmac('sha256', secret)
  .update(JSON.stringify(req.body))  // Wrong if body already parsed
  .digest('hex');

// ✅ Correct - Use the raw request body, not the re-serialized parsed body
const rawBodyString = req.rawBody.toString('utf8');
const computed = crypto.createHmac('sha256', secret)
  .update(rawBodyString)
  .digest('hex');

Solutions:

  1. Use the raw request body for signature verification, never JSON.stringify(req.body)
  2. Ensure consistent character encoding (UTF-8)
  3. Verify you're using the correct subscription secret (fetch from API)
  4. Check HMAC algorithm (SHA256)
  5. Match Hook0's signature format exactly

See Implementing Webhook Authentication for complete implementation guide.

Scenario 4: Rate limiting​

Symptoms:

  • 429 Too Many Requests responses
  • Intermittent failures during high traffic

Diagnosis:

# Monitor request patterns - filter 429 responses
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&subscription_id={SUBSCRIPTION_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN" | \
  jq '.[] | select(.failed_at != null)'

Solutions:

// Implement rate limiting on your endpoint
const rateLimit = require('express-rate-limit');

const webhookLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 1000, // limit each IP to 1000 requests per windowMs
  message: 'Too many webhook requests',
  standardHeaders: true,
  legacyHeaders: false,
});

app.use('/webhooks', webhookLimiter);

Scenario 5: Internal server errors (5xx)​

Symptoms:

  • 500, 502, 503, 504 responses
  • Generic error messages

Diagnosis:

// Add comprehensive logging to your webhook handler
app.post('/webhook', (req, res) => {
  const startTime = Date.now();
  
  try {
    console.log('Webhook received:', {
      timestamp: new Date().toISOString(),
      headers: req.headers,
      body: req.body
    });
    
    // Your webhook processing logic
    processWebhook(req.body);
    
    const duration = Date.now() - startTime;
    console.log('Webhook processed successfully:', { duration });
    
    res.json({ status: 'processed' });
  } catch (error) {
    const duration = Date.now() - startTime;
    console.error('Webhook processing failed:', {
      error: error.message,
      stack: error.stack,
      duration
    });
    
    res.status(500).json({ error: 'Internal server error' });
  }
});

Solutions:

  • Add proper error handling and logging
  • Monitor application performance and resources
  • Implement health checks
  • Use application monitoring tools (APM)
Quick debugging with Hook0 Play

Hook0 Play lets you inspect raw request headers, body, and method in real-time. Use it to verify what your endpoint actually receives before diving into code.

Step 4: Setting up webhook debugging​

Create a debug webhook endpoint​

// debug-webhook.js
const express = require('express');
const crypto = require('crypto');
const fs = require('fs');
const app = express();

// Middleware to log all requests
app.use((req, res, next) => {
  const timestamp = new Date().toISOString();
  const logEntry = {
    timestamp,
    method: req.method,
    url: req.url,
    headers: req.headers,
    query: req.query,
    ip: req.ip
  };
  
  console.log('Request received:', JSON.stringify(logEntry, null, 2));
  next();
});

// Capture raw body for signature verification, then parse JSON
// This is critical: JSON.stringify(req.body) may differ from the original payload
app.use('/webhook', express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

// Note: Express.js normalizes all header names to lowercase
app.post('/webhook', (req, res) => {
  const timestamp = new Date().toISOString();
  const signature = req.headers['x-hook0-signature'];
  // Use raw body for signature verification, not JSON.stringify(req.body)
  const rawBodyString = req.rawBody.toString('utf8');

  const debugInfo = {
    timestamp,
    signature,
    bodyLength: rawBodyString.length,
    bodyPreview: rawBodyString.slice(0, 200),
    headers: req.headers
  };

  console.log('Webhook debug info:', JSON.stringify(debugInfo, null, 2));

  // Save to file for analysis
  fs.appendFileSync('webhook-debug.log', JSON.stringify({
    ...debugInfo,
    fullBody: rawBodyString
  }) + '\n');

  // Always respond successfully for debugging
  res.json({
    status: 'debug_received',
    timestamp,
    bodyLength: rawBodyString.length
  });
});

app.listen(3000, () => {
  console.log('Debug webhook server running on port 3000');
});

Use ngrok for local testing​

# Install ngrok
npm install -g ngrok

# Start your debug server
node debug-webhook.js

# In another terminal, expose it
ngrok http 3000

# Use the ngrok URL in your Hook0 subscription
# https://abc123.ngrok.io/webhook

Step 5: Automated failure detection​

Monitor failure rates with a script​

// monitor-failures.js
const fetch = require('node-fetch');

const HOOK0_TOKEN = '{YOUR_TOKEN}';
const APP_ID = '{APP_ID}';
const SUBSCRIPTION_ID = '{SUBSCRIPTION_ID}';

async function getFailureRate(subscriptionId, hours = 24) {
  // Get request attempts, optionally filtered by subscription
  const url = subscriptionId
    ? `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}&subscription_id=${subscriptionId}`
    : `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}`;

  const response = await fetch(url, {
    headers: {
      'Authorization': `Bearer ${HOOK0_TOKEN}`
    }
  });

  const attempts = await response.json();

  const total = attempts.length;
  // failed_at is set (non-null) only when a delivery attempt has failed
  const failed = attempts.filter(a => a.failed_at !== null).length;
  const failureRate = total > 0 ? (failed / total) * 100 : 0;

  return { total, failed, failureRate, attempts: attempts.slice(0, 5) };
}

async function monitorSubscriptions() {
  try {
    const stats = await getFailureRate(SUBSCRIPTION_ID);

    console.log(`Failure Rate: ${stats.failureRate.toFixed(2)}%`);
    console.log(`Total Attempts: ${stats.total}`);
    console.log(`Failed Attempts: ${stats.failed}`);

    if (stats.failureRate > 10) {
      console.warn('⚠️ High failure rate detected!');

      // Show recent failures (failed_at is set when delivery failed)
      const recentFailures = stats.attempts.filter(a => a.failed_at !== null);
      console.log('Recent failures:', recentFailures);
    }

  } catch (error) {
    console.error('Monitoring error:', error.message);
  }
}

// Run monitoring
monitorSubscriptions();

// Schedule regular monitoring
setInterval(monitorSubscriptions, 5 * 60 * 1000); // Every 5 minutes

Set up alerts​

// alert-system.js
const nodemailer = require('nodemailer');

const transporter = nodemailer.createTransport({
  host: 'smtp.your-domain.com',
  port: 587,
  auth: {
    user: '[email protected]',
    pass: 'your-password'
  }
});

async function sendAlert(subject, message) {
  await transporter.sendMail({
    from: '[email protected]',
    to: '[email protected]',
    subject: `Hook0 Alert: ${subject}`,
    text: message,
    html: `<pre>${message}</pre>`
  });
}

async function checkAndAlert() {
  const stats = await getFailureRate(SUBSCRIPTION_ID);

  if (stats.failureRate > 20) {
    const failedAttempts = stats.attempts.filter(a => a.failed_at !== null);
    await sendAlert('High Webhook Failure Rate', `
Failure Rate: ${stats.failureRate.toFixed(2)}%
Total Attempts: ${stats.total}
Failed Attempts: ${stats.failed}

Recent failures:
${JSON.stringify(failedAttempts, null, 2)}
    `);
  }
}

Step 6: Recovery strategies​

Manual retry of failed events​

# Get events for your application
curl "$HOOK0_API/events/?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Replay a specific event
curl -X POST "$HOOK0_API/events/{EVENT_ID}/replay" \
  -H "Authorization: Bearer $HOOK0_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "application_id": "'"$APP_ID"'"
  }'

Bulk retry script​

// retry-failed.js
const HOOK0_TOKEN = '{YOUR_TOKEN}';
const APP_ID = '{APP_ID}';

async function retryFailedEvents(maxAge = 24) {
  // Get failed request attempts
  const response = await fetch(
    `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}`,
    {
      headers: { 'Authorization': `Bearer ${HOOK0_TOKEN}` }
    }
  );

  const attempts = await response.json();

  // Filter failed attempts (failed_at is set when delivery failed)
  const failedAttempts = attempts.filter(a => a.failed_at !== null);

  // Get unique event IDs
  const uniqueEvents = [...new Set(failedAttempts.map(a => a.event_id))];

  console.log(`Found ${uniqueEvents.length} failed events to retry`);

  for (const eventId of uniqueEvents) {
    try {
      const retryResponse = await fetch(
        `http://localhost:8081/api/v1/events/${eventId}/replay`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${HOOK0_TOKEN}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({ application_id: APP_ID })
        }
      );

      if (retryResponse.ok) {
        console.log(`✅ Replayed event: ${eventId}`);
      } else {
        const error = await retryResponse.text();
        console.log(`❌ Failed to replay event: ${eventId} - ${error}`);
      }

      // Rate limit retries
      await new Promise(resolve => setTimeout(resolve, 100));
    } catch (error) {
      console.error(`Error replaying event ${eventId}:`, error.message);
    }
  }
}

retryFailedEvents();

Step 7: Prevention strategies​

Circuit breaker pattern​

// circuit-breaker.js
class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.threshold = threshold;
    this.timeout = timeout;
    this.failureCount = 0;
    this.lastFailure = null;
    this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
  }
  
  async call(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailure < this.timeout) {
        throw new Error('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
  
  onSuccess() {
    this.failureCount = 0;
    this.state = 'CLOSED';
  }
  
  onFailure() {
    this.failureCount++;
    this.lastFailure = Date.now();
    
    if (this.failureCount >= this.threshold) {
      this.state = 'OPEN';
    }
  }
}

// Use in webhook handler
const circuitBreaker = new CircuitBreaker(3, 30000);

app.post('/webhook', async (req, res) => {
  try {
    await circuitBreaker.call(async () => {
      await processWebhook(req.body);
    });
    
    res.json({ status: 'processed' });
  } catch (error) {
    console.error('Circuit breaker prevented processing:', error.message);
    res.status(503).json({ error: 'Service temporarily unavailable' });
  }
});

Graceful degradation​

// graceful-degradation.js
app.post('/webhook', async (req, res) => {
  try {
    // Try primary processing
    await processPrimary(req.body);
    res.json({ status: 'processed' });
  } catch (primaryError) {
    console.warn('Primary processing failed, trying fallback:', primaryError.message);
    
    try {
      // Try fallback processing
      await processFallback(req.body);
      res.json({ status: 'processed_fallback' });
    } catch (fallbackError) {
      console.error('Both primary and fallback failed:', fallbackError.message);
      
      // Store for later processing
      await storeForRetry(req.body);
      res.status(202).json({ status: 'queued_for_retry' });
    }
  }
});

Best practices​

Endpoint design​

  • ✅ Return appropriate HTTP status codes
  • ✅ Respond within 30 seconds
  • ✅ Implement idempotency
  • ✅ Use structured error responses
  • ✅ Add comprehensive logging

Error handling​

  • ✅ Distinguish between temporary and permanent failures
  • ✅ Rely on Hook0's automatic retries with predefined delays
  • ✅ Use circuit breakers for external dependencies
  • ✅ Monitor and alert on high failure rates
  • ✅ Provide detailed error messages

Testing​

  • ✅ Test webhook endpoints thoroughly
  • ✅ Simulate failure scenarios
  • ✅ Verify signature validation
  • ✅ Test with various payload sizes
  • ✅ Monitor performance under load

Troubleshooting checklist​

When webhook deliveries fail, work through this systematic checklist:

Infrastructure​

  • Endpoint is accessible from internet
  • SSL certificate is valid and not expired
  • DNS resolution works correctly
  • Firewall allows incoming connections
  • Load balancer is configured correctly

Application​

  • Webhook handler is properly implemented
  • Signature verification is working
  • Response times are under 30 seconds
  • Error handling returns appropriate status codes
  • Logging captures enough detail for debugging

Hook0 Configuration​

  • Subscription is enabled and active (see Events Not Being Delivered)
  • Event types match what you're sending
  • Target URL is correct
  • Custom headers are properly configured
  • Retry configuration is appropriate