Skip to main content

Debugging failed webhook deliveries

This guide walks through identifying, diagnosing, and fixing webhook delivery failures in Hook0. Each failed delivery is tracked as a request attempt.

General Troubleshooting

For API errors, connection issues, and authentication problems, see Troubleshooting Guide. This guide focuses specifically on webhook delivery failures.

Quick diagnosis checklist​

Before diving deep, run through this quick checklist:

  • Is the webhook endpoint accessible?
  • Are you returning the correct HTTP status codes?
  • Is the endpoint processing requests within timeout limits?
  • Have you verified the webhook signature?
  • Is your subscription properly configured?
  • Is your subscription enabled? (see Troubleshooting Guide - Events Not Being Delivered)

Understanding webhook failures​

Hook0 categorizes delivery failures by root cause:

HTTP status code categories​

2xx - Success

Request processed successfully. No retry needed.

4xx - Client Errors
  • 400-407, 409-499: Permanent failures, no retry
  • 408 (Timeout), 429 (Rate Limited): Temporary failures, will retry
5xx - Server Errors

All 5xx codes: Temporary failures, will retry. Suggests issues with your webhook endpoint.

Network errors
  • Connection timeouts
  • DNS resolution failures
  • Connection refused

Set up environment variables​

# Set your service token (from dashboard)
export HOOK0_TOKEN="YOUR_TOKEN_HERE"
export HOOK0_API="https://app.hook0.com/api/v1" # Replace by your domain (or http://localhost:8081 locally)

# Set your application ID (shown in dashboard URL or application details)
export APP_ID="YOUR_APPLICATION_ID_HERE"

Save these values:

# Save to .env file for later use
cat > .env <<EOF
HOOK0_TOKEN=$HOOK0_TOKEN
HOOK0_API=$HOOK0_API
APP_ID=$APP_ID
EOF

Step 1: Access Hook0 dashboard diagnostics​

  1. Go to your Hook0 Dashboard
  2. Select your Application
  3. Click on "Events"
  4. Find the failed event and click "View Details"
  5. Review "Request Attempts" section

Analyze delivery attempts​

A request attempt records where a delivery stands and how it got there:

{
  "request_attempt_id": "b3f1c2a4-5d6e-47f8-9a0b-1c2d3e4f5a6b",
  "event_id": "9f8e7d6c-5b4a-4938-8271-6a5b4c3d2e1f",
  "event": {
    "event_id": "9f8e7d6c-5b4a-4938-8271-6a5b4c3d2e1f",
    "event_type_name": "billing.invoice.paid"
  },
  "subscription": {
    "subscription_id": "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d",
    "description": "Billing service webhook"
  },
  "created_at": "2024-01-15T10:30:00Z",
  "picked_at": "2024-01-15T10:30:01Z",
  "failed_at": "2024-01-15T10:30:06Z",
  "succeeded_at": null,
  "delay_until": "2024-01-15T10:34:00Z",
  "response_id": "7c6b5a49-3827-4160-9e5d-4c3b2a1f0e9d",
  "retry_count": 3,
  "http_response_status": 500,
  "status": {
    "type": "waiting",
    "since": "2024-01-15T10:30:06Z",
    "until": "2024-01-15T10:34:00Z"
  }
}

http_response_status and retry_count tell you what happened and how many times Hook0 has tried. The response body, headers, and error name are not on the attempt: it links to them through response_id. Follow that id to GET /responses/{response_id} (Step 2 below) to read what your endpoint actually returned.

Step 2: Using the API for analysis​

Get request attempts via API​

# Get all request attempts for an application
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Filter by event
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&event_id={EVENT_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Filter by subscription
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&subscription_id={SUBSCRIPTION_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

Get response details​

# Get the response body and headers for a failed attempt
curl "$HOOK0_API/responses/{RESPONSE_ID}?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

Step 3: Common failure scenarios​

Scenario 1: Connection timeouts​

Symptoms:

  • Error message: "Connection timeout"
  • No HTTP status code
  • High duration_ms values

Diagnosis:

# Test endpoint connectivity
curl -v --max-time 30 https://your-webhook-endpoint.com/webhook

# Check DNS resolution
nslookup your-webhook-endpoint.com

# Test from different networks
curl -v https://your-webhook-endpoint.com/webhook

Solutions:

  • Optimize webhook processing to respond faster
  • Increase server resources
  • Check network connectivity
  • Verify DNS configuration

Scenario 2: SSL/TLS certificate issues​

Symptoms:

  • Error message: "SSL certificate verification failed"
  • Connection errors for HTTPS endpoints

Diagnosis:

# Check SSL certificate
openssl s_client -connect your-domain.com:443 -servername your-domain.com

# Verify certificate chain
curl -v https://your-webhook-endpoint.com/webhook

Solutions:

# Renew expired certificates
certbot renew

# Fix certificate chain issues
# Ensure intermediate certificates are included

# Test with SSL Labs
# https://www.ssllabs.com/ssltest/

Scenario 3: Webhook signature verification failures​

Symptoms:

  • 401 Unauthorized responses
  • Error messages about invalid signatures
  • Webhook endpoint rejecting Hook0 requests

Diagnosis:

Add logging to compare expected vs received signature:

// Capture raw body for signature verification
app.post('/webhook', express.json({
  verify: (req, res, buf) => { req.rawBody = buf; }
}), (req, res) => {
  const signature = req.headers['x-hook0-signature'];
  // Use raw body for signature verification, not JSON.stringify(req.body)
  const rawBodyString = req.rawBody.toString('utf8');

  console.log('Received signature:', signature);
  console.log('Body length:', rawBodyString.length);
  console.log('Body preview:', rawBodyString.slice(0, 100));

  // Your verification logic here
  const isValid = verifySignature(rawBodyString, signature, secret);
  console.log('Signature valid:', isValid);

  if (!isValid) {
    console.error('Signature verification failed');
    return res.status(401).json({ error: 'Invalid signature' });
  }

  res.json({ status: 'processed' });
});

Common mistakes:

// ❌ Wrong - Using parsed body instead of raw
const computed = crypto.createHmac('sha256', secret)
  .update(JSON.stringify(req.body))  // Wrong if body already parsed
  .digest('hex');

// ✅ Correct - Use the raw request body, not the re-serialized parsed body
const rawBodyString = req.rawBody.toString('utf8');
const computed = crypto.createHmac('sha256', secret)
  .update(rawBodyString)
  .digest('hex');

Solutions:

  1. Use the raw request body for signature verification, never JSON.stringify(req.body)
  2. Ensure consistent character encoding (UTF-8)
  3. Verify you're using the correct subscription secret (fetch from API)
  4. Check HMAC algorithm (SHA256)
  5. Match Hook0's signature format exactly

See Implementing Webhook Authentication for complete implementation guide.

Scenario 4: Rate limiting​

Symptoms:

  • 429 Too Many Requests responses
  • Intermittent failures during high traffic

Diagnosis:

# Monitor request patterns - filter 429 responses
curl "$HOOK0_API/request_attempts/?application_id=$APP_ID&subscription_id={SUBSCRIPTION_ID}" \
  -H "Authorization: Bearer $HOOK0_TOKEN" | \
  jq '.[] | select(.failed_at != null)'

Solutions:

// Implement rate limiting on your endpoint
const rateLimit = require('express-rate-limit');

const webhookLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 1000, // limit each IP to 1000 requests per windowMs
  message: 'Too many webhook requests',
  standardHeaders: true,
  legacyHeaders: false,
});

app.use('/webhooks', webhookLimiter);

Scenario 5: Internal server errors (5xx)​

Symptoms:

  • 500, 502, 503, 504 responses
  • Generic error messages

Diagnosis:

// Add comprehensive logging to your webhook handler
app.post('/webhook', (req, res) => {
  const startTime = Date.now();
  
  try {
    console.log('Webhook received:', {
      timestamp: new Date().toISOString(),
      headers: req.headers,
      body: req.body
    });
    
    // Your webhook processing logic
    processWebhook(req.body);
    
    const duration = Date.now() - startTime;
    console.log('Webhook processed successfully:', { duration });
    
    res.json({ status: 'processed' });
  } catch (error) {
    const duration = Date.now() - startTime;
    console.error('Webhook processing failed:', {
      error: error.message,
      stack: error.stack,
      duration
    });
    
    res.status(500).json({ error: 'Internal server error' });
  }
});

Solutions:

  • Add proper error handling and logging
  • Monitor application performance and resources
  • Implement health checks
  • Use application monitoring tools (APM)
Quick debugging with Hook0 Play

Hook0 Play lets you inspect raw request headers, body, and method in real-time. Use it to verify what your endpoint actually receives before diving into code.

Step 4: Setting up webhook debugging​

Create a debug webhook endpoint​

// debug-webhook.js
const express = require('express');
const crypto = require('crypto');
const fs = require('fs');
const app = express();

// Middleware to log all requests
app.use((req, res, next) => {
  const timestamp = new Date().toISOString();
  const logEntry = {
    timestamp,
    method: req.method,
    url: req.url,
    headers: req.headers,
    query: req.query,
    ip: req.ip
  };
  
  console.log('Request received:', JSON.stringify(logEntry, null, 2));
  next();
});

// Capture raw body for signature verification, then parse JSON
// This is critical: JSON.stringify(req.body) may differ from the original payload
app.use('/webhook', express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

// Note: Express.js normalizes all header names to lowercase
app.post('/webhook', (req, res) => {
  const timestamp = new Date().toISOString();
  const signature = req.headers['x-hook0-signature'];
  // Use raw body for signature verification, not JSON.stringify(req.body)
  const rawBodyString = req.rawBody.toString('utf8');

  const debugInfo = {
    timestamp,
    signature,
    bodyLength: rawBodyString.length,
    bodyPreview: rawBodyString.slice(0, 200),
    headers: req.headers
  };

  console.log('Webhook debug info:', JSON.stringify(debugInfo, null, 2));

  // Save to file for analysis
  fs.appendFileSync('webhook-debug.log', JSON.stringify({
    ...debugInfo,
    fullBody: rawBodyString
  }) + '\n');

  // Always respond successfully for debugging
  res.json({
    status: 'debug_received',
    timestamp,
    bodyLength: rawBodyString.length
  });
});

app.listen(3000, () => {
  console.log('Debug webhook server running on port 3000');
});

Use ngrok for local testing​

# Install ngrok
npm install -g ngrok

# Start your debug server
node debug-webhook.js

# In another terminal, expose it
ngrok http 3000

# Use the ngrok URL in your Hook0 subscription
# https://abc123.ngrok.io/webhook

Step 5: Automated failure detection​

Monitor failure rates with a script​

// monitor-failures.js
const fetch = require('node-fetch');

const HOOK0_TOKEN = '{YOUR_TOKEN}';
const APP_ID = '{APP_ID}';
const SUBSCRIPTION_ID = '{SUBSCRIPTION_ID}';

async function getFailureRate(subscriptionId, hours = 24) {
  // Get request attempts, optionally filtered by subscription
  const url = subscriptionId
    ? `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}&subscription_id=${subscriptionId}`
    : `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}`;

  const response = await fetch(url, {
    headers: {
      'Authorization': `Bearer ${HOOK0_TOKEN}`
    }
  });

  const attempts = await response.json();

  const total = attempts.length;
  // failed_at is set (non-null) only when a delivery attempt has failed
  const failed = attempts.filter(a => a.failed_at !== null).length;
  const failureRate = total > 0 ? (failed / total) * 100 : 0;

  return { total, failed, failureRate, attempts: attempts.slice(0, 5) };
}

async function monitorSubscriptions() {
  try {
    const stats = await getFailureRate(SUBSCRIPTION_ID);

    console.log(`Failure Rate: ${stats.failureRate.toFixed(2)}%`);
    console.log(`Total Attempts: ${stats.total}`);
    console.log(`Failed Attempts: ${stats.failed}`);

    if (stats.failureRate > 10) {
      console.warn('⚠️ High failure rate detected!');

      // Show recent failures (failed_at is set when delivery failed)
      const recentFailures = stats.attempts.filter(a => a.failed_at !== null);
      console.log('Recent failures:', recentFailures);
    }

  } catch (error) {
    console.error('Monitoring error:', error.message);
  }
}

// Run monitoring
monitorSubscriptions();

// Schedule regular monitoring
setInterval(monitorSubscriptions, 5 * 60 * 1000); // Every 5 minutes

Set up alerts​

// alert-system.js
const nodemailer = require('nodemailer');

const transporter = nodemailer.createTransport({
  host: 'smtp.your-domain.com',
  port: 587,
  auth: {
    user: '[email protected]',
    pass: 'your-password'
  }
});

async function sendAlert(subject, message) {
  await transporter.sendMail({
    from: '[email protected]',
    to: '[email protected]',
    subject: `Hook0 Alert: ${subject}`,
    text: message,
    html: `<pre>${message}</pre>`
  });
}

async function checkAndAlert() {
  const stats = await getFailureRate(SUBSCRIPTION_ID);

  if (stats.failureRate > 20) {
    const failedAttempts = stats.attempts.filter(a => a.failed_at !== null);
    await sendAlert('High Webhook Failure Rate', `
Failure Rate: ${stats.failureRate.toFixed(2)}%
Total Attempts: ${stats.total}
Failed Attempts: ${stats.failed}

Recent failures:
${JSON.stringify(failedAttempts, null, 2)}
    `);
  }
}

Step 6: Recovery strategies​

Manual retry of failed events​

# Get events for your application
curl "$HOOK0_API/events/?application_id=$APP_ID" \
  -H "Authorization: Bearer $HOOK0_TOKEN"

# Replay a specific event
curl -X POST "$HOOK0_API/events/{EVENT_ID}/replay" \
  -H "Authorization: Bearer $HOOK0_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "application_id": "'"$APP_ID"'"
  }'

Bulk retry script​

// retry-failed.js
const HOOK0_TOKEN = '{YOUR_TOKEN}';
const APP_ID = '{APP_ID}';

async function retryFailedEvents(maxAge = 24) {
  // Get failed request attempts
  const response = await fetch(
    `http://localhost:8081/api/v1/request_attempts/?application_id=${APP_ID}`,
    {
      headers: { 'Authorization': `Bearer ${HOOK0_TOKEN}` }
    }
  );

  const attempts = await response.json();

  // Filter failed attempts (failed_at is set when delivery failed)
  const failedAttempts = attempts.filter(a => a.failed_at !== null);

  // Get unique event IDs
  const uniqueEvents = [...new Set(failedAttempts.map(a => a.event_id))];

  console.log(`Found ${uniqueEvents.length} failed events to retry`);

  for (const eventId of uniqueEvents) {
    try {
      const retryResponse = await fetch(
        `http://localhost:8081/api/v1/events/${eventId}/replay`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${HOOK0_TOKEN}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({ application_id: APP_ID })
        }
      );

      if (retryResponse.ok) {
        console.log(`✅ Replayed event: ${eventId}`);
      } else {
        const error = await retryResponse.text();
        console.log(`❌ Failed to replay event: ${eventId} - ${error}`);
      }

      // Rate limit retries
      await new Promise(resolve => setTimeout(resolve, 100));
    } catch (error) {
      console.error(`Error replaying event ${eventId}:`, error.message);
    }
  }
}

retryFailedEvents();

Step 7: Prevention strategies​

Circuit breaker pattern​

// circuit-breaker.js
class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.threshold = threshold;
    this.timeout = timeout;
    this.failureCount = 0;
    this.lastFailure = null;
    this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
  }
  
  async call(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailure < this.timeout) {
        throw new Error('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
  
  onSuccess() {
    this.failureCount = 0;
    this.state = 'CLOSED';
  }
  
  onFailure() {
    this.failureCount++;
    this.lastFailure = Date.now();
    
    if (this.failureCount >= this.threshold) {
      this.state = 'OPEN';
    }
  }
}

// Use in webhook handler
const circuitBreaker = new CircuitBreaker(3, 30000);

app.post('/webhook', async (req, res) => {
  try {
    await circuitBreaker.call(async () => {
      await processWebhook(req.body);
    });
    
    res.json({ status: 'processed' });
  } catch (error) {
    console.error('Circuit breaker prevented processing:', error.message);
    res.status(503).json({ error: 'Service temporarily unavailable' });
  }
});

Graceful degradation​

// graceful-degradation.js
app.post('/webhook', async (req, res) => {
  try {
    // Try primary processing
    await processPrimary(req.body);
    res.json({ status: 'processed' });
  } catch (primaryError) {
    console.warn('Primary processing failed, trying fallback:', primaryError.message);
    
    try {
      // Try fallback processing
      await processFallback(req.body);
      res.json({ status: 'processed_fallback' });
    } catch (fallbackError) {
      console.error('Both primary and fallback failed:', fallbackError.message);
      
      // Store for later processing
      await storeForRetry(req.body);
      res.status(202).json({ status: 'queued_for_retry' });
    }
  }
});

Best practices​

Endpoint design​

  • ✅ Return appropriate HTTP status codes
  • ✅ Respond within 30 seconds
  • ✅ Implement idempotency
  • ✅ Use structured error responses
  • ✅ Add comprehensive logging

Error handling​

  • ✅ Distinguish between temporary and permanent failures
  • ✅ Rely on Hook0's automatic retries with predefined delays
  • ✅ Use circuit breakers for external dependencies
  • ✅ Monitor and alert on high failure rates
  • ✅ Provide detailed error messages

Testing​

  • ✅ Test webhook endpoints thoroughly
  • ✅ Simulate failure scenarios
  • ✅ Verify signature validation
  • ✅ Test with various payload sizes
  • ✅ Monitor performance under load

Troubleshooting checklist​

When webhook deliveries fail, work through this systematic checklist:

Infrastructure​

  • Endpoint is accessible from internet
  • SSL certificate is valid and not expired
  • DNS resolution works correctly
  • Firewall allows incoming connections
  • Load balancer is configured correctly

Application​

  • Webhook handler is properly implemented
  • Signature verification is working
  • Response times are under 30 seconds
  • Error handling returns appropriate status codes
  • Logging captures enough detail for debugging

Hook0 Configuration​

  • Subscription is enabled and active (see Events Not Being Delivered)
  • Event types match what you're sending
  • Target URL is correct
  • Custom headers are properly configured
  • Retry configuration is appropriate