Designing Resilient Webhook Endpoints with Redis Queues

· 7 min read ·
Designing Resilient Webhook Endpoints with Redis Queues

What You’ll Need

Table of Contents

The Architecture of High-Throughput Webhook Ingestion

Receiving webhooks directly into a synchronous application server is a recipe for disaster. Third-party providers like Stripe, Shopify, or GitHub expect your API endpoint to respond with an HTTP 200 or 202 within seconds. If your database experiences a temporary lock, or if a downstream service experiences high latency, your application server will time out. The sender will assume your service is down, leading to retry storms, duplicate processing, or missed events entirely.

To build a resilient ingestion engine, you must strictly decouple receipt from processing. The primary rule of webhook design is simple: accept the payload, validate its signature, push it to an in-memory queue, and return an HTTP 202 Accepted status immediately.

                  +-------------------+
                  |  External Sender  |
                  +---------+---------+
                            |
                     1. HTTP POST Payload
                            v
            +---------------+---------------+
            | Express Ingestion Server      |
            | (Validates HMAC & Enqueues)   |
            +---------------+---------------+
                            |
                     2. Fast Redis Write
                            v
                  +---------+---------+
                  |    Redis Queue    |
                  +---------+---------+
                            |
                     3. Async Job Pull
                            v
            +---------------+---------------+
            | Node.js Queue Workers         |
            | (Idempotent Processing)       |
            +---------------+---------------+
                            |
                     4. Persistent Storage
                            v
                  +---------+---------+
                  | Database / APIs   |
                  +-------------------+

By placing an in-memory datastore like Redis between receipt and processing, your server can handle thousands of incoming requests per second without clogging application threads. If your core application goes down for maintenance, incoming webhooks remain safe inside the queue. To maintain visibility over this high-volume pipeline, I rely on Configuring Vector for Centralized Log Aggregation to capture operational logs across ingestion endpoints and queue workers.

Setting Up Node.js, Express, and BullMQ with Redis

Let us set up a production-ready Node.js stack using Express for our HTTP server, BullMQ for job queue management, and IORedis for direct caching operations. Provisioning this environment on a cheap cloud instance from Hetzner VPS gives us dedicated hardware specs capable of handling heavy payload volumes without memory throttle limits.

First, create a project directory and install the necessary dependencies:

mkdir webhook-queue-system
cd webhook-queue-system
npm init -y
npm install express bullmq ioredis dotenv

Create a package.json file to establish script commands and lock dependencies:

{
  "name": "webhook-queue-system",
  "version": "1.0.0",
  "description": "Production webhook receiver using Redis and BullMQ",
  "main": "src/server.js",
  "scripts": {
    "start:server": "node src/server.js",
    "start:worker": "node src/worker.js"
  },
  "dependencies": {
    "bullmq": "^5.1.0",
    "dotenv": "^16.4.5",
    "express": "^4.19.2",
    "ioredis": "^5.3.2"
  }
}

Now, create a configuration loader file in src/config.js to securely manage environment variables:

require('dotenv').config();

module.exports = {
  port: process.env.PORT || 3000,
  redis: {
    host: process.env.REDIS_HOST || 'localhost',
    port: parseInt(process.env.REDIS_PORT || '6379', 10),
    password: process.env.REDIS_PASSWORD || undefined
  },
  webhookSecret: process.env.WEBHOOK_SECRET || 'default_super_secret_signing_key_99'
};

Next, set up the BullMQ queue instance inside src/queue.js. This module instantiates the central queue and defines default retry behaviors, specifying exponential backoff delays:

const { Queue } = require('bullmq');
const config = require('./config');

const redisConnection = {
  host: config.redis.host,
  port: config.redis.port,
  password: config.redis.password
};

const webhookQueue = new Queue('webhook-ingestion', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000
    },
    removeOnComplete: {
      age: 86400,
      count: 5000
    },
    removeOnFail: {
      age: 604800
    }
  }
});

module.exports = {
  webhookQueue,
  redisConnection
};

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Writing the Webhook Receiver and Producer

The web server must be intentionally barebones. Its sole job is to capture the raw body, calculate cryptographic signatures to confirm payload authenticity, generate an idempotency key, push the event into Redis, and instantly return an HTTP 202 Accepted response.

This mechanism allows scrapers or external event producers, such as those discussed in my guide on Building Scalable Playwright Scrapers with Docker, to dump heavy data bursts directly into our pipeline without encountering web server dropouts.

Below is the complete implementation of src/server.js:

const express = require('express');
const crypto = require('crypto');
const config = require('./config');
const { webhookQueue } = require('./queue');

const app = express();

app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

function verifySignature(req) {
  const signature = req.headers['x-webhook-signature'];
  if (!signature) {
    return false;
  }
  
  const hmac = crypto.createHmac('sha256', config.webhookSecret);
  const digest = hmac.update(req.rawBody).digest('hex');
  
  return crypto.timingSafeEqual(
    Buffer.from(signature, 'utf8'),
    Buffer.from(digest, 'utf8')
  );
}

app.post('/webhook', async (req, res) => {
  try {
    const isValid = verifySignature(req);
    if (!isValid) {
      return res.status(401).json({ error: 'Invalid HMAC signature' });
    }

    const eventId = req.headers['x-event-id'] || req.body.id || crypto.randomUUID();
    const eventType = req.headers['x-event-type'] || req.body.type || 'unknown_event';

    const job = await webhookQueue.add(
      eventType,
      {
        eventId: eventId,
        payload: req.body,
        receivedAt: new Date().toISOString()
      },
      {
        jobId: eventId
      }
    );

    return res.status(202).json({
      status: 'accepted',
      message: 'Webhook enqueued successfully',
      jobId: job.id
    });
  } catch (error) {
    return res.status(500).json({
      error: 'Failed to enqueue webhook',
      details: error.message
    });
  }
});

app.get('/health', (req, res) => {
  res.status(200).json({ status: 'healthy' });
});

app.listen(config.port, () => {
  console.log(`Webhook ingestion server listening on port ${config.port}`);
});

Implementing the Resilient Worker and Exponential Backoff

The background worker operates in a separate thread or container. It polls Redis, pulls jobs, handles rate limits, checks for duplicated execution, and executes business logic.

To guarantee idempotency, we use Redis key setting with atomic timeouts (SET key value NX EX seconds). If a job with the exact same event ID was successfully processed within our lock window, the worker skips execution.

Here is the complete implementation of src/worker.js:

const { Worker } = require('bullmq');
const Redis = require('ioredis');
const config = require('./config');
const { redisConnection } = require('./queue');

const redisClient = new Redis(redisConnection);

async function processWebhookLogic(jobData) {
  const { eventId, payload } = jobData;
  
  const lockKey = `processed:${eventId}`;
  const isNew = await redisClient.set(lockKey, 'true', 'EX', 86400, 'NX');
  
  if (!isNew) {
    console.log(`Duplicate event detected: ${eventId}. Skipping execution.`);
    return { status: 'skipped', reason: 'duplicate' };
  }

  if (payload.simulateFailure) {
    throw new Error('Simulated upstream dependency failure.');
  }

  console.log(`Successfully processed event ${eventId} of type ${payload.type || 'generic'}`);
  return { status: 'processed', eventId: eventId };
}

const worker = new Worker(
  'webhook-ingestion',
  async (job) => {
    console.log(`Processing job ${job.id} (Attempt ${job.attemptsMade + 1})`);
    const result = await processWebhookLogic(job.data);
    return result;
  },
  {
    connection: redisConnection,
    concurrency: 10
  }
);

worker.on('completed', (job, returnvalue) => {
  console.log(`Job ${job.id} completed. Result:`, returnvalue);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed with error: ${err.message}. Retries left: ${job.opts.attempts - job.attemptsMade}`);
});

process.on('SIGINT', async () => {
  console.log('Shutting down worker process gracefully...');
  await worker.close();
  await redisClient.quit();
  process.exit(0);
});

Dockerizing the Ingestion Stack

Running the server, the Redis datastore, and background processing workers requires precise service isolation. We can define our service infrastructure using docker-compose.yml, which lets us scale worker containers dynamically based on queue depth.

To keep these containers continuously patched and operational in production environments without manual updates, consider How to Deploy Watchtower for Docker Containers alongside your Compose configuration.

Here is the full deployment topology in docker-compose.yml:

version: '3.8'

services:
  redis:
    image: redis:7.2-alpine
    container_name: webhook-redis
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes --requirepass supersecurepassword123
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "supersecurepassword123", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  ingestion-server:
    build: .
    container_name: webhook-server
    restart: always
    command: npm run start:server
    ports:
      - "3000:3000"
    environment:
      - PORT=3000
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_PASSWORD=supersecurepassword123
      - WEBHOOK_SECRET=my_production_hmac_secret_key
    depends_on:
      redis:
        condition: service_healthy

  queue-worker:
    build: .
    restart: always
    command: npm run start:worker
    deploy:
      replicas: 3
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_PASSWORD=supersecurepassword123
      - WEBHOOK_SECRET=my_production_hmac_secret_key
    depends_on:
      redis:
        condition: service_healthy

volumes:
  redis_data:

Create a dockerfile named Dockerfile in the repository root to build both services:

FROM node:20-alpine

WORKDIR /app

COPY package*.json ./

RUN npm ci --only=production

COPY . .

EXPOSE 3000

CMD ["npm", "run", "start:server"]

Build and launch the stack in detached mode:

docker compose up -d --build

You can scale workers horizontally on demand based on load metrics:

docker compose up -d --scale queue-worker=10

Getting Started

To launch your production webhook queueing infrastructure, spin up an instance on Hetzner VPS or DigitalOcean, acquire a TLS domain via Namecheap, and forward filtered data from BullMQ directly to external endpoints or n8n Cloud instances.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3

Related Guides

system online