Building Distributed Webhook Consumers with Redis Queues

What You’ll Need

To follow along with this tutorial, you will need the following tools and infrastructure:

• A server instance hosted on a Hetzner VPS or Contabo VPS running Ubuntu 22.04 LTS
• An alternative cloud host like DigitalOcean if you prefer managed droplets
• Node.js (v18 or higher) and npm installed on your development machine or server
• A running Redis instance (v6.x or higher)
• A domain name registered via Namecheap for public webhook routing
• An orchestration platform like n8n Cloud if you plan to plug consumer outputs directly into existing automation flows

Table of Contents

• Architecting a High-Throughput Webhook Ingestion Engine
• Building the Ingestion API with Express and Redis BullMQ
• Implementing Distributed Worker Processes with Idempotency
• Securing and Scaling Your Queue Infrastructure
• Getting Started

Architecting a High-Throughput Webhook Ingestion Engine

When third-party providers like Stripe, GitHub, or Shopify fire webhooks at your infrastructure, they expect an HTTP 200 OK status code within a strict timeout window (often 2 to 5 seconds). Processing heavy computational workflows, mutating database records, or sending notification emails directly inside the web request handler is a recipe for failure. Under traffic spikes, your application threads freeze, request queues fill up, and external vendors mark your endpoint as unresponsive, triggering aggressive retry loops that crash your system.

To build a enterprise-grade system, you must decouple request acceptance from task execution. The receiving API endpoint acts purely as a producer: it validates the cryptographic signature of the payload, pushes the raw event data onto an asynchronous queue, and returns an HTTP 202 Accepted response immediately. Distributed worker processes running across multiple hosts consume items from this queue out-of-band.

When deciding between task platforms, engineers often debate Temporal vs n8n for Enterprise Workflow Automation to handle backend orchestration. However, when you need ultra-low latency and custom queue lifecycle management, pairing Node.js with Redis and BullMQ offers the tightest control over memory overhead and execution context. Redis provides an in-memory datastructure store that handles tens of thousands of read and write operations per second, making it the perfect backplane for fast job buffering.

The core design requirements for a distributed webhook consumer system include:

1. Sub-50ms Response Times: Immediate response acknowledging payload receipt.
2. At-Least-Once Delivery Guarantees: Ensures no event is permanently lost if a worker process crashes mid-execution.
3. Strict Idempotency: Prevents duplicate processing of incoming events using unique idempotency keys.
4. Dead Letter Queue (DLQ) Handling: Captures permanently failing jobs for manual inspection without halting the queue processing pipeline.

Building the Ingestion API with Express and Redis BullMQ

Let us build the ingestion server using Express and BullMQ. In this step, deploy a lightweight Node.js web server hosted on a Hetzner VPS. The server verifies the HMAC signature of incoming webhooks to ensure authenticity, structures the job object, and places it inside a Redis-backed queue.

First, initialize your project and install the required dependencies:

mkdir webhook-engine
cd webhook-engine
npm init -y
npm install express bullmq ioredis dotenv crypto

Next, create the main entry point file named server.js. This code defines our producer instance, establishes a connection pool to Redis, and exposes a high-throughput endpoint.

const express = require('express');
const { Queue } = require('bullmq');
const Redis = require('ioredis');
const crypto = require('crypto');
require('dotenv').config();

const app = express();
app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

const redisConnection = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
  password: process.env.REDIS_PASSWORD || undefined,
  maxRetriesPerRequest: null,
});

const webhookQueue = new Queue('incoming-webhooks', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000,
    },
    removeOnComplete: {
      age: 86400,
      count: 10000,
    },
    removeOnFail: {
      age: 604800,
    },
  },
});

function verifySignature(req) {
  const secret = process.env.WEBHOOK_SECRET || 'supersecretkey';
  const signature = req.headers['x-signature'];
  if (!signature) {
    return false;
  }
  const hmac = crypto.createHmac('sha256', secret);
  const digest = hmac.update(req.rawBody).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(digest));
}

app.post('/api/v1/webhooks', async (req, res) => {
  const startTime = Date.now();
  
  if (!verifySignature(req)) {
    return res.status(401).json({ error: 'Invalid HMAC signature' });
  }

  const eventType = req.body.event_type;
  const eventId = req.body.id;

  if (!eventType || !eventId) {
    return res.status(400).json({ error: 'Missing required event fields' });
  }

  try {
    await webhookQueue.add(
      eventType,
      {
        eventId: eventId,
        payload: req.body,
        receivedAt: new Date().toISOString(),
      },
      {
        jobId: eventId,
      }
    );

    const processingTimeMs = Date.now() - startTime;
    return res.status(202).json({
      status: 'queued',
      jobId: eventId,
      latencyMs: processingTimeMs,
    });
  } catch (error) {
    return res.status(500).json({ error: 'Failed to enqueue webhook payload' });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook Ingestion Server active on port ${PORT}`);
});

This HTTP server accepts post requests, verifies the cryptographic signature in a timing-safe manner, and offloads the data to Redis in under 10 milliseconds.

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Implementing Distributed Worker Processes with Idempotency

Once the ingestion API is enqueuing jobs, you need worker processes running across multiple nodes to pick up and process those jobs concurrently. Because network drops and worker restarts happen, duplicate processing can occur. To guard against double-charging accounts or firing duplicate notifications, workers must enforce idempotency using Redis key sets.

Create a worker file named worker.js. This script acts as a consumer process. You can spin up multiple instances of this process across different hosts, and BullMQ will manage concurrent task distribution automatically.

const { Worker } = require('bullmq');
const Redis = require('ioredis');
require('dotenv').config();

const redisConnection = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
  password: process.env.REDIS_PASSWORD || undefined,
  maxRetriesPerRequest: null,
});

async function processPaymentSucceeded(payload) {
  console.log(`Processing payment_succeeded for order: ${payload.data.order_id}`);
  return { status: 'processed', orderId: payload.data.order_id };
}

async function processUserCreated(payload) {
  console.log(`Processing user_created for email: ${payload.data.email}`);
  return { status: 'processed', userId: payload.data.user_id };
}

async function handleWebhookJob(job) {
  const { eventId, payload } = job.data;
  const lockKey = `lock:webhook:${eventId}`;
  
  const acquired = await redisConnection.set(lockKey, 'processing', 'NX', 'EX', 3600);
  
  if (!acquired) {
    console.log(`Duplicate event skipped: ${eventId}`);
    return { status: 'skipped', reason: 'already_processed' };
  }

  try {
    let result;
    switch (job.name) {
      case 'payment.succeeded':
        result = await processPaymentSucceeded(payload);
        break;
      case 'user.created':
        result = await processUserCreated(payload);
        break;
      default:
        console.warn(`Unhandled event type: ${job.name}`);
        result = { status: 'ignored', reason: 'unknown_event_type' };
    }
    return result;
  } catch (error) {
    await redisConnection.del(lockKey);
    throw error;
  }
}

const worker = new Worker('incoming-webhooks', handleWebhookJob, {
  connection: redisConnection,
  concurrency: 10,
});

worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed successfully. Result:`, result);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed after ${job.attemptsMade} attempts. Error: ${err.message}`);
});

console.log('Distributed Webhook Worker started listening for jobs...');

The worker uses atomic Redis SET operations with NX (Set if Not Exists) and EX (Expiration in seconds) flags. If two instances grab the exact same job execution call simultaneously, only one instance successfully obtains the key lock.

Securing and Scaling Your Queue Infrastructure

Deploying queue producers and worker pools in a production environment requires setting up firewalls, SSL termination, and reverse proxies. Never expose your raw Node.js port or Redis instance to the public internet.

First, set up strict network boundaries on your cloud instance. When configuring hosting environments on a Contabo VPS or Hetzner VPS, secure your exposed infrastructure endpoints by Configuring UFW Firewall Rules On Linux Servers. Restrict public inbound traffic strictly to ports 80 and 443, locking down port 6379 (Redis) so it only accepts connections from internal node IP addresses.

Execute these commands on your application server to configure UFW:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.0.0.0/16 to any port 6379
sudo ufw enable

Next, place Nginx in front of your Node.js application to handle SSL encryption, payload buffering, and domain termination. If you need step-by-step guidance on setting up HTTPS endpoint routing for your custom domain, consult our comprehensive guide on Configuring Nginx SSL Certificates for Custom Subdomains.

Use this Nginx configuration to proxy incoming traffic to your Express ingestion engine safely:

server {
    listen 80;
    server_name webhooks.yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name webhooks.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/webhooks.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/webhooks.yourdomain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    client_max_body_size 5M;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 10s;
        proxy_connect_timeout 5s;
    }
}

This proxy configuration bounds total request body sizes to 5 megabytes, drops stale connection requests quickly, forwards real client IP addresses for auditing, and redirects unencrypted HTTP requests to HTTPS seamlessly.

Getting Started

To spin up your own enterprise ingestion network today:

1. Provision an Ubuntu cloud host on a Hetzner VPS, Contabo VPS, or DigitalOcean.
2. Point a domain name purchased on Namecheap to your server’s static IP address.
3. Install Redis server and configure authentication passwords along with strict binding interfaces.
4. Clone your Express producer code and distributed BullMQ worker scripts onto your execution hosts.
5. Launch worker processes using a process supervisor like PM2 to maintain high operational uptime across server restarts.
6. Connect downstream webhooks directly to automation tools like n8n Cloud or custom backend microservices.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.