Designing Idempotent Webhook Receivers for Distributed Systems

What You’ll Need

• n8n Cloud or self-hosted n8n
• Hetzner VPS or Contabo VPS for hosting
• DigitalOcean as alternative
• Namecheap if domain needed

Table of Contents

• Understanding At-Least-Once Webhook Delivery
• Anatomy of an Idempotent Architecture
• Implementing an Atomic Deduplication Lock with Redis
• Asynchronous Job Processing and Queue Worker Architecture
• Handling Distributed Edge Cases and Database State
• Getting Started

Understanding At-Least-Once Webhook Delivery

Every developer building integrations eventually encounters duplicate webhook events. Payment gateways, messaging APIs, and enterprise SaaS platforms design their notification infrastructure around “at-least-once” delivery semantics. Because network partitions, transient HTTP timeouts, and service restarts are inevitable in distributed systems, the webhook provider prioritizes reliability over uniqueness. If your API server takes 5001 milliseconds to respond to an HTTP POST request and the provider has a strict 5000 millisecond timeout, the provider marks the attempt as failed and schedules an automated retry.

If your receiver processed the business logic before timing out (for example, incrementing a user balance or triggering an shipment dispatch), that automated retry will cause a double-execution bug unless your endpoint is explicitly idempotent. An operation is idempotent if executing it multiple times produces the exact same system state as executing it once.

+------------------+         HTTP POST Event (ID: evt_123)        +-------------------+
|                  | -------------------------------------------> |                   |
|                  | <------------------------------------------- |                   |
|                  |      Timeout / Network Flake (504)           |                   |
|  Webhook Provider|                                              | Webhook Receiver  |
|  (e.g., Stripe)  |         HTTP POST Retry (ID: evt_123)        | (Your Application)|
|                  | -------------------------------------------> |                   |
|                  | <------------------------------------------- |                   |
+------------------+             200 OK (Processed)               +-------------------+

Building a resilient webhook ingestion pipeline requires shifting away from synchronous request-response models. You cannot process heavy workflows inside the HTTP lifecycle of the incoming webhook. Instead, you need a deterministic state engine that validates, records, locks, and defers processing.

Anatomy of an Idempotent Architecture

To construct an idempotent receiver, your architecture must combine cryptographic validation, short-term atomic locks, persistent event logs, and asynchronous worker queues.

When choosing storage backends for event state management, disk write latency becomes a massive bottleneck under heavy webhook throughput. You can read my guide on Optimizing Container Volume Performance On Hetzner VPS to ensure your underlying host handles high IOPS persistent logs without degrading server response times.

Here is the step-by-step lifecycle of an incoming idempotent webhook request:

1. Signature Verification: Validate the HMAC signature supplied in the request headers to ensure the payload was sent by an authorized provider and was not altered in transit.
2. Deduplication Check: Extract the unique event identifier (such as evt_123 or X-Webhook-ID). Perform an atomic check against a fast key-value cache like Redis to determine if this event ID is currently processing or already completed.
3. Immediate Acknowledgement: Store the raw payload in a fast persistent queue and return an HTTP 202 Accepted status back to the provider in under 100 milliseconds. This prevents provider retries caused by local processing delays.
4. Asynchronous Execution: Background worker processes consume the queued message, acquire a database-level transaction lock, execute the business domain logic, and update the event status to COMPLETED.

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Implementing an Atomic Deduplication Lock with Redis

Let us implement a production-grade webhook ingestion service using Node.js, Express, and Redis on a cloud server instance hosted on Hetzner VPS.

We will use the ioredis library to execute atomic SET key value NX PX operations. The NX option ensures the key is set only if it does not already exist, and PX sets an automatic expiration time in milliseconds to prevent deadlocks if a worker node crashes mid-execution.

Create a file named server.js and paste the full implementation below:

const express = require('express');
const crypto = require('crypto');
const Redis = require('ioredis');

const app = express();
const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
});

const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET || 'super-secret-key-12345';
const LOCK_TTL_MS = 30000;

app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

function verifySignature(req) {
  const signature = req.headers['x-webhook-signature'];
  if (!signature) {
    return false;
  }
  const expectedSignature = crypto
    .createHmac('sha256', WEBHOOK_SECRET)
    .update(req.rawBody)
    .digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expectedSignature));
}

app.post('/api/v1/webhooks', async (req, res) => {
  if (!verifySignature(req)) {
    return res.status(401).json({ error: 'Invalid HMAC signature' });
  }

  const eventId = req.headers['x-webhook-id'] || req.body.id;
  if (!eventId) {
    return res.status(400).json({ error: 'Missing unique webhook event ID' });
  }

  const lockKey = `webhook:lock:${eventId}`;
  const statusKey = `webhook:status:${eventId}`;

  const existingStatus = await redis.get(statusKey);
  if (existingStatus === 'COMPLETED') {
    return res.status(200).json({ status: 'ignored', reason: 'Event already processed' });
  }

  const acquiredLock = await redis.set(lockKey, 'LOCKED', 'NX', 'PX', LOCK_TTL_MS);
  if (!acquiredLock) {
    return res.status(202).json({ status: 'pending', reason: 'Event currently processing' });
  }

  try {
    const queuePayload = JSON.stringify({
      eventId: eventId,
      payload: req.body,
      receivedAt: new Date().toISOString()
    });

    await redis.rpush('webhook:queue', queuePayload);
    await redis.set(statusKey, 'PROCESSING', 'EX', 86400);

    return res.status(202).json({ status: 'queued', eventId: eventId });
  } catch (error) {
    await redis.del(lockKey);
    return res.status(500).json({ error: 'Internal server error queueing webhook' });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook receiver listening on port ${PORT}`);
});

In this implementation, the verifySignature function uses crypto.timingSafeEqual to eliminate timing attack vectors when verifying cryptographic hashes. If an identical webhook payload hits this endpoint concurrently, the atomic redis.set(..., 'NX') guarantees that only one request acquires the execution rights. Subsequent parallel calls immediately receive an HTTP 202 status informing them that the event is already pending or queued.

Asynchronous Job Processing and Queue Worker Architecture

Now that the HTTP ingestion server handles verification and atomic deduplication within milliseconds, we must build a dedicated worker process to consume jobs off the Redis list (webhook:queue) and perform robust, transactionally isolated tasks.

If you are orchestrating complex multi-container microservices on a remote Linux host, check out my guide on Deploying Telegram Bots Using Docker Compose to learn best practices for multi-service docker networks and systemd auto-restart policies. For enterprise-wide DAG workflows, you can also learn How to Self Host Kestra on Cloud VPS.

Here is the complete code for our background background execution node in worker.js:

const Redis = require('ioredis');
const { Pool } = require('pg');

const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
});

const pgPool = new Pool({
  connectionString: process.env.DATABASE_URL || 'postgres://user:pass@127.0.0.1:5432/app_db'
});

async function processWebhookEvent(eventData) {
  const client = await pgPool.connect();
  try {
    await client.query('BEGIN');

    const checkRes = await client.query(
      'SELECT id FROM processed_webhooks WHERE event_id = $1 FOR UPDATE',
      [eventData.eventId]
    );

    if (checkRes.rowCount > 0) {
      await client.query('ROLLBACK');
      return true;
    }

    const { userId, amount } = eventData.payload;
    await client.query(
      'UPDATE user_balances SET balance = balance + $1 WHERE user_id = $2',
      [amount, userId]
    );

    await client.query(
      'INSERT INTO processed_webhooks (event_id, processed_at) VALUES ($1, NOW())',
      [eventData.eventId]
    );

    await client.query('COMMIT');
    return true;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

async function startWorker() {
  console.log('Worker started. Listening for webhook jobs...');
  while (true) {
    try {
      const result = await redis.blpop('webhook:queue', 0);
      const rawMessage = result[1];
      const eventData = JSON.parse(rawMessage);

      const statusKey = `webhook:status:${eventData.eventId}`;
      const lockKey = `webhook:lock:${eventData.eventId}`;

      try {
        await processWebhookEvent(eventData);
        await redis.set(statusKey, 'COMPLETED', 'EX', 604800);
        await redis.del(lockKey);
        console.log(`Successfully processed event ${eventData.eventId}`);
      } catch (processingError) {
        console.error(`Error processing event ${eventData.eventId}:`, processingError);
        await redis.set(statusKey, 'FAILED', 'EX', 86400);
        await redis.del(lockKey);
        await redis.rpush('webhook:dead_letter_queue', JSON.stringify({
          eventData: eventData,
          error: processingError.message,
          failedAt: new Date().toISOString()
        }));
      }
    } catch (loopError) {
      console.error('Worker loop encountered an operational error:', loopError);
      await new Promise((resolve) => setTimeout(resolve, 1000));
    }
  }
}

startWorker();

To run this architecture seamlessly, deploy the complete stack using Docker Compose. Create a docker-compose.yml file to bind the app, the worker, Redis, and PostgreSQL together on your instance deployed via DigitalOcean or Contabo VPS.

version: '3.8'

services:
  redis:
    image: redis:7-alpine
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  postgres:
    image: postgres:15-alpine
    restart: always
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: app_db
    ports:
      - "5432:5432"
    volumes:
      - pg_data:/var/lib/postgresql/data

  webhook-api:
    build: .
    command: node server.js
    restart: always
    ports:
      - "3000:3000"
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - DATABASE_URL=postgres://user:pass@postgres:5432/app_db
      - WEBHOOK_SECRET=super-secret-key-12345
    depends_on:
      - redis
      - postgres

  webhook-worker:
    build: .
    command: node worker.js
    restart: always
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - DATABASE_URL=postgres://user:pass@postgres:5432/app_db
    depends_on:
      - redis
      - postgres

volumes:
  redis_data:
  pg_data:

Handling Distributed Edge Cases and Database State

Even with an atomic Redis lock and an asynchronous queue, distributed systems present unique failure modes that require defensive coding in your SQL logic.

1. Database Row Locking (FOR UPDATE)

Notice the explicit SELECT ... FOR UPDATE query in worker.js. If two worker threads somehow obtain the same queue message (for example, if a Redis node fails over or a process times out without releasing its item cleanly), PostgreSQL will serialize access to that specific event_id row inside the transaction block. The second worker process will block until the first worker completes its transaction, read the newly inserted processed_webhooks record, trigger the early return condition, and safely roll back without modifying balances twice.

2. Dead Letter Queues (DLQs) for Poison Pill Payloads

If an incoming webhook payload contains malformed schema values, database queries will throw unhandled exceptions repeatedly. Without explicit exception handling, a queue consumer will either drop the message silently or enter a crash loop. Pushing failed messages to a dedicated webhook:dead_letter_queue preserves the raw message payload alongside the exception stack trace. This enables developer inspection, root-cause diagnosis, and manual re-processing once the fix is deployed.

3. Graceful TTL Expiration Strategies

Lock keys in Redis must have an explicit expiry time (LOCK_TTL_MS). Never create indefinite locks. If a server hosting a worker node experiences a hardware crash, kernel panic, or sudden out-of-memory container termination while holding an unlimited lock, that webhook event will remain permanently blocked from reprocessing. Setting a 30-second Redis lock TTL ensures that if a server dies mid-task, another worker process can pick up the pending retry event automatically once the timer elapses.

Getting Started

Building resilient, idempotent webhook ingestion infrastructure protects your web applications against double-billing, duplicate records, and cascade system failures under high network traffic.

To quickly deploy this stack on low-latency infrastructure, set up a server using Hetzner VPS or Contabo VPS. If you prefer managed cloud platforms, launch your containers on DigitalOcean. Point your external domain to your server using Namecheap, and configure automated webhook automation pipelines easily using n8n Cloud.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.