Handling Webhook Event Deduplication in Distributed Systems

What You’ll Need

Before building a resilient webhook deduplication pipeline, ensure you have access to the following infrastructure and tools:

• A cloud server such as a Hetzner VPS or Contabo VPS running Linux, or a virtual machine on DigitalOcean.
• Node.js (v18 or higher) installed on your environment.
• A running Redis instance (v6 or higher) for fast key storage and locking.
• A running PostgreSQL database (v13 or higher) for persistent storage.
• A domain managed via Namecheap if you plan to expose public webhooks via HTTPS.
• An orchestration server like n8n Cloud or self-hosted n8n for workflow integration. (If comparing with alternative platforms, tools like Make.com offer managed webhooks, but custom backend code gives you fine-grained control over execution guarantees).

Table of Contents

• The Anatomy of Duplicate Webhook Events
• Architecture Strategy: Redis Distributed Locking and Idempotency Keys
• Building a Production Deduplication Middleware in Node.js
• Database Level Deduplication and Persistence in PostgreSQL
• Handling Race Conditions and Edge Cases
• Getting Started

The Anatomy of Duplicate Webhook Events

When third-party providers like Stripe, GitHub, Shopify, or PayPal issue webhooks, they operate under an at-least-once delivery guarantee. This design pattern ensures that an event is never lost due to transient network failures, but it introduces a major side effect: duplicate payload delivery.

Duplicates happen for several distinct reasons:

1. Network Retries: The vendor sends an HTTP POST request, but an intermediate router drops the response payload. The provider considers the request timed out and re-sends the exact same payload.
2. High Internal Queue Concurrency: Vendor systems split delivery across multiple background workers. Under heavy loads, two workers might dequeue and send the exact same event milliseconds apart.
3. Slow Receiver Processing: If your application takes longer than 3 to 5 seconds to respond with an HTTP 200 OK, the vendor marks the attempt as failed and enqueues a retry.

If your endpoint increments a user’s wallet balance or triggers a physical fulfillment process, executing duplicate webhooks will corrupt data and cost real money. Identifying duplicates requires capturing unique identifiers sent in the request headers or body, such as X-Webhook-ID, Stripe-Signature timestamps, or event UUIDs.

When tracking issues across distributed worker instances, monitoring tools become critical. Inspecting request trails with structured logging systems helps diagnose whether duplicates stem from retries or concurrent dispatches. For details on centralizing telemetry, read our guide on Building Structured Log Pipelines with Fluent Bit.

Architecture Strategy: Redis Distributed Locking and Idempotency Keys

To handle webhooks safely at scale, you must implement a multi-tiered deduplication layer. Relying solely on your primary SQL database can lead to table bloat and CPU spikes under high concurrency. Instead, split deduplication into two distinct phases:

1. In-Memory Atomic Cache (Fast Path): Use Redis to check whether an idempotency key (derived from the webhook event ID) is currently being processed or was processed recently.
2. Persistent Storage with Unique Constraints (Slow Path): Save every successfully processed event ID into PostgreSQL using a strict unique index constraint.

When deploying this pipeline inside a containerized setup, ensure your worker containers have enough CPU and memory headroom to avoid container throttling during traffic spikes. Learn how to manage container boundaries in our complete walkthrough on How to Restrict Docker Container Resource Usage.

Here is the flow of an incoming request through the architecture:

1. Webhook hits your HTTP server endpoint on a server hosted on a Hetzner VPS.
2. The server extracts the provider’s unique event ID.
3. The server executes an atomic SET key value NX EX TTL command in Redis.
4. If Redis returns null, the event is already being processed or was recently finished. The server drops or defers the request with HTTP 200 or HTTP 409.
5. If Redis returns OK, the server acquires the lock and processes the business logic.
6. Upon completion, the server inserts the record into PostgreSQL and optionally releases or updates the lock state in Redis.

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Building a Production Deduplication Middleware in Node.js

Let us write a complete Express middleware application in Node.js using ioredis. This middleware extracts an idempotency key, attempts to set an atomic lock in Redis, and rejects duplicate requests before they touch down into downstream services.

Save this script as server.js. Notice that every line is fully written out so you can execute it immediately in your development or production environment.

const express = require('express');
const Redis = require('ioredis');
const crypto = require('crypto');

const app = express();
app.use(express.json());

const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
  lazyConnect: false
});

const DEDUPLICATION_TTL_SECONDS = 86400;

function deriveEventKey(req) {
  const eventId = req.headers['x-webhook-id'] || req.body.id;
  if (!eventId) {
    const fallbackHash = crypto
      .createHash('sha256')
      .update(JSON.stringify(req.body))
      .digest('hex');
    return `webhook:dedup:hash:${fallbackHash}`;
  }
  return `webhook:dedup:id:${eventId}`;
}

async function deduplicateMiddleware(req, res, next) {
  const redisKey = deriveEventKey(req);
  const lockAcquired = await redis.set(
    redisKey,
    'PROCESSING',
    'EX',
    DEDUPLICATION_TTL_SECONDS,
    'NX'
  );

  if (!lockAcquired) {
    const currentState = await redis.get(redisKey);
    if (currentState === 'PROCESSING') {
      return res.status(409).json({
        error: 'Conflict',
        message: 'Event is currently being processed by another worker.'
      });
    }
    return res.status(200).json({
      status: 'ignored',
      message: 'Duplicate event already processed.'
    });
  }

  req.dedupKey = redisKey;
  next();
}

app.post('/api/v1/webhooks', deduplicateMiddleware, async (req, res) => {
  try {
    const event = req.body;
    
    console.log(`Processing payload for event type: ${event.type || 'unknown'}`);
    
    await new Promise((resolve) => setTimeout(resolve, 500));

    await redis.set(req.dedupKey, 'COMPLETED', 'EX', DEDUPLICATION_TTL_SECONDS);

    return res.status(200).json({
      success: true,
      processedAt: new Date().toISOString()
    });
  } catch (err) {
    console.error('Error processing webhook:', err);
    await redis.del(req.dedupKey);
    return res.status(500).json({
      error: 'Internal Server Error',
      message: err.message
    });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Deduplication service online on port ${PORT}`);
});

Database Level Deduplication and Persistence in PostgreSQL

In-memory locks in Redis handle immediate duplicate spikes, but Redis keys expire over time. To achieve permanent audit capabilities and guarantee true idempotency across months or years, persist event execution metadata to PostgreSQL.

Below is a complete schema script and Node.js database handler using the pg driver that implements ON CONFLICT constraints.

PostgreSQL Table Schema

CREATE TABLE IF NOT EXISTS processed_webhooks (
    id SERIAL PRIMARY KEY,
    event_id VARCHAR(255) NOT NULL,
    provider VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(32) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    CONSTRAINT uk_provider_event UNIQUE (provider, event_id)
);

CREATE INDEX IF NOT EXISTS idx_processed_webhooks_provider_event 
ON processed_webhooks (provider, event_id);

When managing high-volume webhook writes in relational storage, query execution paths must be tuned properly. For a deep dive into indexing strategies, check out our companion piece on Optimizing PostgreSQL Performance With Indexing Strategies.

Node.js Database Handler

Here is a fully runnable script using pg that handles database level insertion with conflict detection:

const { Pool } = require('pg');

const pool = new Pool({
  user: process.env.DB_USER || 'postgres',
  host: process.env.DB_HOST || '127.0.0.1',
  database: process.env.DB_NAME || 'webhooks_db',
  password: process.env.DB_PASSWORD || 'secret',
  port: parseInt(process.env.DB_PORT || '5432', 10),
});

async function recordWebhookEvent(provider, eventId, payload) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const insertQuery = `
      INSERT INTO processed_webhooks (provider, event_id, payload, status)
      VALUES ($1, $2, $3, $4)
      ON CONFLICT (provider, event_id) DO NOTHING
      RETURNING id, created_at;
    `;

    const result = await client.query(insertQuery, [
      provider,
      eventId,
      JSON.stringify(payload),
      'SUCCESS'
    ]);

    if (result.rowCount === 0) {
      await client.query('ROLLBACK');
      return {
        isDuplicate: true,
        record: null
      };
    }

    await client.query('COMMIT');
    return {
      isDuplicate: false,
      record: result.rows[0]
    };
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

module.exports = { recordWebhookEvent, pool };

Handling Race Conditions and Edge Cases

Even with atomic cache keys and unique SQL indexes, complex edge cases emerge when operating distributed workers at high throughput.

Worker Crashes mid-Execution

If a worker server crashes after setting the Redis key to PROCESSING, the key could stay locked indefinitely unless a time-to-live (TTL) is established. Always set an explicit TTL during the SET ... NX operation. A 60-second TTL is usually enough for typical HTTP handler tasks.

Redlock for Distributed Redis Clusters

If your infrastructure uses a multi-node Redis cluster, SETNX on a single master node might not protect against race conditions if that master fails before replicating to replicas. In high-risk environments (like financial ledgers), use the Redlock algorithm.

Below is a complete implementation using the redlock package:

const Redis = require('ioredis');
const Redlock = require('redlock');

const clientA = new Redis({ host: '127.0.0.1', port: 6379 });
const clientB = new Redis({ host: '127.0.0.1', port: 6380 });
const clientC = new Redis({ host: '127.0.0.1', port: 6381 });

const redlock = new Redlock(
  [clientA, clientB, clientC],
  {
    driftFactor: 0.01,
    retryCount: 0,
    retryDelay: 200,
    retryJitter: 200
  }
);

async function processWithDistributedLock(eventId, taskFunction) {
  const resource = `locks:webhooks:${eventId}`;
  const ttl = 10000;

  try {
    const lock = await redlock.acquire([resource], ttl);
    console.log(`Acquired lock for event: ${eventId}`);

    try {
      const result = await taskFunction();
      return result;
    } finally {
      await lock.release().catch((err) => {
        console.error('Failed to release lock cleanly:', err.message);
      });
    }
  } catch (err) {
    console.warn(`Could not acquire lock for event ${eventId}, skipping execution.`);
    return { skipped: true, reason: 'Concurrent execution lock held.' };
  }
}

module.exports = { processWithDistributedLock };

Getting Started

Building a resilient webhook processing system requires combining fast in-memory caching with strict database constraints.

1. Deploy a Linux virtual server on Hetzner VPS, Contabo VPS, or DigitalOcean.
2. Configure your domain records through Namecheap to route incoming HTTPS traffic safely to your service backend.
3. Use n8n Cloud or a self-hosted instance to visually manage webhook retry queues and connect upstream APIs.
4. Integrate the express deduplication middleware and PostgreSQL schema provided in this guide into your microservices.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.