Custom Software DevelopmentSelf-Healing Webhook Bridges: Syncing Third-Party Vendor Data into Custom Portals Without Loss

Self-Healing Webhook Bridges: Syncing Third-Party Vendor Data into Custom Portals Without Loss

Webhooks are at-least-once, unordered and best-effort. Build a durable inbox, version-guarded processor and reconciliation loop that keeps your custom portal converged with vendor data even when deliveries fail.

D

Danisur Rahman

Verified
Principal Distributed Systems Architect•Oct 9, 2026•14 min read
Self-Healing Webhook Bridges: Syncing Third-Party Vendor Data into Custom Portals Without Loss

Every custom customer or partner portal eventually depends on data it does not own: invoices from a billing platform, shipment events from a logistics vendor, ticket updates from a support suite, compliance statuses from a KYC provider. The vendor's answer to "how do I get this data?" is almost always the same: register a webhook URL and we will POST you events.

That sentence hides a hard distributed-systems problem. Webhooks are at-least-once, unordered, best-effort notifications delivered over the public internet to an endpoint you operate. Your deploy will restart pods mid-request. The vendor will retry for a finite window and then give up. Two events for the same object will arrive in the wrong order. A payload will be delivered twice, or never. If your portal treats each webhook as the source of truth, it will drift from the vendor's reality, and the drift will be discovered by a customer rather than by your monitoring.

This article describes the architecture we use for a self-healing webhook bridge: a pipeline that accepts events durably, applies them idempotently and in the right order, detects its own divergence from the vendor, and repairs it without a human. The code is illustrative (Node.js and PostgreSQL), but the patterns apply to any stack.

1. Why Naive Webhook Handlers Lose Data#

The typical first implementation looks like this: an HTTP handler parses the JSON body, runs a few UPDATE statements against the portal database, and returns 200. It works in staging. In production it fails in at least six distinct ways:

Failure modeWhat happensConsequence
Handler crashes after vendor sent, before commitVendor sees 5xx and retries, or sees timeout and eventually stopsSilent loss once retries are exhausted
Duplicate deliverySame event applied twiceDouble-counted balances, duplicate notifications
Out-of-order deliverystatus=shipped arrives after status=deliveredPortal shows a regressed state
Slow handlerVendor times out (often within a few seconds) and retries while the first attempt still runsConcurrent duplicate processing
Vendor outage or paused endpointEvents during the window are never sentPermanent gap
Schema driftVendor adds or renames a fieldHandler throws, event is dropped
Notice that only some of these are fixable by making the handler "more careful". Others, such as the vendor giving up on retries, cannot be solved by receiving code at all. They require a second channel that does not depend on delivery: reconciliation. A bridge that is genuinely lossless needs both a reliable push path and an authoritative pull path.

2. Architecture Overview#

The design separates receiving from processing from verifying:

  1. Receiver - verifies the signature, writes the raw event to an append-only inbox table, and returns 202 in milliseconds. It performs no business logic.
  2. Processor - claims inbox rows, applies them to the portal's domain tables idempotently with version guards, and records outcomes.
  3. Reconciler - periodically pulls authoritative state from the vendor API, compares it with the portal, and injects synthetic events for any divergence.
  4. Dead-letter and replay tooling - parks poison events with context and allows safe re-processing after a fix.
  5. Observability - lag, divergence, and DLQ metrics with alerts that fire before customers notice.

The key principle is that the inbox is the system's memory. Anything the receiver acknowledged is on disk before the vendor is told it succeeded. Everything after that is replayable.

3. The Receiver: Verify, Persist, Acknowledge#

The receiver must do as little as possible. Its job is to convert an unreliable network delivery into a durable row.

3.1 Inbox schema#

sql
400 font-semibold">CREATE 400 font-semibold">TABLE webhook_inbox (
  id            BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  vendor        TEXT        NOT NULL,
  event_id      TEXT        NOT NULL,           -- vendor-supplied unique id
  event_type    TEXT        NOT NULL,
  object_id     TEXT        NOT NULL,           -- the vendor entity 400 font-semibold">this concerns
  object_version BIGINT,                        -- vendor revision / updated_at epoch, 400 font-semibold">if offered
  payload       JSONB       NOT NULL,
  received_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  status        TEXT        NOT NULL DEFAULT 400 font-semibold">class="text-emerald-300">'pending'
                CHECK (status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'processing',400 font-semibold">class="text-emerald-300">'done',400 font-semibold">class="text-emerald-300">'failed',400 font-semibold">class="text-emerald-300">'dead')),
  attempts      INT         NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_error    TEXT,
  source        TEXT        NOT NULL DEFAULT 400 font-semibold">class="text-emerald-300">'webhook'
                CHECK (source IN (400 font-semibold">class="text-emerald-300">'webhook',400 font-semibold">class="text-emerald-300">'reconciler',400 font-semibold">class="text-emerald-300">'replay')),
  UNIQUE (vendor, event_id)
);

400 font-semibold">CREATE 400 font-semibold">INDEX webhook_inbox_ready_idx
  ON webhook_inbox (next_attempt_at)
  400 font-semibold">WHERE status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'failed');

400 font-semibold">CREATE 400 font-semibold">INDEX webhook_inbox_object_idx
  ON webhook_inbox (vendor, object_id, received_at);

The UNIQUE (vendor, event_id) constraint is the first line of defense against duplicates: inserting the same delivery twice becomes a no-op. The partial index keeps the "what is ready to work on" query fast even when the table holds millions of finished rows.

3.2 The handler#

Signature verification must use the raw request bytes, not re-serialized JSON, and a constant-time comparison. Most vendors (Stripe, GitHub, Shopify and others) use HMAC-SHA256 over the body, sometimes with a timestamp to limit replay.

javascript
400 font-semibold">import express 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"express";
400 font-semibold">import crypto 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"node:crypto";
400 font-semibold">import { pool } 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"./db.js";

400 font-semibold">const app = express();

app.post(
  400 font-semibold">class="text-emerald-300">"/webhooks/acme",
  express.raw({ 400 font-semibold">type: 400 font-semibold">class="text-emerald-300">"application/json", limit: 400 font-semibold">class="text-emerald-300">"1mb" }),
  400 font-semibold">async (req, res) => {
    400 font-semibold">const secret = process.env.ACME_WEBHOOK_SECRET;
    400 font-semibold">const sigHeader = req.get(400 font-semibold">class="text-emerald-300">"X-Acme-Signature") ?? 400 font-semibold">class="text-emerald-300">"";
    400 font-semibold">const timestamp = req.get(400 font-semibold">class="text-emerald-300">"X-Acme-Timestamp") ?? 400 font-semibold">class="text-emerald-300">"";

    400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// Reject stale deliveries to limit replay of captured requests.
    400 font-semibold">if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) {
      400 font-semibold">return res.status(400).end();
    }

    400 font-semibold">const expected = crypto
      .createHmac(400 font-semibold">class="text-emerald-300">"sha256", secret)
      .update(400 font-semibold">class="text-emerald-300">`${timestamp}.`)
      .update(req.body)            400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// Buffer: the exact bytes received
      .digest(400 font-semibold">class="text-emerald-300">"hex");

    400 font-semibold">const a = Buffer.400 font-semibold">from(expected);
    400 font-semibold">const b = Buffer.400 font-semibold">from(sigHeader);
    400 font-semibold">if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
      400 font-semibold">return res.status(401).end();
    }

    400 font-semibold">let evt;
    400 font-semibold">try {
      evt = JSON.parse(req.body.toString(400 font-semibold">class="text-emerald-300">"utf8"));
    } 400 font-semibold">catch {
      400 font-semibold">return res.status(400).end();
    }

    400 font-semibold">await pool.query(
      400 font-semibold">class="text-emerald-300">`400 font-semibold">INSERT INTO webhook_inbox
         (vendor, event_id, event_type, object_id, object_version, payload)
       VALUES ($1,$2,$3,$4,$5,$6)
       ON CONFLICT (vendor, event_id) DO NOTHING`,
      [400 font-semibold">class="text-emerald-300">"acme", evt.id, evt.400 font-semibold">type, evt.data.object.id,
       evt.data.object.revision ?? 400">null, evt]
    );

    res.status(202).end();       400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// durable on disk before we answer
  }
);

If the insert throws, let the error produce a 500: the vendor's retry then becomes a free durability mechanism for the one case where you could not persist.

4. The Processor: Idempotent, Ordered, Version-Guarded#

Because delivery is at-least-once and unordered, the processor must make two guarantees: applying an event twice is harmless, and applying an older event after a newer one is harmless.

4.1 Claiming work without double-processing#

FOR UPDATE SKIP LOCKED lets multiple worker replicas pull from the same table without blocking each other:

sql
WITH batch AS (
  400 font-semibold">SELECT id
  400 font-semibold">FROM webhook_inbox
  400 font-semibold">WHERE status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'failed')
    AND next_attempt_at <= now()
  400 font-semibold">ORDER BY next_attempt_at
  LIMIT 50
  FOR 400 font-semibold">UPDATE SKIP LOCKED
)
400 font-semibold">UPDATE webhook_inbox w
SET status = 400 font-semibold">class="text-emerald-300">'processing', attempts = attempts + 1
400 font-semibold">FROM batch
400 font-semibold">WHERE w.id = batch.id
RETURNING w.*;

A sweeper that resets rows stuck in processing beyond a lease interval (say, five minutes) back to pending recovers from crashed workers.

4.2 Per-object ordering#

Parallel workers can still process two events for the same object concurrently. There are two common strategies:

  • Serialize per object by taking a transaction-scoped advisory lock keyed on the object: pg_advisory_xact_lock(hashtextextended(vendor || ':' || object_id, 0)).
  • Make order irrelevant by applying only state that is newer than what you already hold. This is the more robust option and the one we rely on, with the advisory lock as a belt-and-braces addition.

4.3 Version-guarded upsert#

Vendors usually give you a monotonic marker: a revision integer, an updated_at timestamp, or a sequence. Store it alongside the domain row and let the database enforce "newest wins":

sql
400 font-semibold">CREATE 400 font-semibold">TABLE portal_shipments (
  vendor_id       TEXT PRIMARY KEY,
  status          TEXT        NOT NULL,
  eta             TIMESTAMPTZ,
  vendor_version  BIGINT      NOT NULL,
  synced_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

400 font-semibold">INSERT INTO portal_shipments (vendor_id, status, eta, vendor_version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (vendor_id) DO 400 font-semibold">UPDATE
SET status         = EXCLUDED.status,
    eta            = EXCLUDED.eta,
    vendor_version = EXCLUDED.vendor_version,
    synced_at      = now()
400 font-semibold">WHERE portal_shipments.vendor_version < EXCLUDED.vendor_version;

If a stale event arrives, the WHERE clause makes the statement affect zero rows, which is exactly the desired outcome. Duplicates are handled by the same clause because equal versions do not satisfy <.

When the vendor provides no version, derive one from the object's own updated_at field. As a last resort, treat the event as a trigger rather than a payload: ignore its body, call the vendor's GET /shipments/{id} endpoint, and store whatever it returns. This "thin event, fat fetch" pattern is slower but removes ordering concerns almost entirely, and it is what several large vendors recommend.

4.4 Retry policy and the dead-letter state#

Failures fall into two classes, and the processor should distinguish them:

  • Transient (database timeout, vendor API 429/503): retry with exponential backoff and jitter.
  • Permanent (payload fails validation, foreign key to an unknown parent): park for inspection after a bounded number of attempts.

javascript
400 font-semibold">const MAX_ATTEMPTS = 8;

400 font-semibold">function nextDelaySeconds(attempts) {
  400 font-semibold">const base = Math.min(2 ** attempts * 5, 3600);       400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// 10s, 20s, 40s ... cap 1h
  400 font-semibold">return base * (0.5 + Math.random() * 0.5);             400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// jitter
}

400 font-semibold">async 400 font-semibold">function recordFailure(client, row, err, permanent) {
  400 font-semibold">const dead = permanent || row.attempts &gt;= MAX_ATTEMPTS;
  400 font-semibold">await client.query(
    400 font-semibold">class="text-emerald-300">`400 font-semibold">UPDATE webhook_inbox
        SET status = $2,
            last_error = $3,
            next_attempt_at = now() + make_interval(secs =&gt; $4)
      400 font-semibold">WHERE id = $1`,
    [row.id, dead ? 400 font-semibold">class="text-emerald-300">"dead" : 400 font-semibold">class="text-emerald-300">"failed", String(err).slice(0, 2000),
     dead ? 0 : nextDelaySeconds(row.attempts)]
  );
}

A frequent real-world cause of "permanent" errors is child-before-parent arrival: an invoice.line_added event shows up before invoice.created. Treat a missing parent as transient for a few attempts, or fetch the parent on demand, rather than dead-lettering immediately.

5. The Healing Layer: Reconciliation#

Everything so far makes the push path as reliable as it can be. It still cannot guarantee completeness, because some events are simply never sent. Self-healing comes from assuming this and continuously comparing against the source of truth.

5.1 Two reconciliation modes#

ModeCadenceStrategyCatches
Incremental sweepEvery few minutesList vendor objects with updated_since = watermark - overlapRecently missed events, short vendor outages
Full auditNightly or weekly, off-peakPage through all objects, compare checksums with the portalLong-tail drift, deletions, bugs in the processor
The incremental sweep must overlap its watermark (for example, re-read the last 15 minutes each run) because vendor updated_at values and your clock are not perfectly aligned, and because index lag on the vendor side can make recent records appear late.

5.2 Injecting synthetic events#

The reconciler does not write to domain tables directly. It writes to the same inbox, with source = 'reconciler', so that every repair flows through the same validation, ordering, and observability as a webhook:

javascript
400 font-semibold">export 400 font-semibold">async 400 font-semibold">function reconcileShipments(vendorClient, pool) {
  400 font-semibold">const { rows: [{ watermark }] } = 400 font-semibold">await pool.query(
    400 font-semibold">class="text-emerald-300">`400 font-semibold">SELECT COALESCE(max(synced_at), now() - interval '1 day') - interval '15 minutes'
            AS watermark
       400 font-semibold">FROM portal_shipments`
  );

  400 font-semibold">for 400 font-semibold">await (400 font-semibold">const s of vendorClient.listShipments({ updatedSince: watermark })) {
    400 font-semibold">const { rows } = 400 font-semibold">await pool.query(
      400 font-semibold">class="text-emerald-300">`400 font-semibold">SELECT vendor_version 400 font-semibold">FROM portal_shipments 400 font-semibold">WHERE vendor_id = $1`,
      [s.id]
    );
    400 font-semibold">const local = rows[0]?.vendor_version ?? -1;
    400 font-semibold">if (s.revision &gt; local) {
      400 font-semibold">await pool.query(
        400 font-semibold">class="text-emerald-300">`400 font-semibold">INSERT INTO webhook_inbox
           (vendor, event_id, event_type, object_id, object_version, payload, source)
         VALUES ('acme', $1, 'shipment.reconciled', $2, $3, $4, 'reconciler')
         ON CONFLICT (vendor, event_id) DO NOTHING`,
        [400 font-semibold">class="text-emerald-300">`recon:${s.id}:${s.revision}`, s.id, s.revision,
         { data: { object: s } }]
      );
    }
  }
}

The deterministic event_id (recon:<id>:<revision>) makes the reconciler itself idempotent: running it twice produces the same rows once.

5.3 Detecting deletions and orphans#

An incremental sweep only sees objects that exist. To detect objects the vendor deleted, the full audit collects the set of vendor IDs and compares it against the portal's IDs. Anything present locally but absent remotely is either soft-deleted or flagged for review, depending on the domain. Never hard-delete automatically on the first miss: a transient vendor API error returning an empty page would otherwise wipe your data. Require the same absence on two consecutive audits.

5.4 Scheduling it#

On Kubernetes, reconciliation is a natural CronJob. concurrencyPolicy: Forbid prevents overlapping runs from racing over the same watermark:

yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: acme-reconciler-incremental
  namespace: portal
spec:
  schedule: 400 font-semibold">class="text-emerald-300">"*/5 * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 240
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: reconciler
              image: registry.example.com/portal/reconciler:1.14.0
              args: [400 font-semibold">class="text-emerald-300">"--vendor=acme", 400 font-semibold">class="text-emerald-300">"--mode=incremental", 400 font-semibold">class="text-emerald-300">"--overlap=15m"]
              envFrom:
                - secretRef:
                    name: acme-api-credentials
              resources:
                requests: { cpu: 100m, memory: 128Mi }
                limits:   { cpu: 500m, memory: 256Mi }

Respect vendor rate limits: throttle with a token bucket and honor Retry-After.

6. Replay and Dead-Letter Operations#

A bridge is only as maintainable as its recovery tooling. Because the inbox retains the raw payload, fixing a processor bug becomes a data operation rather than a vendor support ticket:

sql
-- Inspect what is stuck and why
400 font-semibold">SELECT event_type, left(last_error, 80) AS error, count(*)
400 font-semibold">FROM webhook_inbox
400 font-semibold">WHERE status = 400 font-semibold">class="text-emerald-300">'dead'
400 font-semibold">GROUP BY 1, 2
400 font-semibold">ORDER BY 3 DESC;

-- After deploying the fix, requeue a specific 400 font-semibold">class of failures
400 font-semibold">UPDATE webhook_inbox
SET status = 400 font-semibold">class="text-emerald-300">'pending', attempts = 0, next_attempt_at = now(), source = 400 font-semibold">class="text-emerald-300">'replay'
400 font-semibold">WHERE status = 400 font-semibold">class="text-emerald-300">'dead'
  AND event_type = 400 font-semibold">class="text-emerald-300">'invoice.line_added'
  AND last_error LIKE 400 font-semibold">class="text-emerald-300">'missing parent%';

Since the version guard makes processing idempotent, replaying a wide range is safe: events that are already reflected in the domain tables simply update zero rows. Retain inbox rows for a defined period (30 to 90 days is common) and archive older partitions; partitioning the table by received_at makes that retention a cheap DROP PARTITION.

7. Observability: Knowing Before the Customer Does#

A self-healing system should still tell you when it is healing a lot, because heavy repair traffic usually points to a root cause worth fixing. Track four signals:

  1. Inbox lag: age of the oldest pending or failed row.
  2. Dead-letter depth: count of rows in dead.
  3. Reconciler repair rate: synthetic events that changed domain state, per run.
  4. Receive-to-apply latency: histogram from received_at to completion.

The repair rate is the most informative: a sudden spike suggests the endpoint was down, a secret was rotated, or the vendor changed behavior.

yaml
groups:
  - name: webhook-bridge
    rules:
      - alert: WebhookInboxLagHigh
        expr: max(webhook_inbox_oldest_pending_age_seconds{vendor=400 font-semibold">class="text-emerald-300">"acme"}) &gt; 300
        400 font-semibold">for: 10m
        labels: { severity: page }
        annotations:
          summary: 400 font-semibold">class="text-emerald-300">"Acme inbox has events waiting longer than 5 minutes"

      - alert: WebhookDeadLetterGrowing
        expr: increase(webhook_inbox_dead_total{vendor=400 font-semibold">class="text-emerald-300">"acme"}[1h]) &gt; 10
        labels: { severity: ticket }
        annotations:
          summary: 400 font-semibold">class="text-emerald-300">"More than 10 Acme events dead-lettered in the last hour"

      - alert: ReconcilerRepairSpike
        expr: sum(increase(reconciler_repairs_total{vendor=400 font-semibold">class="text-emerald-300">"acme"}[30m])) &gt; 100
        labels: { severity: ticket }
        annotations:
          summary: 400 font-semibold">class="text-emerald-300">"Reconciler is repairing unusually many objects; check webhook delivery"

Tune thresholds from your own baseline.

8. Capacity and Cost: A Worked Example#

Consider a hypothetical portal syncing 20 events per second on average, with an assumed 4 KB payload and 60-day inbox retention.

  • Inbox growth: 20 x 86,400 is about 1.73 million rows/day, or roughly 6.9 GB/day of raw payload before compression and indexes. Over 60 days that is on the order of 400 GB, which is why partitioning and JSONB compression matter.
  • Reconciliation cost: a nightly full audit of 500,000 objects at 100 per page is 5,000 API calls, which can fit inside many vendors' published rate limits when throttled.

These figures come from stated assumptions, not measurement. The method is what transfers: size the inbox from event rate, payload size, and retention, then verify against production metrics.

9. Design Notes#

Using the database as the queue avoids the dual-write problem (persisting to the database and publishing to a broker atomically), a common source of subtle loss. If you later need a broker for fan-out, add the transactional outbox pattern on the processing side rather than moving the inbox itself.

The test that proves the system is self-healing rather than merely careful is simple: drop a set of events entirely, run the reconciler, and assert that the portal converges to the vendor fixture. Pair it with tests that deliver the same event 50 times concurrently and deliver versions 5, 3, 4 in that order.

Conclusion#

Lossless vendor synchronization is not achieved by a smarter webhook handler. It comes from composing a few simple guarantees: durable acknowledgement into an inbox, idempotent version-guarded application, bounded retries with a dead-letter path, and a reconciliation loop that treats the vendor API as the final authority. Each layer assumes the one before it will occasionally fail, and the combination converges to the correct state even when individual deliveries do not.

If your portal depends on third-party data and you want an architecture review of its ingestion path, the team at knetwork.live can help you design and harden a bridge like this for your vendors and traffic profile. Reach out to schedule an architecture consultation.

Frequently Asked Questions

Key questions answered regarding this architectural implementation.

Senior Systems Architecture Advisory
Available for Q4 Advisory & Audits

Planning a high-concurrency overhaul or critical migration?

Move from theoretical documentation to production verification. You talk directly with the principal architects who build, benchmark, and deploy these systems—never a business development rep or account manager.

Direct WhatsApp ChannelFastest

Chat with an Architect

Direct access during working hours. Outline your stack, latency constraints, or bottlenecks for rapid feedback.

Open WhatsApp Discussion
Formal Architecture Brief1-Day SLA

Request an Engineering Review

Share your specification, RFP, or architecture diagram. We reply with technical scope and feasibility.

Mutual NDA Protected1 Working Day Response100% IP & Code Ownership
Explore All Contact Channels
D

Danisur Rahman

Lead Author

Principal Distributed Systems Architect • KNetwork Systems

Request Technical Review

Principal architect specializing in enterprise distributed systems, edge caching, and hardware integration pipelines. Leads engineering audits, high-concurrency database optimizations, and zero-trust VPC deployments across high-growth ventures.

Distributed BackendsEvent StreamingPrivate RAGIoT Telemetry
The Engineering Dispatch

Enjoyed this technical breakdown?

Subscribe to receive new architectural guides, system teardowns, and engineering benchmarks directly in your inbox.