Self-Healing Webhook Bridges: Syncing Third-Party Vendor Data into Custom Portals Without Loss
Webhooks are at-least-once, unordered and best-effort. Build a durable inbox, version-guarded processor and reconciliation loop that keeps your custom portal converged with vendor data even when deliveries fail.

Every custom customer or partner portal eventually depends on data it does not own: invoices from a billing platform, shipment events from a logistics vendor, ticket updates from a support suite, compliance statuses from a KYC provider. The vendor's answer to "how do I get this data?" is almost always the same: register a webhook URL and we will POST you events.
That sentence hides a hard distributed-systems problem. Webhooks are at-least-once, unordered, best-effort notifications delivered over the public internet to an endpoint you operate. Your deploy will restart pods mid-request. The vendor will retry for a finite window and then give up. Two events for the same object will arrive in the wrong order. A payload will be delivered twice, or never. If your portal treats each webhook as the source of truth, it will drift from the vendor's reality, and the drift will be discovered by a customer rather than by your monitoring.
This article describes the architecture we use for a self-healing webhook bridge: a pipeline that accepts events durably, applies them idempotently and in the right order, detects its own divergence from the vendor, and repairs it without a human. The code is illustrative (Node.js and PostgreSQL), but the patterns apply to any stack.
1. Why Naive Webhook Handlers Lose Data#
The typical first implementation looks like this: an HTTP handler parses the JSON body, runs a few UPDATE statements against the portal database, and returns 200. It works in staging. In production it fails in at least six distinct ways:
| Failure mode | What happens | Consequence |
|---|---|---|
| Handler crashes after vendor sent, before commit | Vendor sees 5xx and retries, or sees timeout and eventually stops | Silent loss once retries are exhausted |
| Duplicate delivery | Same event applied twice | Double-counted balances, duplicate notifications |
| Out-of-order delivery | status=shipped arrives after status=delivered | Portal shows a regressed state |
| Slow handler | Vendor times out (often within a few seconds) and retries while the first attempt still runs | Concurrent duplicate processing |
| Vendor outage or paused endpoint | Events during the window are never sent | Permanent gap |
| Schema drift | Vendor adds or renames a field | Handler throws, event is dropped |
2. Architecture Overview#
The design separates receiving from processing from verifying:
- Receiver - verifies the signature, writes the raw event to an append-only inbox table, and returns
202in milliseconds. It performs no business logic. - Processor - claims inbox rows, applies them to the portal's domain tables idempotently with version guards, and records outcomes.
- Reconciler - periodically pulls authoritative state from the vendor API, compares it with the portal, and injects synthetic events for any divergence.
- Dead-letter and replay tooling - parks poison events with context and allows safe re-processing after a fix.
- Observability - lag, divergence, and DLQ metrics with alerts that fire before customers notice.
The key principle is that the inbox is the system's memory. Anything the receiver acknowledged is on disk before the vendor is told it succeeded. Everything after that is replayable.
3. The Receiver: Verify, Persist, Acknowledge#
The receiver must do as little as possible. Its job is to convert an unreliable network delivery into a durable row.
3.1 Inbox schema#
400 font-semibold">CREATE 400 font-semibold">TABLE webhook_inbox (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
vendor TEXT NOT NULL,
event_id TEXT NOT NULL, -- vendor-supplied unique id
event_type TEXT NOT NULL,
object_id TEXT NOT NULL, -- the vendor entity 400 font-semibold">this concerns
object_version BIGINT, -- vendor revision / updated_at epoch, 400 font-semibold">if offered
payload JSONB NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
status TEXT NOT NULL DEFAULT 400 font-semibold">class="text-emerald-300">'pending'
CHECK (status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'processing',400 font-semibold">class="text-emerald-300">'done',400 font-semibold">class="text-emerald-300">'failed',400 font-semibold">class="text-emerald-300">'dead')),
attempts INT NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_error TEXT,
source TEXT NOT NULL DEFAULT 400 font-semibold">class="text-emerald-300">'webhook'
CHECK (source IN (400 font-semibold">class="text-emerald-300">'webhook',400 font-semibold">class="text-emerald-300">'reconciler',400 font-semibold">class="text-emerald-300">'replay')),
UNIQUE (vendor, event_id)
);
400 font-semibold">CREATE 400 font-semibold">INDEX webhook_inbox_ready_idx
ON webhook_inbox (next_attempt_at)
400 font-semibold">WHERE status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'failed');
400 font-semibold">CREATE 400 font-semibold">INDEX webhook_inbox_object_idx
ON webhook_inbox (vendor, object_id, received_at);
The UNIQUE (vendor, event_id) constraint is the first line of defense against duplicates: inserting the same delivery twice becomes a no-op. The partial index keeps the "what is ready to work on" query fast even when the table holds millions of finished rows.
3.2 The handler#
Signature verification must use the raw request bytes, not re-serialized JSON, and a constant-time comparison. Most vendors (Stripe, GitHub, Shopify and others) use HMAC-SHA256 over the body, sometimes with a timestamp to limit replay.
400 font-semibold">import express 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"express";
400 font-semibold">import crypto 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"node:crypto";
400 font-semibold">import { pool } 400 font-semibold">from 400 font-semibold">class="text-emerald-300">"./db.js";
400 font-semibold">const app = express();
app.post(
400 font-semibold">class="text-emerald-300">"/webhooks/acme",
express.raw({ 400 font-semibold">type: 400 font-semibold">class="text-emerald-300">"application/json", limit: 400 font-semibold">class="text-emerald-300">"1mb" }),
400 font-semibold">async (req, res) => {
400 font-semibold">const secret = process.env.ACME_WEBHOOK_SECRET;
400 font-semibold">const sigHeader = req.get(400 font-semibold">class="text-emerald-300">"X-Acme-Signature") ?? 400 font-semibold">class="text-emerald-300">"";
400 font-semibold">const timestamp = req.get(400 font-semibold">class="text-emerald-300">"X-Acme-Timestamp") ?? 400 font-semibold">class="text-emerald-300">"";
400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// Reject stale deliveries to limit replay of captured requests.
400 font-semibold">if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) {
400 font-semibold">return res.status(400).end();
}
400 font-semibold">const expected = crypto
.createHmac(400 font-semibold">class="text-emerald-300">"sha256", secret)
.update(400 font-semibold">class="text-emerald-300">`${timestamp}.`)
.update(req.body) 400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// Buffer: the exact bytes received
.digest(400 font-semibold">class="text-emerald-300">"hex");
400 font-semibold">const a = Buffer.400 font-semibold">from(expected);
400 font-semibold">const b = Buffer.400 font-semibold">from(sigHeader);
400 font-semibold">if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
400 font-semibold">return res.status(401).end();
}
400 font-semibold">let evt;
400 font-semibold">try {
evt = JSON.parse(req.body.toString(400 font-semibold">class="text-emerald-300">"utf8"));
} 400 font-semibold">catch {
400 font-semibold">return res.status(400).end();
}
400 font-semibold">await pool.query(
400 font-semibold">class="text-emerald-300">`400 font-semibold">INSERT INTO webhook_inbox
(vendor, event_id, event_type, object_id, object_version, payload)
VALUES ($1,$2,$3,$4,$5,$6)
ON CONFLICT (vendor, event_id) DO NOTHING`,
[400 font-semibold">class="text-emerald-300">"acme", evt.id, evt.400 font-semibold">type, evt.data.object.id,
evt.data.object.revision ?? 400">null, evt]
);
res.status(202).end(); 400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// durable on disk before we answer
}
);
If the insert throws, let the error produce a 500: the vendor's retry then becomes a free durability mechanism for the one case where you could not persist.
4. The Processor: Idempotent, Ordered, Version-Guarded#
Because delivery is at-least-once and unordered, the processor must make two guarantees: applying an event twice is harmless, and applying an older event after a newer one is harmless.
4.1 Claiming work without double-processing#
FOR UPDATE SKIP LOCKED lets multiple worker replicas pull from the same table without blocking each other:
WITH batch AS (
400 font-semibold">SELECT id
400 font-semibold">FROM webhook_inbox
400 font-semibold">WHERE status IN (400 font-semibold">class="text-emerald-300">'pending',400 font-semibold">class="text-emerald-300">'failed')
AND next_attempt_at <= now()
400 font-semibold">ORDER BY next_attempt_at
LIMIT 50
FOR 400 font-semibold">UPDATE SKIP LOCKED
)
400 font-semibold">UPDATE webhook_inbox w
SET status = 400 font-semibold">class="text-emerald-300">'processing', attempts = attempts + 1
400 font-semibold">FROM batch
400 font-semibold">WHERE w.id = batch.id
RETURNING w.*;
A sweeper that resets rows stuck in processing beyond a lease interval (say, five minutes) back to pending recovers from crashed workers.
4.2 Per-object ordering#
Parallel workers can still process two events for the same object concurrently. There are two common strategies:
- Serialize per object by taking a transaction-scoped advisory lock keyed on the object:
pg_advisory_xact_lock(hashtextextended(vendor || ':' || object_id, 0)). - Make order irrelevant by applying only state that is newer than what you already hold. This is the more robust option and the one we rely on, with the advisory lock as a belt-and-braces addition.
4.3 Version-guarded upsert#
Vendors usually give you a monotonic marker: a revision integer, an updated_at timestamp, or a sequence. Store it alongside the domain row and let the database enforce "newest wins":
400 font-semibold">CREATE 400 font-semibold">TABLE portal_shipments (
vendor_id TEXT PRIMARY KEY,
status TEXT NOT NULL,
eta TIMESTAMPTZ,
vendor_version BIGINT NOT NULL,
synced_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
400 font-semibold">INSERT INTO portal_shipments (vendor_id, status, eta, vendor_version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (vendor_id) DO 400 font-semibold">UPDATE
SET status = EXCLUDED.status,
eta = EXCLUDED.eta,
vendor_version = EXCLUDED.vendor_version,
synced_at = now()
400 font-semibold">WHERE portal_shipments.vendor_version < EXCLUDED.vendor_version;
If a stale event arrives, the WHERE clause makes the statement affect zero rows, which is exactly the desired outcome. Duplicates are handled by the same clause because equal versions do not satisfy <.
When the vendor provides no version, derive one from the object's own updated_at field. As a last resort, treat the event as a trigger rather than a payload: ignore its body, call the vendor's GET /shipments/{id} endpoint, and store whatever it returns. This "thin event, fat fetch" pattern is slower but removes ordering concerns almost entirely, and it is what several large vendors recommend.
4.4 Retry policy and the dead-letter state#
Failures fall into two classes, and the processor should distinguish them:
- Transient (database timeout, vendor API 429/503): retry with exponential backoff and jitter.
- Permanent (payload fails validation, foreign key to an unknown parent): park for inspection after a bounded number of attempts.
400 font-semibold">const MAX_ATTEMPTS = 8;
400 font-semibold">function nextDelaySeconds(attempts) {
400 font-semibold">const base = Math.min(2 ** attempts * 5, 3600); 400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// 10s, 20s, 40s ... cap 1h
400 font-semibold">return base * (0.5 + Math.random() * 0.5); 400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">// jitter
}
400 font-semibold">async 400 font-semibold">function recordFailure(client, row, err, permanent) {
400 font-semibold">const dead = permanent || row.attempts >= MAX_ATTEMPTS;
400 font-semibold">await client.query(
400 font-semibold">class="text-emerald-300">`400 font-semibold">UPDATE webhook_inbox
SET status = $2,
last_error = $3,
next_attempt_at = now() + make_interval(secs => $4)
400 font-semibold">WHERE id = $1`,
[row.id, dead ? 400 font-semibold">class="text-emerald-300">"dead" : 400 font-semibold">class="text-emerald-300">"failed", String(err).slice(0, 2000),
dead ? 0 : nextDelaySeconds(row.attempts)]
);
}
A frequent real-world cause of "permanent" errors is child-before-parent arrival: an invoice.line_added event shows up before invoice.created. Treat a missing parent as transient for a few attempts, or fetch the parent on demand, rather than dead-lettering immediately.
5. The Healing Layer: Reconciliation#
Everything so far makes the push path as reliable as it can be. It still cannot guarantee completeness, because some events are simply never sent. Self-healing comes from assuming this and continuously comparing against the source of truth.
5.1 Two reconciliation modes#
| Mode | Cadence | Strategy | Catches |
|---|---|---|---|
| Incremental sweep | Every few minutes | List vendor objects with updated_since = watermark - overlap | Recently missed events, short vendor outages |
| Full audit | Nightly or weekly, off-peak | Page through all objects, compare checksums with the portal | Long-tail drift, deletions, bugs in the processor |
updated_at values and your clock are not perfectly aligned, and because index lag on the vendor side can make recent records appear late.5.2 Injecting synthetic events#
The reconciler does not write to domain tables directly. It writes to the same inbox, with source = 'reconciler', so that every repair flows through the same validation, ordering, and observability as a webhook:
400 font-semibold">export 400 font-semibold">async 400 font-semibold">function reconcileShipments(vendorClient, pool) {
400 font-semibold">const { rows: [{ watermark }] } = 400 font-semibold">await pool.query(
400 font-semibold">class="text-emerald-300">`400 font-semibold">SELECT COALESCE(max(synced_at), now() - interval '1 day') - interval '15 minutes'
AS watermark
400 font-semibold">FROM portal_shipments`
);
400 font-semibold">for 400 font-semibold">await (400 font-semibold">const s of vendorClient.listShipments({ updatedSince: watermark })) {
400 font-semibold">const { rows } = 400 font-semibold">await pool.query(
400 font-semibold">class="text-emerald-300">`400 font-semibold">SELECT vendor_version 400 font-semibold">FROM portal_shipments 400 font-semibold">WHERE vendor_id = $1`,
[s.id]
);
400 font-semibold">const local = rows[0]?.vendor_version ?? -1;
400 font-semibold">if (s.revision > local) {
400 font-semibold">await pool.query(
400 font-semibold">class="text-emerald-300">`400 font-semibold">INSERT INTO webhook_inbox
(vendor, event_id, event_type, object_id, object_version, payload, source)
VALUES ('acme', $1, 'shipment.reconciled', $2, $3, $4, 'reconciler')
ON CONFLICT (vendor, event_id) DO NOTHING`,
[400 font-semibold">class="text-emerald-300">`recon:${s.id}:${s.revision}`, s.id, s.revision,
{ data: { object: s } }]
);
}
}
}
The deterministic event_id (recon:<id>:<revision>) makes the reconciler itself idempotent: running it twice produces the same rows once.
5.3 Detecting deletions and orphans#
An incremental sweep only sees objects that exist. To detect objects the vendor deleted, the full audit collects the set of vendor IDs and compares it against the portal's IDs. Anything present locally but absent remotely is either soft-deleted or flagged for review, depending on the domain. Never hard-delete automatically on the first miss: a transient vendor API error returning an empty page would otherwise wipe your data. Require the same absence on two consecutive audits.
5.4 Scheduling it#
On Kubernetes, reconciliation is a natural CronJob. concurrencyPolicy: Forbid prevents overlapping runs from racing over the same watermark:
apiVersion: batch/v1
kind: CronJob
metadata:
name: acme-reconciler-incremental
namespace: portal
spec:
schedule: 400 font-semibold">class="text-emerald-300">"*/5 * * * *"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 120
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
jobTemplate:
spec:
backoffLimit: 1
activeDeadlineSeconds: 240
template:
spec:
restartPolicy: Never
containers:
- name: reconciler
image: registry.example.com/portal/reconciler:1.14.0
args: [400 font-semibold">class="text-emerald-300">"--vendor=acme", 400 font-semibold">class="text-emerald-300">"--mode=incremental", 400 font-semibold">class="text-emerald-300">"--overlap=15m"]
envFrom:
- secretRef:
name: acme-api-credentials
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 256Mi }
Respect vendor rate limits: throttle with a token bucket and honor Retry-After.
6. Replay and Dead-Letter Operations#
A bridge is only as maintainable as its recovery tooling. Because the inbox retains the raw payload, fixing a processor bug becomes a data operation rather than a vendor support ticket:
-- Inspect what is stuck and why
400 font-semibold">SELECT event_type, left(last_error, 80) AS error, count(*)
400 font-semibold">FROM webhook_inbox
400 font-semibold">WHERE status = 400 font-semibold">class="text-emerald-300">'dead'
400 font-semibold">GROUP BY 1, 2
400 font-semibold">ORDER BY 3 DESC;
-- After deploying the fix, requeue a specific 400 font-semibold">class of failures
400 font-semibold">UPDATE webhook_inbox
SET status = 400 font-semibold">class="text-emerald-300">'pending', attempts = 0, next_attempt_at = now(), source = 400 font-semibold">class="text-emerald-300">'replay'
400 font-semibold">WHERE status = 400 font-semibold">class="text-emerald-300">'dead'
AND event_type = 400 font-semibold">class="text-emerald-300">'invoice.line_added'
AND last_error LIKE 400 font-semibold">class="text-emerald-300">'missing parent%';
Since the version guard makes processing idempotent, replaying a wide range is safe: events that are already reflected in the domain tables simply update zero rows. Retain inbox rows for a defined period (30 to 90 days is common) and archive older partitions; partitioning the table by received_at makes that retention a cheap DROP PARTITION.
7. Observability: Knowing Before the Customer Does#
A self-healing system should still tell you when it is healing a lot, because heavy repair traffic usually points to a root cause worth fixing. Track four signals:
- Inbox lag: age of the oldest
pendingorfailedrow. - Dead-letter depth: count of rows in
dead. - Reconciler repair rate: synthetic events that changed domain state, per run.
- Receive-to-apply latency: histogram from
received_atto completion.
The repair rate is the most informative: a sudden spike suggests the endpoint was down, a secret was rotated, or the vendor changed behavior.
groups:
- name: webhook-bridge
rules:
- alert: WebhookInboxLagHigh
expr: max(webhook_inbox_oldest_pending_age_seconds{vendor=400 font-semibold">class="text-emerald-300">"acme"}) > 300
400 font-semibold">for: 10m
labels: { severity: page }
annotations:
summary: 400 font-semibold">class="text-emerald-300">"Acme inbox has events waiting longer than 5 minutes"
- alert: WebhookDeadLetterGrowing
expr: increase(webhook_inbox_dead_total{vendor=400 font-semibold">class="text-emerald-300">"acme"}[1h]) > 10
labels: { severity: ticket }
annotations:
summary: 400 font-semibold">class="text-emerald-300">"More than 10 Acme events dead-lettered in the last hour"
- alert: ReconcilerRepairSpike
expr: sum(increase(reconciler_repairs_total{vendor=400 font-semibold">class="text-emerald-300">"acme"}[30m])) > 100
labels: { severity: ticket }
annotations:
summary: 400 font-semibold">class="text-emerald-300">"Reconciler is repairing unusually many objects; check webhook delivery"
Tune thresholds from your own baseline.
8. Capacity and Cost: A Worked Example#
Consider a hypothetical portal syncing 20 events per second on average, with an assumed 4 KB payload and 60-day inbox retention.
- Inbox growth: 20 x 86,400 is about 1.73 million rows/day, or roughly 6.9 GB/day of raw payload before compression and indexes. Over 60 days that is on the order of 400 GB, which is why partitioning and JSONB compression matter.
- Reconciliation cost: a nightly full audit of 500,000 objects at 100 per page is 5,000 API calls, which can fit inside many vendors' published rate limits when throttled.
These figures come from stated assumptions, not measurement. The method is what transfers: size the inbox from event rate, payload size, and retention, then verify against production metrics.
9. Design Notes#
Using the database as the queue avoids the dual-write problem (persisting to the database and publishing to a broker atomically), a common source of subtle loss. If you later need a broker for fan-out, add the transactional outbox pattern on the processing side rather than moving the inbox itself.
The test that proves the system is self-healing rather than merely careful is simple: drop a set of events entirely, run the reconciler, and assert that the portal converges to the vendor fixture. Pair it with tests that deliver the same event 50 times concurrently and deliver versions 5, 3, 4 in that order.
Conclusion#
Lossless vendor synchronization is not achieved by a smarter webhook handler. It comes from composing a few simple guarantees: durable acknowledgement into an inbox, idempotent version-guarded application, bounded retries with a dead-letter path, and a reconciliation loop that treats the vendor API as the final authority. Each layer assumes the one before it will occasionally fail, and the combination converges to the correct state even when individual deliveries do not.
If your portal depends on third-party data and you want an architecture review of its ingestion path, the team at knetwork.live can help you design and harden a bridge like this for your vendors and traffic profile. Reach out to schedule an architecture consultation.
Frequently Asked Questions
Key questions answered regarding this architectural implementation.
Planning a high-concurrency overhaul or critical migration?
Move from theoretical documentation to production verification. You talk directly with the principal architects who build, benchmark, and deploy these systems—never a business development rep or account manager.
Chat with an Architect
Direct access during working hours. Outline your stack, latency constraints, or bottlenecks for rapid feedback.
Request an Engineering Review
Share your specification, RFP, or architecture diagram. We reply with technical scope and feasibility.
Danisur Rahman
Lead AuthorPrincipal Distributed Systems Architect • KNetwork Systems
Principal architect specializing in enterprise distributed systems, edge caching, and hardware integration pipelines. Leads engineering audits, high-concurrency database optimizations, and zero-trust VPC deployments across high-growth ventures.
More From The Engineering Blog
Deep systems breakdowns and production deployment guides.
Server-Side Event Tracking: Recovering Missing Conversion Data Lost to Ad Blockers
A full-stack guide to server-side event tracking (sGTM): bypass client-side ad-blockers, defeat Safari ITP cookie expiration, slash 1.2MB of bloated tracking scripts, and recover up to 30% of lost conversion revenue.
Unified Customer Timelines: Combining Support Tickets, Purchases, and Product Usage into One Feed
Stop switching between helpdesk, billing, and analytics tabs. Build a unified customer timeline with a canonical event model, identity resolution, idempotent Kafka ingestion, and a partitioned PostgreSQL feed.
Enjoyed this technical breakdown?
Subscribe to receive new architectural guides, system teardowns, and engineering benchmarks directly in your inbox.