Reliable integrations assume failure: idempotent writes, exponential backoff with jitter, circuit breakers, dead letter queues with replay, and reconciliation that detects the drift retries could not fix.
The short answer
Assume every external call can time out, return a partial result, be delivered twice or silently do nothing. Design so that each of those is recoverable without human archaeology: make writes idempotent, retry with backoff, isolate failing dependencies, queue what cannot be delivered, and reconcile periodically to catch what retries missed.
Timeouts are not failures
A timeout tells you the response did not arrive. It tells you nothing about whether the operation executed. This ambiguity is the root of most duplicate-record problems in integration work.
The resolution is to make repetition harmless. Generate an idempotency key deterministically from the operation — not randomly per attempt — and have the receiving side recognise a key it has already processed and return the original result. Where the external system offers no such mechanism, keep a local record of attempted operations keyed the same way, and check before retrying.
Retry with backoff and jitter
Immediate retries against a struggling system make it worse. Exponential backoff — one second, two, four, eight — gives the dependency room to recover.
Add jitter. Without it, every client that failed during the same outage retries in lockstep the moment it ends, producing a thundering herd that causes the second outage. A randomised offset spreads the load.
Bound the retries. After a defined number of attempts the operation belongs in a dead letter queue, not in an infinite loop consuming capacity.
Circuit breakers stop the bleeding
When a dependency is clearly down, continuing to call it wastes resources and delays every queued operation behind the failures. A circuit breaker tracks the recent failure rate and, past a threshold, fails fast without making the call.
After a cooldown it allows a single probe through. If that succeeds, normal operation resumes. This turns a dependency outage into a bounded degradation rather than a cascading one.
Dead letter queues need replay tooling
A dead letter queue that nobody can act on is a slower way of losing data. Each entry needs the original payload, the failure reason, the attempt history and enough context to understand what it was trying to achieve.
It also needs an interface — even an internal one — where an operator can inspect entries, correct the underlying problem and replay them. Replay must be safe, which it is if the idempotency work above was done.
Reconciliation catches what retries cannot
Retries handle transient failure. They do not handle a webhook that was never sent, a record deleted directly in one system, or a mapping rule that was wrong for a fortnight.
Scheduled reconciliation compares both sides on the fields that matter and reports divergence. It is unglamorous and it is the only mechanism that reliably detects the silent drift class of failure. Run it regularly, report the count even when it is zero, and investigate when the number moves.
Decide who owns each record
Most painful integration incidents are not technical failures. They are two systems both believing they are authoritative for the same field, overwriting each other in a loop.
Decide ownership per entity, sometimes per field, and write it down. One system is authoritative, the others follow. Where genuine bidirectional editing is required, the conflict resolution rule has to be explicit — last write wins is a legitimate choice if it is a choice rather than an accident.
Make it observable
Every integration should report throughput, error rate, latency distribution, queue depth and reconciliation divergence. Alerts should fire on sustained error rate and growing queue depth, not on individual failures — individual failures are normal and retries exist to handle them.
The failure you most need to detect is the sync that quietly stopped. Alert on the absence of expected traffic, not only on the presence of errors.
Written by the Webnatrix engineering team. This is evergreen technical writing, maintained as our practice changes rather than published to a date.