Data Synchronisation Patterns

Why synchronise

When the same data lives in two systems, it drifts apart unless you keep it aligned. Synchronisation is the ongoing job of making one system reflect changes from another — for example mirroring account records from an API into your own store.

There is no single right approach; the best choice depends on data volume, how often it changes, and how fresh it must be. The three broad patterns — full, incremental, and event-driven — trade simplicity against efficiency and latency. Understanding all three lets you match the strategy to the data rather than forcing one approach everywhere. Often a real system uses different patterns for different data sets.

Full vs incremental

A full sync fetches the entire data set every time and replaces your copy. It is dead simple and self-correcting — any past drift is wiped out — but it is wasteful and slow for large sets, and you cannot run it often.

An incremental sync fetches only what changed since last time, usually by a timestamp or change marker. It is far more efficient and can run frequently, keeping data fresh. The catch is correctness: you must reliably know what changed, handle deletions (which are easy to miss), and cope with clock differences. Many systems combine the two — frequent incremental syncs for freshness, with an occasional full sync to correct any accumulated drift.

Event-driven sync

Event-driven synchronisation reacts to change notifications — typically webhooks — applying each change as it happens. This gives the lowest latency and avoids polling for changes that have not occurred.

• Timely — your copy updates moments after the source does.
• Efficient — no wasted polls when nothing changes.
• Demanding — you must handle duplicate, missing and out-of-order events.

Because event delivery is rarely perfect, event-driven sync is strongest when backed by a periodic reconciliation that catches anything a dropped event missed. Treat the event stream as the fast path and reconciliation as the guarantee of completeness.

Conflicts and reconciliation

If data can change on both sides, two edits can collide. You need a conflict policy decided in advance — common choices are last-writer-wins, source-of-truth-wins, or flagging the conflict for a human. The wrong default can silently lose data, so choose deliberately.

Regardless of strategy, plan for drift. Bugs, outages and missed events mean copies will eventually diverge. A periodic reconciliation compares the two sides and repairs differences, acting as a safety net beneath whichever sync pattern you use. Designing reconciliation in from the start — rather than bolting it on after data goes wrong — is what keeps a long-running integration trustworthy. See eventual consistency for why temporary divergence is normal.