# Reconciling Divergent Spatial Domain Replicas

Two replicas of a spatial domain disagree eventually — a dropped tombstone, an out-of-order arrival applied by wall-clock, a rebuild from a compacted feed that resurrected deleted features. The disagreement is silent, because both replicas answer every query successfully and neither has any reason to suspect the other. This guide detects divergence cheaply, localises it to the features involved, and reconciles it deterministically so both replicas converge on the same answer rather than on whichever one was checked. It applies the ordering and compaction rules from Domain Sync Protocols for Spatial Data within Federated Ownership & Routing Architecture.

# Prerequisites

# Step-by-Step Implementation

# 1. Detect divergence with a partition digest

Comparing feature by feature across two replicas is expensive. A per-partition digest reduces routine checking to one number each.

-- partition_digest.sql — one comparable value per partition, per replica.
-- Parameters: :partition (text), :precision (int)
SELECT
    md5(string_agg(row_digest, '' ORDER BY feature_id)) AS partition_digest,
    count(*)                                            AS feature_count
FROM (
    SELECT
        feature_id,
        md5(
            feature_id
            -- Snap before hashing: two replicas may hold coordinates differing in
            -- noise digits below the declared precision, which is not divergence.
            || ST_AsBinary(ST_SnapToGrid(geom, 1.0 / power(10, :precision)))::text
            || coalesce(attrs::text, '')
            || coalesce(deleted_at::text, '')
        ) AS row_digest
    FROM domain_features
    WHERE partition_key = :'partition'
) rows;

Verify the digest is stable across repeated runs on one replica before comparing two:

for i in 1 2; do
  psql "$REPLICA_A_DSN" --csv -f partition_digest.sql \
    -v partition="$PARTITION" -v precision=6 | tail -1
done
# Identical output. A digest that varies within one replica cannot detect anything.

# 2. Compare and localise only where digests differ

#!/usr/bin/env bash
# diverged.sh — compare every partition, then localise only the ones that differ.
set -euo pipefail

for p in $(psql "$REPLICA_A_DSN" -At -c "SELECT DISTINCT partition_key FROM domain_features"); do
  a=$(psql "$REPLICA_A_DSN" -At -f partition_digest.sql -v partition="$p" -v precision=6 | cut -d'|' -f1)
  b=$(psql "$REPLICA_B_DSN" -At -f partition_digest.sql -v partition="$p" -v precision=6 | cut -d'|' -f1)
  [ "$a" = "$b" ] && continue
  echo "DIVERGED: $p"
done

-- localise.sql — the specific features that differ, for one diverged partition.
-- Run with replica B attached as a foreign schema.
SELECT
    coalesce(a.feature_id, b.feature_id) AS feature_id,
    CASE
        WHEN b.feature_id IS NULL THEN 'missing_on_b'
        WHEN a.feature_id IS NULL THEN 'missing_on_a'
        WHEN a.deleted_at IS DISTINCT FROM b.deleted_at THEN 'tombstone_mismatch'
        WHEN NOT ST_Equals(a.geom, b.geom) THEN 'geometry_differs'
        ELSE 'attributes_differ'
    END AS divergence,
    a.event_time AS a_event_time,
    b.event_time AS b_event_time
FROM domain_features a
FULL OUTER JOIN replica_b.domain_features b USING (feature_id)
WHERE a.partition_key = :'partition' OR b.partition_key = :'partition';

# 3. Reconcile by event time, never by arrival time

The reconciliation rule has to be a total order both replicas compute identically, or they converge on different answers.

# reconcile.py — deterministic convergence.
def winner(a: dict, b: dict) -> dict:
    """Event time decides; the record identifier breaks exact ties. Never wall-clock
    arrival, which differs between replicas and makes the outcome depend on network
    timing rather than on the data."""
    if a["event_time"] != b["event_time"]:
        return a if a["event_time"] > b["event_time"] else b
    if a["record_id"] != b["record_id"]:
        return a if a["record_id"] > b["record_id"] else b
    return a                                    # genuinely identical


def resolve(divergence: str, a: dict | None, b: dict | None) -> dict:
    """A tombstone always wins over a live record at the same event time: a feature
    deleted and then re-observed produces a later event, so the ordering handles it.
    Treating absence as deletion is what resurrects features on a rebuild."""
    if a is None:
        return b
    if b is None:
        return a
    if divergence == "tombstone_mismatch":
        deleted = a if a.get("deleted_at") else b
        live = b if a.get("deleted_at") else a
        return deleted if deleted["event_time"] >= live["event_time"] else live
    return winner(a, b)

# 4. Repair through the feed, never by direct write

A replica repaired by hand diverges again on the next rebuild, because the feed still holds whatever caused the divergence.

# Emit the resolved records back onto the feed, so every replica — including ones
# not yet built — converges on the same answer.
python3 reconcile.py --partition "$PARTITION" --emit /tmp/resolved.ndjson
while read -r rec; do
  kcat -P -b "$BROKER" -t "domain.parcels" \
       -k "$(jq -r .feature_id <<<"$rec")" <<<"$rec"
done < /tmp/resolved.ndjson

# Re-digest both replicas after the feed has drained.

Verify convergence rather than assuming it:

a=$(psql "$REPLICA_A_DSN" -At -f partition_digest.sql -v partition="$PARTITION" -v precision=6)
b=$(psql "$REPLICA_B_DSN" -At -f partition_digest.sql -v partition="$PARTITION" -v precision=6)
[ "$a" = "$b" ] && echo "converged" || echo "STILL DIVERGED"

# Configuration Reference

# Common Failure Modes & Fixes

Every partition reports divergence. Root cause: the two replicas hold different coordinate precision, so every geometry hashes differently. Fix: snap to the declared precision before hashing, as above; if the underlying precision genuinely differs, that is a contract violation on one replica.

Digests match and query results differ. Root cause: the digest omits something the query reads — commonly an attribute column added later, or the tombstone field. Fix: include every column the contract publishes.

Reconciliation converges to different answers on each replica. Root cause: the tiebreak uses arrival time or a local sequence. Fix: event time plus a stable record identifier; both replicas must compute the same winner from the same inputs.

A repaired divergence returns after a rebuild. Root cause: repaired by direct write, leaving the feed unchanged. Fix: emit resolved records to the feed so the repair is part of the log rather than a local edit.

Deleted features reappear on a replica rebuilt from a compacted feed. Root cause: tombstone retention shorter than the rebuild lag, so absence was read as never-existed. Fix: retention must exceed the maximum permitted consumer lag, and that lag belongs in the feed’s published contract.

# FAQ

# Related

• Domain Sync Protocols for Spatial Data — the parent topic and the ordering/compaction rules
• Configuring Domain Sync for Real-Time Spatial Feeds — the feed whose guarantees this backstops
• Spatial Data Quality and Validation Standards — where divergence is reported as a drift signal