# Configuring domain sync for real-time spatial feeds

When synchronizing real-time spatial feeds across a federated mesh, the primary failure vector is schema drift in vector and tile payloads during cross-domain propagation. Silent coordinate-system mismatches (EPSG:4326 vs EPSG:3857) and topology degradation bypass standard validation layers, corrupting downstream tile generators and routing engines. This page is the concrete operational procedure for wiring a change-data-capture pipeline that enforces strict contract validation at the ingestion boundary and commits offsets deterministically so geometry payloads and their positions stay atomic. It sits directly under the Domain Sync Protocols for Spatial Data reference within the broader Federated Ownership & Routing Architecture, and it assumes the payload shape it propagates is already governed by Schema Contracts for Vector/Tile Data. Get the offset and contract handling right and you hold sub-second routing SLAs while preventing the cascading partition rebalances that desync a federated estate.

# Prerequisites

# Step-by-Step Implementation

Sync resolution must decouple capture from dispatch: the connector captures spatial state changes, the registry enforces the contract, and offsets commit inside the producer transaction. Each step below is verifiable with a diagnostic command before you proceed.

# 1. Deploy the deterministic source connector

Deploy the following Kafka Connect distributed worker configuration. The architecture relies on schema.compatibility=BACKWARD_TRANSITIVE to permit additive field evolution while rejecting structural geometry mutations. The RegexRouter transform rewrites the destination topic name from the source table so spatial partition locality is preserved, and errors.tolerance=none routes any malformed envelope straight to spatial.dlq.geo instead of silently dropping it.

{
  "name": "spatial-feed-sync-prod",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "pg-primary-geo.internal",
    "database.port": "5432",
    "database.dbname": "mesh_spatial_registry",
    "table.include.list": "public.realtime_feeds,public.vector_tiles",
    "plugin.name": "pgoutput",
    "schema.registry.url": "https://schema-registry.mesh.internal:8081",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "https://schema-registry.mesh.internal:8081",
    "value.converter": "io.confluent.connect.json.JsonSchemaConverter",
    "value.converter.schema.registry.url": "https://schema-registry.mesh.internal:8081",
    "value.converter.schemas.enable": "true",
    "transforms": "routeByEnvelope",
    "transforms.routeByEnvelope.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.routeByEnvelope.regex": "mesh_spatial_registry\\.public\\.(.*)",
    "transforms.routeByEnvelope.replacement": "spatial.sync.$1",
    "offset.flush.timeout.ms": "10000",
    "max.batch.size": "2048",
    "poll.interval.ms": "50",
    "tasks.max": "4",
    "errors.tolerance": "none",
    "errors.deadletterqueue.topic.name": "spatial.dlq.geo",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true"
  }
}

ExtractField$Key extracts a single field from a Kafka record key (a Struct), not from the geometry payload, so it cannot route on a spatial envelope. Use RegexRouter on the topic name (derived from the source table) as shown, and implement spatial partitioning in a custom producer-side Partitioner — the Debezium connector does not natively support geometry-based routing transforms.

Apply via the Kafka Connect REST API. Use PUT for idempotent deployment, which overwrites existing connector state without creating duplicates:

curl -X PUT -H "Content-Type: application/json" \
  --data @spatial-sync-config.json \
  http://connect-cluster.mesh.internal:8083/connectors/spatial-feed-sync-prod/config

Verify: confirm partition assignment and consumer-group stability before sending production load.

kafka-consumer-groups \
  --bootstrap-server broker-1.mesh.internal:9092 \
  --describe \
  --group spatial-feed-sync-prod

# 2. Bind offset commits to the producer transaction

Spatial routing requires deterministic offset commits to prevent duplicate tile generation or coordinate-interpolation gaps. Enable exactly.once.source.support=enabled in the distributed worker configuration (connect-distributed.properties) so offset commits bind to the producer transaction and a geometry payload is never committed without its offset. offset.flush.timeout.ms=10000 prevents worker threads from blocking indefinitely during high-throughput ingestion bursts; if the timeout is breached, the connector pauses polling, logs OffsetFlushTimeoutException, and triggers backpressure handling.

Verify: confirm exactly-once is active on the running worker.

curl -s http://connect-cluster.mesh.internal:8083/connectors/spatial-feed-sync-prod/status \
  | jq '.connector.state, .tasks[].state'

# 3. Validate the schema registry contract

Query the active schema version for the vector_tiles subject and cross-reference the geometry field type against the OGC GeoJSON specification RFC 7946. This is the contract that the gateway-level checks in Schema Contracts for Vector/Tile Data also enforce, so a payload that passes here is not rejected downstream.

curl -s http://schema-registry.mesh.internal:8081/subjects/vector_tiles-value/versions/latest \
  | jq '.schema' | python3 -m json.tool

Failure indicators: a present crs property (RFC 7946 explicitly removes the crs member, so its presence signals a pre-RFC 7946 schema), type changed from object to string, or an altered coordinates array depth.

# 4. Isolate offset lag from payload corruption

When downstream consumers report InvalidGeometryException or tile misalignment, the first triage decision is schema drift versus offset lag. Run consumer-group lag analysis: high lag with zero dead-letter throughput indicates a downstream processing bottleneck, while zero lag with high dead-letter throughput indicates schema or topology violations.

kafka-consumer-groups \
  --bootstrap-server broker-1.mesh.internal:9092 \
  --describe \
  --group spatial-feed-sync-prod \
  --members \
  --verbose

Verify: filter Connect worker logs for deterministic failure signatures.

grep -E "InvalidGeometryException|SchemaVersionMismatch|OffsetCommitFailed|TopologyDegradation" \
  /var/log/kafka/connect-distributed.log | tail -n 50

# 5. Restart tasks and reset offsets safely

When a single task fails, restart only that task to avoid a full connector rebalance — a rebalance pauses every partition and is the most common cause of a self-inflicted sync gap.

curl -X POST \
  http://connect-cluster.mesh.internal:8083/connectors/spatial-feed-sync-prod/tasks/0/restart

Never use --reset-offsets --to-earliest on production spatial topics; it replays the entire history into tile generators. Reset to a specific position aligned with the last valid geometry envelope checkpoint instead:

kafka-consumer-groups \
  --bootstrap-server broker-1.mesh.internal:9092 \
  --group spatial-feed-sync-prod \
  --reset-offsets \
  --to-offset 3 \
  --topic spatial.sync.vector_tiles:0 \
  --execute

# Configuration Reference

# Common Failure Modes & Fixes

Pin every desync to a deterministic log signature first — the failure flag tells you which layer to act on.

SchemaVersionMismatch — zero consumer lag, high dead-letter throughput. Root cause: the registry rejected a payload for a BACKWARD_TRANSITIVE violation (a producer published a structural geometry mutation). Fix: pause the connector, revert the subject to the last stable version, and replay from the checkpoint — curl -X PUT http://connect-cluster.mesh.internal:8083/connectors/spatial-feed-sync-prod/pause.

TopologyDegradation — geometry rejected during WKT parsing. Root cause: self-intersection or invalid ring ordering in the source feature. Fix: quarantine the producing pipeline and validate the offending geometry with ogrinfo -al -so against the source table before re-admitting it; do not widen tolerance to force it through.

OffsetCommitFailed — task drops out of the consumer group. Root cause: the worker lost coordination with the group coordinator, usually a network partition or broker unavailability. Fix: confirm broker reachability (kafka-broker-api-versions --bootstrap-server broker-1.mesh.internal:9092), then restart only the affected task per Step 5.

Geometry-based routing silently fails — records land on the wrong partition. Root cause: an ExtractField$Key transform was used to route by envelope, but it only reads the record-key Struct. Fix: route by topic name with RegexRouter and move spatial partitioning into a custom producer Partitioner.

Sub-second routing SLA breached, TopologyDegradation reaching tile generators. Root cause: a corrupt envelope is propagating across domains faster than triage. Fix: isolate the affected domain, trip the Federated Ownership & Routing Architecture circuit breaker, and shed read traffic to the Fallback Chains for Geocoding Services until the contract is restored.

Tiered escalation keeps these actions bounded. Log every action in the incident tracker with the correlation ID from the X-Trace-Id header.

For a P1, halt the connector, drain pending offsets to the current position, validate the contract against the latest stable registry release, then resume and watch consumer-lag and record-error-rate for 15 minutes before closing:

curl -X PUT http://connect-cluster.mesh.internal:8083/connectors/spatial-feed-sync-prod/pause
kafka-consumer-groups \
  --bootstrap-server broker-1.mesh.internal:9092 \
  --group spatial-feed-sync-prod \
  --reset-offsets --to-current --execute

All spatial sync configurations must be version-controlled in the infrastructure-as-code repository, and manual curl overrides are prohibited outside active incident windows. For heavy spatial query backpressure, route work to the Async Execution for Heavy Spatial Queries pipelines rather than blocking the ingestion worker, and let validated changes travel through the Cross-Domain Routing Strategies layer.

# FAQ

# Related

• Domain Sync Protocols for Spatial Data — the parent reference for cross-domain state propagation.
• Enforcing schema contracts for GeoJSON and Shapefiles — the format-level validation rules behind the registry contract.
• Mapping API gateways to distributed GIS endpoints — how synced topology changes reach the ingress route table.
• Federated Ownership & Routing Architecture — up one level to the routing architecture overview.