# Fallback Chains for Geocoding Services

A tiered, circuit-breaker-driven resolution path that keeps coordinate lookups available when any single geocoding provider degrades or fails.

In enterprise geospatial platforms, coordinate resolution reliability is a foundational requirement for downstream spatial analytics, logistics routing, and asset tracking. A single-provider geocoding dependency introduces an unacceptable failure surface: when the resolver throttles, returns 5xx, or breaches its latency budget, the failure cascades into data-quality degradation and SLA violations across every consumer that touches an address. This page sits within the Federated Ownership & Routing Architecture and treats geocoding not as a remote call but as a governed, versioned resolution contract that each domain owns independently of ingestion or tile-rendering pipelines. It is the health-aware degradation tier that the API Gateway Mapping for GIS Services invokes as the final step of its routing precedence, so the two patterns must agree on idempotency keys, CRS normalization, and failure semantics.

Figure — A circuit-breaker cascade: each tier fails over to the next, with all results normalized to a canonical CRS.

# Architectural Boundaries & Design Rationale

A fallback chain exists to convert a hard dependency into a graceful gradient. Without it, the platform’s address-resolution availability is bounded by the weakest external provider; with it, availability becomes the union of several independent resolution sources, each with a distinct failure profile. The design principle is strict isolation: the chain is a stateless routing layer that owns provider health, retry budgets, and normalization rules, and it must not leak those concerns into ingestion or rendering. That separation mirrors the domain ownership model established by Spatial Domain Boundary Design — the geocoding domain governs its own resolution contract, and no consumer reaches a provider directly.

The chain is built from three ordered tiers, each chosen because its failure mode is uncorrelated with the others:

• Tier 1 — commercial resolver. Highest precision and freshest reference data, but rate-limited, billed per call, and subject to vendor-side outages. It is the default target and the most expensive to over-call, which is why idempotency matters most here.
• Tier 2 — open-source geocoder. A self-hosted Nominatim or Pelias deployment inside the mesh. It removes the external billing and rate-limit dependency at the cost of coverage gaps and staleness, so it absorbs traffic when Tier 1’s circuit opens.
• Tier 3 — cached spatial index. A last-resort lookup against a redis_cluster_geo or PostGIS index of previously resolved coordinates. It cannot resolve novel addresses, but it guarantees a bounded-latency answer for frequently repeated lookups during a full upstream outage.

Treating this as a federated routing concern prevents three recurring failure modes. First, retry storms: a naive client that retries a throttled Tier 1 multiplies load on an already-degraded provider; a circuit breaker that trips to Tier 2 sheds that load deterministically. Second, silent precision drift: when a fallback returns a lower-confidence match, consumers must be told, so every result carries a resolution_source and confidence field rather than masquerading as a Tier 1 answer. Third, CRS contamination: providers return coordinates in different spatial reference systems, and propagating an unnormalized result corrupts every downstream spatial join — which is why normalization to a canonical CRS is a boundary the chain enforces before any result leaves the domain. When a fallback fires, the corrected resolution must propagate to downstream caches and consumers atomically, which is why this layer integrates the Domain Sync Protocols for Spatial Data rather than letting each consumer discover the change independently.

# Specification & Contract Reference

Every tier in the chain declares an explicit contract: the timeout it is allowed, the retry budget it may spend, the circuit-breaker thresholds that govern its transitions, and the strictness of the schema check applied to its response. The surface below is the minimum a production chain must pin in version control before it is allowed to serve traffic.

The normalization contract is the load-bearing detail. Address strings are lower-cased, whitespace-collapsed, and component-ordered before hashing, so "123 Main St." and "123 main street" produce the same idempotency key and therefore the same deduplicated lookup. The chain’s response schema is shared with the producer-side definitions enforced by Schema Contracts for Vector/Tile Data, so a geocoding result that passes the chain also satisfies the gateway’s edge validation. Coordinate transformation matrices used to normalize tier outputs are version-controlled and validated against ISO 19112 spatial referencing standards to eliminate floating-point accumulation errors during bulk reconciliation.

# Production Implementation

A fallback chain is configured declaratively and managed through GitOps so that tier order, timeouts, and breaker thresholds are versioned alongside the contracts they enforce. The routing manifest below is the source of truth: each tier declares its endpoint, latency budget, retry budget, and circuit-breaker policy, and the idempotency block defines how requests are fingerprinted and deduplicated.

# enterprise-geocoding-routing-manifest.yaml
# Zero-trust: every endpoint is reached over mTLS with a per-tier scoped credential;
# no tier shares a token, so a compromised provider cannot pivot laterally.
routing:
  idempotency:
    key_strategy: "sha256(normalized_address + bounding_box)"
    ttl_seconds: 3600
    deduplication_store: "redis_cluster_geo"

  tiers:
    - name: "primary_commercial"
      endpoint: "https://resolver.internal/v1/geocode"
      timeout_ms: 1200
      retry_budget: 2
      circuit_breaker:
        failure_threshold: 5
        half_open_timeout_ms: 30000
        success_threshold: 3
      schema_validation: "strict"

    - name: "secondary_opensource"
      endpoint: "https://geocoder.mesh.internal/v2/resolve"
      timeout_ms: 2000
      retry_budget: 1
      circuit_breaker:
        failure_threshold: 10
        half_open_timeout_ms: 60000
        success_threshold: 5
      schema_validation: "strict"

    - name: "tertiary_cached_index"
      endpoint: "https://spatial-cache.mesh.internal/v1/lookup"
      timeout_ms: 500
      retry_budget: 0
      circuit_breaker:
        failure_threshold: 20
        half_open_timeout_ms: 120000
        success_threshold: 2
      schema_validation: "relaxed"

The resolver that consumes this manifest must guarantee idempotency end to end. Every retry — within a tier or across a failover — carries the same key, so a provider can safely return a cached result without re-billing or re-processing, and a transient network partition cannot enqueue duplicate work. The reference implementation below walks the tiers in order, honours each circuit breaker, and normalizes every output to EPSG:4326 before returning. Idempotency and per-tier credential isolation are called out explicitly because both are mandatory, not optional hardening.

# geocode_chain.py — tiered resolver with idempotency and per-tier circuit breakers.
# Run: python geocode_chain.py "123 Main St, Springfield" --bbox -89.7,39.7,-89.6,39.8
import hashlib
import httpx
from pyproj import Transformer

def idempotency_key(normalized_address: str, bbox: str) -> str:
    # Deterministic fingerprint — identical inputs dedupe to one upstream call.
    return hashlib.sha256(f"{normalized_address}|{bbox}".encode()).hexdigest()

def resolve(address: str, bbox: str, manifest, breakers, dedup) -> dict:
    norm = " ".join(address.lower().split())
    key = idempotency_key(norm, bbox)

    if (cached := dedup.get(key)) is not None:
        return cached  # idempotent replay: never re-call a provider for the same key

    for tier in manifest["routing"]["tiers"]:
        if breakers[tier["name"]].is_open():
            continue  # circuit open — shed load to the next tier deterministically
        try:
            # Per-tier scoped credential; no shared token across tiers (zero-trust).
            resp = httpx.get(
                tier["endpoint"],
                params={"q": norm, "bbox": bbox, "key": key},
                timeout=tier["timeout_ms"] / 1000,
                headers={"Idempotency-Key": key},
            )
            resp.raise_for_status()
            hit = resp.json()
            result = normalize_to_4326(hit, tier["name"])
            breakers[tier["name"]].record_success()
            dedup.setex(key, manifest["routing"]["idempotency"]["ttl_seconds"], result)
            return result
        except (httpx.TimeoutException, httpx.HTTPStatusError):
            breakers[tier["name"]].record_failure()  # may trip the circuit open
            continue

    return {"resolution_source": "none", "confidence": 0.0, "degraded": True}

def normalize_to_4326(hit: dict, source: str) -> dict:
    # Reproject every tier's output to the canonical CRS before it leaves the domain.
    transformer = Transformer.from_crs(hit["crs"], "EPSG:4326", always_xy=True)
    lon, lat = transformer.transform(hit["x"], hit["y"])
    return {
        "lon": round(lon, 7),
        "lat": round(lat, 7),
        "crs": "EPSG:4326",
        "resolution_source": source,
        "confidence": hit.get("confidence", 0.0),
    }

Heavy spatial workloads — batch reverse-geocoding for historical datasets, bulk address standardization — must never run on the synchronous resolution path, because a single multi-minute batch would exhaust the connection budget that real-time lookups depend on. Route those jobs through the dedicated queues owned by Async Execution for Heavy Spatial Queries, which apply backpressure and idempotent submission so a retried batch never enqueues twice. The procedural detail of wiring a chain to live routing APIs — manifest syntax, breaker tuning, and failover testing — is covered in Building fallback chains for routing APIs.

Security enforcement is zero-trust by default. Ingress to every tier requires mutual TLS, a per-tier scoped API key, and rate limiting aligned with the provider’s own quota; token rotation and credential isolation per tier mean a compromised provider cannot be used to reach another. Every fallback activation is logged with structured telemetry capturing resolution_source, confidence, and the transformation metadata, and the chain injects W3C Trace-Context so a degraded answer can be correlated across tiers, following OpenTelemetry semantic conventions.

# Diagnostic Runbook

When a chain misbehaves, the fault is almost always in breaker state, payload normalization, idempotency keying, or CRS reconciliation — rarely the raw network. Work the steps in order; each isolates one boundary the chain enforces.

1. Confirm chain activation state. Query the telemetry index for fallback_chain_activated=true within the target window and cross-reference provider health dashboards. This distinguishes transient network jitter from a sustained upstream outage and tells you which tier actually served the traffic.
2. Validate payload normalization. Inspect rejected payloads for schema drift. Address components must conform to the declared contract before Tier 1 is evaluated; a malformed payload that slips past validation points to a gateway misconfiguration, not a provider fault.
3. Audit circuit-breaker transitions. Read the half-open/closed states in the routing proxy and compare them against the manifest thresholds. Persistent 5xx or timeouts from Tier 1 should trip isolation automatically — a breaker that never opens is degrading silently and starving Tier 2.
4. Reconcile CRS drift. Compare fallback output coordinates against the canonical EPSG:4326 projection using the version-controlled transformation matrix. Validate precision against enterprise tolerances (typically ±0.0001° for web mapping) to catch a tier returning an unprojected or wrong-datum result.
5. Trace idempotency-key collisions. Monitor the redis_cluster_geo deduplication store for collisions that return a stale cache entry during rapid retry cycles, and confirm the ttl_seconds window aligns with the slowest tier’s provider rate-limit window.
6. Verify manifest integrity. Confirm the deployed routing manifest revision matches the Git SHA in source control. A partial GitOps apply can leave a stale endpoint or a retired credential pinned on one tier.
7. Replay against a synthetic outage. If the cause is still unclear, force Tier 1’s circuit open in a staging drill and re-run step 1 to confirm traffic fails over within the latency budget and that confidence scores degrade as expected.

# SLA Targets & Performance Baselines

The chain carries thin, predictable overhead; the precision/latency trade-off lives in which tier serves the request. The budget below is what the resolution domain must hold so downstream spatial SLAs stay meaningful.

Holding these baselines relies on keeping the cached index warm with the live resolution stream, multiplexing connections to each tier, and keeping breaker cooldowns event-driven rather than fixed — so a recovered provider is re-admitted within its half_open_timeout_ms rather than on a coarse schedule.

# Governance & Compliance Notes

Resolution decisions are governance decisions, so the chain treats its manifest as an auditable artifact. Tier order, timeouts, breaker thresholds, and credential bindings are committed to source control, reviewed, and promoted only after CI validates schema compliance, tests idempotency replay, and simulates each breaker’s failover under synthetic load. This keeps the resolution contract aligned with the catalog records described in Metadata Cataloging for Raster/Vector, so a consumer-visible geocoding endpoint maps to a governed, documented data product with a known lineage.

Audit requirements are concrete. Every fallback activation is written to an append-only, tenant-partitioned stream carrying resolution_source, confidence, the idempotency key, and the transformation metadata, so a compliance reviewer can reconstruct exactly which tier answered any historical request and at what precision. Where jurisdictional constraints apply — data-residency rules that bind a tenant’s address data to a specific region — the chain refuses to route to an out-of-region tier even during an outage, degrading to the in-region cached index rather than crossing a residency boundary. Regularly rotate tiers during chaos-engineering drills to validate failover latency, credential rotation, and data fidelity across every mesh domain, and keep the transformation matrices under the same review gate as the manifest so a datum change can never ship unaudited.

# Related

• Up to the parent: Federated Ownership & Routing Architecture
• API Gateway Mapping for GIS Services — the ingress layer that invokes this chain as its final routing tier
• Domain Sync Protocols for Spatial Data — how corrected resolutions propagate atomically across the mesh
• Async Execution for Heavy Spatial Queries — the queues that own batch reverse-geocoding off the synchronous path
• Building fallback chains for routing APIs — manifest syntax and failover testing in detail