# Configuring Exponential Backoff for Nominatim Fallback

When a primary geocoder starts to degrade — rising latency, intermittent 503s, sporadic rate-limit rejections — retrying it immediately and in lockstep only synchronizes every caller into a thundering herd that finishes the provider off. This operation, part of Fallback Chains for Geocoding Services within the Federated Ownership & Routing Architecture, configures exponential backoff with full jitter on the primary before failing over to a self-hosted Nominatim tier, sizing the base, cap, and retry count so retries spread out, respect the provider’s rate limit, and stay idempotent so a slow-but-eventual success is never geocoded twice. It works hand in glove with Circuit Breaker Tuning for Geocoding Providers, which ejects a provider the moment backoff stops paying off.

Figure — each retry waits a random interval inside an exponentially growing window, capped, until the retry budget is spent and the request fails over to Nominatim.

# Prerequisites

# Step-by-Step Implementation

Backoff is enforced in two places that must agree: the client library spreads its own retries with full jitter, and the Envoy gateway caps the total retry budget so no client can exceed the provider’s rate limit. Each step verifies before moving on.

# 1. Implement full-jitter backoff in the client

Full jitter — random(0, min(cap, base * 2**n)) — decorrelates retries across callers far better than equal or “equal jitter” schemes, which is what stops a thundering herd. Cap the window so a late attempt never blocks longer than the request budget.

import random, time, hashlib, requests

BASE, CAP, MAX_RETRIES = 0.2, 2.0, 3   # seconds, seconds, count

def geocode(query, crs="EPSG:4326"):
    key = hashlib.sha256(f"{query}|{crs}".encode()).hexdigest()
    headers = {"x-idempotency-key": key}
    for n in range(MAX_RETRIES + 1):
        r = requests.get("http://geocode-primary.mesh.internal/v1/search",
                         params={"q": query, "crs": crs}, headers=headers, timeout=1.0)
        if r.status_code < 500 and r.status_code != 429:
            return r.json()                      # success or a real 4xx
        delay = random.uniform(0, min(CAP, BASE * (2 ** n)))   # full jitter
        time.sleep(delay)
    return nominatim_fallback(query, crs, headers)  # budget exhausted

Verify the delays actually grow and vary run to run:

python3 -c "import random; BASE,CAP=0.2,2.0; \
print([round(random.uniform(0,min(CAP,BASE*2**n)),3) for n in range(4)])"
# windows grow 0.2 -> 0.4 -> 0.8 -> 1.6, jittered, none above 2.0

# 2. Route the fallback to Nominatim idempotently

The fallback carries the same x-idempotency-key, so if the primary eventually succeeded on a request the client had already abandoned, the dedup gate returns that result instead of geocoding again. This is the same key discipline the routing fallback chain uses.

def nominatim_fallback(query, crs, headers):
    r = requests.get("http://nominatim.mesh.internal/search",
                     params={"q": query, "format": "jsonv2"},
                     headers=headers, timeout=3.0)
    r.raise_for_status()
    hit = r.json()[0]
    # Normalize to the mesh CRS contract before returning.
    return {"lon": float(hit["lon"]), "lat": float(hit["lat"]), "crs": crs}

Verify a replayed query does not double-geocode:

KEY=$(printf '%s' "1600 Pennsylvania Ave|EPSG:4326" | sha256sum | cut -d' ' -f1)
for i in 1 2; do
  curl -s -D - -o /dev/null -H "x-idempotency-key: $KEY" \
    "http://geocode-primary.mesh.internal/v1/search?q=1600+Pennsylvania+Ave" | grep -i 'x-cache'
done   # request 2 should report x-cache: HIT

# 3. Cap the retry budget at the Envoy gateway

Gateway retries are the backstop for clients that misbehave. retryOn arms on the same conditions; perTryTimeout times each attempt; and the total attempts times the per-try timeout must stay inside the global timeout.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: geocode-primary-backoff
  namespace: geospatial-mesh
spec:
  hosts: ["geocode-primary.mesh.internal"]
  http:
  - route:
    - destination: { host: geocode-primary, subset: v1 }
    timeout: 4s
    retries:
      attempts: 3
      perTryTimeout: 1s                       # 3 x 1s < 4s global budget
      retryOn: 5xx,reset,connect-failure,retriable-status-codes
      retriableStatusCodes: [429]             # honor rate-limit as retriable
      backoff:
        baseInterval: 0.2s                     # matches client BASE
        maxInterval: 2s                        # matches client CAP

Verify the policy loaded exactly as committed:

istioctl proxy-config route <gateway-pod> -n geospatial-mesh \
  --name geocode-primary-backoff -o json | \
  jq '.virtualHosts[].routes[].route.retryPolicy | {numRetries, retryOn, perTryTimeout}'

# 4. Keep backoff inside the Nominatim rate limit

Nominatim tolerates roughly 1 rps per client. The failover path must not itself become a herd, so serialize fallbacks behind a token bucket and confirm the effective rate under a burst.

# Fire 10 fallbacks and confirm they land no faster than ~1 rps.
for i in $(seq 1 10); do
  curl -s -o /dev/null -w '%{time_starttransfer}\n' \
    "http://nominatim.mesh.internal/search?q=test$i&format=jsonv2"
done | awk '{print NR": "$1"s"}'

If the inter-arrival gap drops below one second, tighten the client-side bucket before Nominatim starts returning 429.

# Configuration Reference

# Common Failure Modes & Fixes

• Symptom: the primary recovers, then immediately collapses again. Root cause: fixed-interval retries synchronized every client into a thundering herd. Fix: switch to full jitter (random(0, min(cap, base*2**n))); confirm delays vary run to run with the Step 1 check.
• Symptom: clients keep hammering a rate-limited primary. Root cause: 429 was treated as a hard failure and never retried with backoff. Fix: add 429 to retriableStatusCodes and honor Retry-After when present.
• Symptom: the request times out before failover fires. Root cause: attempts × perTryTimeout ≥ timeout, so the budget is spent inside retries. Fix: keep 3 × 1s under the 4s global timeout, leaving headroom for the Nominatim hop.
• Symptom: an address is geocoded twice — once by a late primary success and once by the fallback. Root cause: missing or query-only idempotency key. Fix: derive the key from SHA-256(query+crs) and check it on both tiers, as in Step 2.
• Symptom: Nominatim starts returning 429 during an incident. Root cause: every abandoned primary request failed over at once, exceeding 1 rps. Fix: serialize fallbacks behind a token bucket and let the Circuit Breaker Tuning for Geocoding Providers eject the primary so retries stop sooner.

# FAQ

# Related

• Fallback Chains for Geocoding Services — parent reference for tiered geocoder resolution.
• Circuit Breaker Tuning for Geocoding Providers — ejects a provider once backoff stops helping.
• Building Fallback Chains for Routing APIs — the multi-tier cascade this backoff feeds into.
• Tuning Redis Idempotency Cache TTL for Spatial Jobs — sizing the dedup TTL that spans the retry window.
• Federated Ownership & Routing Architecture — up one level to the routing architecture overview.