# Building fallback chains for routing APIs

When a primary routing API inside a federated mesh degrades — latency creep, payload rejection, or an upstream circuit-breaker trip — the failure does not stay local: route resolution propagates asymmetrically across spatial tile boundaries and vector schema contracts, and every consumer holding a sub-200ms SLA pays for it. This page is a focused operational procedure for wiring a deterministic, multi-tier fallback chain (Primary → Secondary → Static Cache) at the Envoy/Istio ingress so that partial mesh degradation never becomes a cascading outage. It sits under the Fallback Chains for Geocoding Services reference within the broader Federated Ownership & Routing Architecture, and it assumes the ingress route table it extends was already wired according to Mapping API gateways to distributed GIS endpoints. Get the timeout budgets and idempotency semantics right and a failed primary fails over silently, without duplicate spatial computation or malformed geometry reaching a consumer.

# Prerequisites

# Step-by-Step Implementation

Fallback chains are enforced at the gateway, never inside application code, so a degraded upstream is shed before it consumes the request budget. Each step below is verifiable with a diagnostic command before you proceed.

# 1. Define the tiered route table with timeout guardrails

The VirtualService establishes a three-tier hierarchy with a strict global timeout and a perTryTimeout that is deliberately smaller, so the gateway can evaluate fallback eligibility before the budget is exhausted. The primary carries weight: 100 in steady state; weight is shifted at runtime when an incident is declared. This aligns with the routing precedence defined across the Federated Ownership & Routing Architecture and prevents timeout propagation across domain boundaries.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: routing-api-fallback-chain
  namespace: geospatial-mesh
spec:
  hosts: ["routing-api.mesh.internal"]
  http:
  - match:
    - uri:
        prefix: /v2/route
    route:
    - destination:
        host: routing-primary
        subset: v1
      weight: 100
    - destination:
        host: routing-secondary
        subset: v1
      weight: 0
    timeout: 1.5s
    retries:
      attempts: 1
      perTryTimeout: 0.5s          # MUST stay < global timeout
      retryOn: 5xx,reset,connect-failure,envoy-ratelimited
    headers:
      request:
        add:
          x-fallback-triggered: "false"
          x-fallback-reason: "none"
      response:
        add:
          x-mesh-routing-tier: "primary"

If perTryTimeout >= timeout, the single retry attempt consumes the whole budget and the gateway has no time to cascade to the secondary before the request expires — the most common silent misconfiguration in this pattern.

Verify: confirm the weighted clusters loaded exactly as committed.

istioctl proxy-config route <gateway-pod> \
  -n geospatial-mesh \
  --name routing-api-fallback-chain \
  -o json | \
  jq '.virtualHosts[].routes[].route.weightedClusters.clusters[] | {name, weight}'

# 2. Bound the primary with outlier detection

The DestinationRule ejects an unhealthy primary endpoint with circuit-breaker thresholds, so consecutive 5xx responses pull the endpoint out of rotation rather than letting the gateway keep paying the per-try timeout. Connection-pool ceilings cap the blast radius of a saturated upstream.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: routing-primary-circuit-breaker
  namespace: geospatial-mesh
spec:
  host: routing-primary
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 25
    connectionPool:
      tcp:
        maxConnections: 1000
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 500
        http2MaxRequests: 1000

Verify: confirm the breaker gauges are exported per cluster.

istioctl proxy-config endpoint <gateway-pod> -n geospatial-mesh \
  --cluster "outbound|443||routing-primary.geospatial-mesh.svc.cluster.local" \
  -o json | jq '.[].healthStatus'

# 3. Enforce idempotent fallback execution

A fallback hop must not re-run a spatial computation that the primary already started. Inject x-idempotency-key: <sha256(payload+timestamp_window)> at ingress; the secondary cluster validates the key against the Redis dedup store before resolving a route, returning the cached corridor on a hit. Payloads crossing into an adjacent spatial domain carry x-domain-boundary: true, which triggers the Cross-Domain Routing Strategies bypass so the fallback returns a pre-aggregated corridor instead of re-running a heavy spatial join. Any secondary response that violates the producer contract is rejected — not propagated — and routed to the static cache tier, keeping fallback aligned with Schema Contracts for Vector/Tile Data.

Verify: replay an identical payload during a forced fallback and confirm the second request hits cache, not the upstream.

KEY=$(printf '%s' "from=-1.20,52.60|to=-0.98,52.81|crs=EPSG:4326|tw=2026-06-26T10" | sha256sum | cut -d' ' -f1)
for i in 1 2; do
  curl -s -D - -o /dev/null \
    -H "x-idempotency-key: $KEY" \
    -H "x-domain-boundary: true" \
    "http://routing-api.mesh.internal/v2/route" | grep -i 'x-cache'
done   # request 2 should report x-cache: HIT

# 4. Wire telemetry to fallback-activation thresholds

Fallback that exceeds 12% of total routing requests over a 5-minute window is an incident signal, not noise. Export the retry/timeout counters and the per-tier request rate to Prometheus, then alert on the activation ratio and the retry-to-timeout ratio that precedes upstream saturation.

# Retry-to-timeout ratio (> 0.35 indicates upstream saturation)
rate(envoy_cluster_upstream_rq_retry_total[5m])
  / rate(envoy_cluster_upstream_rq_timeout_total[5m])

# Fallback activation rate (> 0.12 triggers P2)
sum(rate(istio_requests_total{response_code="200",
  request_protocol="http",
  destination_service_name="routing-secondary"}[5m]))
/ sum(rate(istio_requests_total{destination_service_name=~"routing-.*"}[5m]))

Verify: pull the raw retry/timeout counters straight from the gateway sidecar.

kubectl exec <gateway-pod> -n geospatial-mesh -c istio-proxy -- \
  curl -s localhost:15000/stats/prometheus | \
  grep -E 'envoy_cluster_upstream_rq_(retry|timeout)_total'

# 5. Validate the chain before promotion

A fallback chain that has never been exercised is a fallback chain that does not work. Gate promotion on chaos injection: use Envoy fault injection to return 5xx and connect-failure at 15% volume, then inject artificial latency (500ms, 800ms, 1.2s) at the primary and confirm the gateway aborts at the 1.5s global timeout and resolves through the secondary within 0.8s. Topology changes that move a fallback target must arrive through the Domain Sync Protocols for Spatial Data rather than manual edits, so the route table and tenant entitlements move atomically.

# Force 25% of primary traffic to the static cache during a drill
kubectl patch virtualservice routing-api-fallback-chain -n geospatial-mesh --type merge -p \
  '{"spec":{"http":[{"route":[{"destination":{"host":"routing-primary","subset":"v1"},"weight":75},
   {"destination":{"host":"static-cache-cluster","subset":"v1"},"weight":25}]}]}}'
istioctl proxy-status   # confirm config propagated to every gateway pod

# Configuration Reference

# Common Failure Modes & Fixes

Read the x-fallback-reason header and the per-tier metrics together — the pair localizes the failure to a specific hop.

Fallback never fires — the request just times out at the client. Root cause: perTryTimeout >= timeout, so the lone retry consumes the whole budget and the gateway never reaches the secondary. Fix: set perTryTimeout: 0.5s against the 1.5s global budget and re-dump the route — istioctl proxy-config route <gateway-pod> -n geospatial-mesh -o json | jq '..|.perTryTimeout? // empty'.

Fallback activation rate climbs above 12% (5m). Root cause: the primary is shedding 5xx faster than consecutive5xxErrors ejects it, usually from connection-pool exhaustion. Fix: confirm http2MaxRequests / http1MaxPendingRequests headroom, scale the primary horizontally, and verify outlier detection is actually ejecting — escalate to platform-engineering on-call as a P2.

Secondary returns malformed geometry and consumers degrade silently. Root cause: schema drift between primary and secondary (coordinate order flip, nullable attribute, mismatched tile matrix set such as ogc-tiles/v1.2.0-crs:EPSG:4326-res:10m). Fix: enforce strict content-type negotiation at the fallback boundary so a contract violation is rejected to the static cache, then roll back the secondary deployment per Schema Contracts for Vector/Tile Data.

Duplicate route corridors / double compute during fallover. Root cause: the x-idempotency-key is absent or its timestamp_window is too coarse, so the secondary re-runs a job the primary already materialized. Fix: inject the key at ingress, narrow the window to 1–5 minute buckets, and confirm from, to, and crs are all inside the digest before the TTL is set.

Cross-domain timeout exceeds 2s under fallback. Root cause: the fallback path is running a synchronous spatial join across a domain boundary instead of bypassing it. Fix: set x-domain-boundary: true to engage the pre-aggregated corridor path, and offload genuinely heavy joins to the async worker described in Optimizing async execution for spatial joins.

When degradation passes 25% activation (5m), declare a P1: shift weight: 0 on routing-primary and weight: 100 on static-cache-cluster, disable the heavy spatial query endpoints, and close the incident with a blameless review of timeout calibration and breaker thresholds.

# FAQ

# Related

• Fallback Chains for Geocoding Services — the parent reference for tiered, circuit-breaker-driven resolution.
• Mapping API gateways to distributed GIS endpoints — the ingress route table this fallback chain extends.
• Optimizing async execution for spatial joins — where heavy joins go when a synchronous fallback would breach budget.
• Federated Ownership & Routing Architecture — up one level to the routing architecture overview.