# Building fallback chains for routing APIs

Deterministic fallback routing is the primary SLA safeguard when primary routing APIs within a federated mesh experience latency degradation, payload rejection, or upstream circuit-breaker trips. In a Geospatial Data Mesh & Domain-Driven Architecture, routing failures propagate asymmetrically across spatial tile boundaries and vector schema contracts. This guide specifies the exact Envoy-based configuration pattern, idempotent execution guarantees, diagnostic telemetry patterns, and escalation thresholds required to maintain sub-200ms routing SLAs during partial mesh degradation.

# 1. Configuration Pattern: Weighted Cluster Routing with Timeout Guardrails

Fallback chains must be enforced at the API gateway layer using explicit timeout budgets, retry isolation, and circuit-breaker thresholds. The following configuration establishes a three-tier routing hierarchy (Primary → Secondary → Static Cache) with strict per-hop timeout enforcement. This pattern directly aligns with the operational boundaries defined in Federated Ownership & Routing Architecture and prevents cascading timeout propagation across domain boundaries.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: routing-api-fallback-chain
  namespace: geospatial-mesh
spec:
  hosts: ["routing-api.mesh.internal"]
  http:
  - match:
    - uri:
        prefix: /v2/route
    route:
    - destination:
        host: routing-primary
        subset: v1
      weight: 100
    - destination:
        host: routing-secondary
        subset: v1
      weight: 0
    timeout: 1.5s
    retries:
      attempts: 1
      perTryTimeout: 0.5s
      retryOn: 5xx,reset,connect-failure,envoy-ratelimited
    headers:
      request:
        add:
          x-fallback-triggered: "false"
          x-fallback-reason: "none"
      response:
        add:
          x-mesh-routing-tier: "primary"
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: routing-primary-circuit-breaker
  namespace: geospatial-mesh
spec:
  host: routing-primary
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 25
    connectionPool:
      tcp:
        maxConnections: 1000
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 500
        http2MaxRequests: 1000

Critical implementation constraint: The perTryTimeout must be strictly less than the global timeout to allow the gateway to evaluate fallback eligibility before exhausting the request budget. Misalignment here causes synchronous blocking during [Async Execution for Heavy Spatial Queries], which violates domain isolation guarantees and forces unnecessary failovers. Header injection (x-fallback-triggered) must be evaluated downstream to prevent duplicate spatial computations when routing to secondary clusters.

# 2. Idempotent Execution & State Isolation

Fallback activation must not introduce state drift or duplicate geospatial computations. Implement the following execution guarantees:

• Idempotency Key Propagation: Inject x-idempotency-key: <sha256(payload+timestamp)> at the ingress gateway. Secondary routing clusters must validate this key against a distributed Redis-backed deduplication store before executing route resolution.
• Schema Contract Validation: Enforce strict protobuf/JSON schema validation at the fallback boundary. If the secondary cluster returns a payload violating [Schema Contracts for Vector/Tile Data], the gateway must reject the response and route to the static cache tier rather than propagating malformed geometry.
• Cross-Domain Routing Isolation: Fallback chains must respect domain ownership boundaries. Route resolution payloads crossing into adjacent spatial domains must carry x-domain-boundary: true headers to trigger [Cross-Domain Routing Strategies] that bypass heavy spatial joins and return pre-aggregated routing corridors.

# 3. Diagnostic Log Patterns & Telemetry Correlation

When fallback activation exceeds 12% of total routing requests over a 5-minute window, immediate isolation is required. Execute the following diagnostic sequence to identify the failure vector:

1. Verify Active Route Weights & Envoy Config:

   istioctl proxy-config route <gateway-pod> -n geospatial-mesh --name routing-api-fallback-chain -o json | jq '.routes[].route.weighted_clusters.clusters[].weight'
   

2. Extract Timeout & Retry Metrics:

   kubectl logs -l app=routing-gateway -n geospatial-mesh --tail=5000 | grep -E "upstream_rq_timeout|fallback_triggered|503|envoy_cluster_upstream_rq_retry"
   

3. Correlate with Mesh Telemetry (PromQL):

   # Retry-to-Timeout Ratio (Threshold: >0.35 indicates upstream saturation)
   rate(envoy_cluster_upstream_rq_retry[5m]) / rate(envoy_cluster_upstream_rq_timeout[5m])
   
   # Fallback Activation Rate (Threshold: >12% triggers P2)
   sum(rate(envoy_http_downstream_rq_xx{code="200", header_x_fallback_triggered="true"}[5m])) / sum(rate(envoy_http_downstream_rq_xx[5m]))
   

Root-cause analysis typically reveals one of three dependency failures:

• Schema Drift: Secondary cluster returns incompatible GeoJSON/Protobuf geometry. Validate against [Schema Contracts for Vector/Tile Data] and enforce strict content-type negotiation at the gateway.
• Connection Pool Exhaustion: maxConnections or http2MaxRequests thresholds breached. Scale connection pools or enable HTTP/2 multiplexing per [Latency Optimization for Spatial Routing] standards.
• Upstream Circuit-Breaker Ejection: consecutive5xxErrors threshold triggered prematurely. Adjust baseEjectionTime and verify health check endpoints return valid spatial topology hashes.

# 4. Escalation Paths & Incident Runbooks

Fallback chain degradation requires deterministic escalation aligned with enterprise operational standards. Adhere to the following thresholds and actions:

Runbook Execution Sequence:

1. Confirm fallback activation via Prometheus dashboard.
2. Execute istioctl proxy-status to verify configuration propagation across gateway pods.
3. If upstream_rq_timeout correlates with fallback spikes, reduce perTryTimeout to 0.3s and increase retryOn to include retriable-4xx.
4. If secondary cluster returns 503 or malformed geometry, immediately shift weight: 0 to routing-primary and weight: 100 to static-cache-cluster.
5. Document incident in mesh telemetry ledger; trigger post-incident review for [Fallback Chains for Geocoding Services] alignment.

# 5. Operational Validation & Continuous Testing

Fallback chains must be validated idempotently before production promotion. Implement the following CI/CD validation gates:

• Chaos Injection: Use Envoy fault injection to simulate 5xx and connect-failure responses at 15% volume. Verify automatic weight shift and header propagation.
• Timeout Budget Testing: Inject artificial latency (500ms, 800ms, 1.2s) at the primary cluster. Confirm gateway aborts primary routing at 1.5s global timeout and routes to secondary within 0.8s.
• Idempotency Verification: Replay identical routing payloads during fallback activation. Confirm downstream services return identical route geometries without duplicate computation logs.

Reference the official Envoy Proxy documentation on Route Configuration and Retry Policies and Istio’s Traffic Management Best Practices for upstream configuration validation standards.

# Related Pages

• Fallback Chains for Geocoding Services