# Circuit Breaker Tuning for Geocoding Providers

Exponential backoff buys a degrading geocoder time to recover, but if the provider is genuinely failing, every retry still spends part of a fixed timeout budget the mesh cannot get back — so a hard breaker must eject the endpoint before it drains that budget across the whole fleet. This operation, part of Fallback Chains for Geocoding Services within the Federated Ownership & Routing Architecture, tunes Envoy outlier detection and connection-pool ceilings on geocoding upstreams — consecutive-5xx thresholds, ejection percentage, scan interval, and pending-request caps — so a failing provider is pulled from rotation fast and the request cascades to the fallback tier. It is the enforcement half of the pair whose retry logic lives in Configuring Exponential Backoff for Nominatim Fallback.

Figure — the breaker moves Closed → Open on consecutive failures, waits out the base ejection time, then probes Half-Open before it trusts the provider again.

# Prerequisites

# Step-by-Step Implementation

The breaker has two independent controls that must both be set: outlier detection ejects a failing endpoint, and connection-pool limits stop a saturated one from queuing work it can never serve. Tune both, then prove ejection actually fires under fault injection.

# 1. Set outlier detection on the primary geocoder

consecutive5xxErrors is the trip threshold; interval is how often Envoy scans; baseEjectionTime is the minimum time out of rotation, multiplied on each re-ejection; maxEjectionPercent caps how much of the pool can be ejected at once so you never eject your way into zero capacity.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: geocode-primary-circuit-breaker
  namespace: geospatial-mesh
spec:
  host: geocode-primary
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5        # trip after 5 straight 5xx
      interval: 10s                  # scan window
      baseEjectionTime: 30s          # min time ejected; scales per re-eject
      maxEjectionPercent: 50         # never eject more than half the pool
      minHealthPercent: 40           # stop ejecting below 40% healthy

Verify the rule is applied and parsed:

istioctl proxy-config cluster <gateway-pod> -n geospatial-mesh \
  --fqdn "geocode-primary.geospatial-mesh.svc.cluster.local" -o json | \
  jq '.[].outlierDetection'

# 2. Cap the connection pool to bound the blast radius

Outlier detection handles endpoints returning errors; connection-pool ceilings handle an endpoint that is merely slow. http2MaxRequests and http1MaxPendingRequests cap in-flight and queued work so a stalled provider sheds load with 503 instead of absorbing the whole timeout budget.

    connectionPool:
      tcp:
        maxConnections: 400
        connectTimeout: 250ms
      http:
        http1MaxPendingRequests: 200   # queue ceiling before backpressure
        http2MaxRequests: 400          # concurrent request ceiling
        maxRequestsPerConnection: 100  # recycle connections periodically

Verify the ceilings loaded:

istioctl proxy-config cluster <gateway-pod> -n geospatial-mesh \
  --fqdn "geocode-primary.geospatial-mesh.svc.cluster.local" -o json | \
  jq '.[].circuitBreakers.thresholds'

# 3. Confirm ejection fires under injected faults

A breaker never exercised is a breaker you cannot trust. Inject 5xx above the trip threshold and confirm the endpoint leaves rotation and traffic moves to the fallback within one scan interval.

# Force the primary to return 5xx, then watch the ejection counter climb.
kubectl exec <gateway-pod> -n geospatial-mesh -c istio-proxy -- \
  curl -s localhost:15000/stats | \
  grep -E 'geocode-primary.*(outlier_detection.ejections_active|ejections_enforced_consecutive_5xx)'

ejections_active should reach 1 within interval of the fifth consecutive 5xx, and the fallback tier’s request rate should rise correspondingly.

# 4. Tune the ejection window against the timeout budget

The breaker must trip well before the fleet’s 4s timeout budget is exhausted. With interval: 10s the worst-case detection latency is one scan; keep consecutive5xxErrors low enough that a fast-failing provider is ejected inside two or three scans, not minutes.

# Measure detection-to-ejection latency during a drill.
istioctl proxy-config endpoint <gateway-pod> -n geospatial-mesh \
  --cluster "outbound|443||geocode-primary.geospatial-mesh.svc.cluster.local" \
  -o json | jq '.[] | {address, health: .healthStatus}'

If a provider stays HEALTHY after clearly returning errors, consecutive5xxErrors is too high or the failures are not classified as 5xx — check for 429/connect-failure that outlier detection ignores by default.

# 5. Alert on ejection so a breaker trip pages, not hides

An ejected primary is an incident even though the fallback is masking it. Export the outlier gauges and alert when any geocoding cluster holds an active ejection for longer than one base ejection time.

# Any geocoding upstream ejected for > baseEjectionTime
max_over_time(
  envoy_cluster_outlier_detection_ejections_active{
    envoy_cluster_name=~"outbound.*geocode-.*"}[1m]
) > 0

Verify the gauge is scraped:

kubectl exec <gateway-pod> -n geospatial-mesh -c istio-proxy -- \
  curl -s localhost:15000/stats/prometheus | \
  grep 'envoy_cluster_outlier_detection_ejections_active'

# Configuration Reference

# Common Failure Modes & Fixes

• Symptom: the primary keeps draining the timeout budget and never ejects. Root cause: failures are 429 or connect-failure, which consecutive5xxErrors does not count. Fix: add consecutiveGatewayErrors and consecutiveLocalOriginFailures so connect and gateway faults also trip the breaker.
• Symptom: ejecting the primary takes the whole pool down. Root cause: maxEjectionPercent is too high or minHealthPercent too low, so Envoy ejects every endpoint at once. Fix: cap maxEjectionPercent: 50 and set minHealthPercent: 40 so a common upstream fault cannot zero out capacity.
• Symptom: the breaker flaps — eject, admit, eject — every interval. Root cause: baseEjectionTime is shorter than the provider’s recovery time, so the Half-Open probe keeps failing. Fix: raise baseEjectionTime; each re-ejection already multiplies it, but the base must exceed a typical recovery.
• Symptom: a slow-but-not-erroring provider stalls requests. Root cause: connection-pool ceilings are unset, so pending requests pile up inside the timeout. Fix: set http1MaxPendingRequests and http2MaxRequests so saturation sheds as 503 and cascades to the fallback.
• Symptom: the breaker trips silently and no one notices until the fallback also degrades. Root cause: no alert on ejections_active. Fix: wire the Step 5 PromQL alert so any geocoding ejection pages, and pair it with the Configuring Exponential Backoff for Nominatim Fallback retry metrics to localize the cause.

# FAQ

# Related

• Fallback Chains for Geocoding Services — parent reference for tiered geocoder resolution.
• Configuring Exponential Backoff for Nominatim Fallback — the retry logic the breaker backstops when it stops paying off.
• Building Fallback Chains for Routing APIs — the multi-tier cascade an ejection triggers.
• Mapping API Gateways to Distributed GIS Endpoints — the ingress route table these breakers protect.
• Federated Ownership & Routing Architecture — up one level to the routing architecture overview.