# HPA Configuration for Spatial Tile Cache Nodes

Autoscaling a tile-cache Deployment on CPU alone is a mismatch for spatial traffic, because a cache node saturates on request concurrency and p95 latency long before its CPU does — a burst of large-bbox map pans floods the cache while cores sit idle. This operation configures a Kubernetes HorizontalPodAutoscaler (v2) that scales the cache on a custom Prometheus metric surfaced through prometheus-adapter, with stabilization windows tuned to stop the fleet flapping. It is the scaling response to the burn-rate signals defined in SLA Monitoring for Spatial Data Products under Spatial Pipeline Orchestration & Observability, and it consumes the very same p95 series produced in Monitoring Vector Query p95 Latency.

# Prerequisites

Figure — Prometheus feeds a per-pod metric through prometheus-adapter to the HPA, which scales the tile-cache Deployment within min/max bounds.

# Step-by-Step Implementation

Each step is verifiable before the next; the adapter and HPA are wired bottom-up so a missing metric is caught before the HPA depends on it.

# 1. Expose the cache metric through prometheus-adapter

The adapter translates a Prometheus series into a Kubernetes custom metric. This rule surfaces per-pod requests-per-second under a stable metric name the HPA can target.

# prometheus-adapter-config.yaml (rules.custom)
rules:
  - seriesQuery: 'tile_cache_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "tile_cache_requests_total"
      as: "tile_cache_requests_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Verify: confirm the custom metric is served by the aggregated API.

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/geospatial-mesh/pods/*/tile_cache_requests_per_second" \
  | jq '.items[0]'

# 2. Define the HPA on the custom metric

Target an average of 50 requests per second per pod. The HPA reads the adapter metric and computes the desired replica count as the ratio of current to target.

# tile-cache-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tile-cache
  namespace: geospatial-mesh
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tile-cache
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: tile_cache_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"

Verify: apply and confirm the HPA reads a live metric value, not <unknown>.

kubectl apply -f tile-cache-hpa.yaml
kubectl get hpa tile-cache -n geospatial-mesh

# 3. Add stabilization windows to stop flapping

Spatial traffic is spiky, so an unguarded HPA thrashes replicas on every map pan. A long scale-down stabilization window holds capacity through short lulls while scale-up stays responsive.

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100          # may double pods per step
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # hold capacity 5m before shrinking
      policies:
        - type: Pods
          value: 2            # remove at most 2 pods per minute
          periodSeconds: 60

Verify: confirm the behavior block is registered on the HPA.

kubectl get hpa tile-cache -n geospatial-mesh -o jsonpath='{.spec.behavior}' | jq

# 4. Scale on p95 latency instead of throughput (optional)

Where the SLA is expressed in latency, target the recorded p95 directly so the cache scales out before the latency ceiling is breached. Reuse the tile:cache:p95_5m series exposed through the adapter as an Object or external metric.

  metrics:
    - type: Object
      object:
        metric:
          name: tile_cache_p95_seconds
        describedObject:
          apiVersion: v1
          kind: Service
          name: tile-cache
        target:
          type: Value
          value: "0.25"       # scale out before the 300ms objective

Verify: confirm the object metric resolves through the adapter.

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/geospatial-mesh/services/tile-cache/tile_cache_p95_seconds" \
  | jq '.items[0].value'

# 5. Observe a scaling event end to end

Drive load and watch the HPA move replicas, confirming it converges rather than oscillates.

kubectl describe hpa tile-cache -n geospatial-mesh | sed -n '/Events/,$p'
kubectl get deployment tile-cache -n geospatial-mesh -w   # watch replica count settle

# Configuration Reference

# Common Failure Modes & Fixes

HPA shows targets: <unknown>. Root cause: the adapter is not serving the metric, usually a seriesQuery that matches no series or a missing pod/namespace resource override. Fix: query custom.metrics.k8s.io with kubectl get --raw to confirm the metric exists before blaming the HPA.

Replicas oscillate every scrape. Root cause: no scale-down stabilization window, so the HPA shrinks the instant a spike clears and grows again immediately. Fix: set scaleDown.stabilizationWindowSeconds: 300 and cap the per-step scale-down policy.

Cache scales on CPU and never reacts to latency. Root cause: a leftover Resource CPU metric dominates because the HPA takes the max recommendation across metrics. Fix: remove the CPU metric or keep it only as a floor, and drive scaling from the request-rate or p95 metric.

Scale-up lags a traffic burst. Root cause: the adapter’s [2m] rate window smooths too aggressively for a sharp spike. Fix: shorten the rate window to [1m] for the scaling series and lower the scale-up stabilization window, accepting slightly noisier decisions.

HPA hits maxReplicas and latency still breaches. Root cause: the cache tier is genuinely undersized or an upstream origin is the bottleneck, not the cache. Fix: raise maxReplicas with a cost review and confirm origin capacity; a cache that is always cold cannot absorb the load a larger fleet would.

# FAQ

# Related

• Up to the parent: SLA Monitoring for Spatial Data Products
• Monitoring Vector Query p95 Latency — the p95 series this HPA can scale on
• Prometheus Alerting Rules for Tile Availability — the availability backstop when scaling lags
• Spatial Pipeline Orchestration & Observability — up to the section overview