HPA Configuration for Spatial Tile Cache Nodes

Autoscaling a tile-cache Deployment on CPU alone is a mismatch for spatial traffic, because a cache node saturates on request concurrency and p95 latency long before its CPU does — a burst of large-bbox map pans floods the cache while cores sit idle. This operation configures a Kubernetes HorizontalPodAutoscaler (v2) that scales the cache on a custom Prometheus metric surfaced through prometheus-adapter, with stabilization windows tuned to stop the fleet flapping. It is the scaling response to the burn-rate signals defined in SLA Monitoring for Spatial Data Products under Spatial Pipeline Orchestration & Observability, and it consumes the very same p95 series produced in Monitoring Vector Query p95 Latency.

Prerequisites

Requirement Value / Assumption Notes
Kubernetes >= 1.27, autoscaling/v2 Supports behavior stabilization
Metrics bridge prometheus-adapter >= 0.11 Exposes custom.metrics.k8s.io
Prometheus Scraping the tile-cache exporter Source of the custom metric
Custom metric tile_cache_requests_per_second per pod Or tile:cache:p95_5m
Workload Deployment/tile-cache in geospatial-mesh Stateless cache nodes
CRS context Serves EPSG:3857 tiles from EPSG:4326 source Cache is CRS-agnostic
Access role hpa-editor + apiservice-viewer (RBAC) Apply HPA + read adapter API
Verification kubectl against the Kubernetes cluster get hpa, get --raw

Figure — Prometheus feeds a per-pod metric through prometheus-adapter to the HPA, which scales the tile-cache Deployment within min/max bounds.

Custom-metric autoscaling path for the tile cache The tile-cache pods expose a requests-per-second metric that Prometheus scrapes. prometheus-adapter surfaces it on the custom metrics API. The HorizontalPodAutoscaler reads that metric and scales the tile-cache Deployment between a minimum and maximum replica count, with stabilization windows damping scale-down. Cache pods emit req/s Prometheus scrapes metric prom-adapter custom.metrics API HPA v2 target 50 rps Deployment min 3 · max 30

Step-by-Step Implementation

Each step is verifiable before the next; the adapter and HPA are wired bottom-up so a missing metric is caught before the HPA depends on it.

1. Expose the cache metric through prometheus-adapter

The adapter translates a Prometheus series into a Kubernetes custom metric. This rule surfaces per-pod requests-per-second under a stable metric name the HPA can target.

yaml
# prometheus-adapter-config.yaml (rules.custom)
rules:
  - seriesQuery: 'tile_cache_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "tile_cache_requests_total"
      as: "tile_cache_requests_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Verify: confirm the custom metric is served by the aggregated API.

bash
kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/geospatial-mesh/pods/*/tile_cache_requests_per_second" \
  | jq '.items[0]'

2. Define the HPA on the custom metric

Target an average of 50 requests per second per pod. The HPA reads the adapter metric and computes the desired replica count as the ratio of current to target.

yaml
# tile-cache-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tile-cache
  namespace: geospatial-mesh
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tile-cache
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: tile_cache_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"

Verify: apply and confirm the HPA reads a live metric value, not <unknown>.

bash
kubectl apply -f tile-cache-hpa.yaml
kubectl get hpa tile-cache -n geospatial-mesh

3. Add stabilization windows to stop flapping

Spatial traffic is spiky, so an unguarded HPA thrashes replicas on every map pan. A long scale-down stabilization window holds capacity through short lulls while scale-up stays responsive.

Why p95 latency gets worse for two minutes after the autoscaler adds a replicaFive points along a scale-up. At zero seconds a traffic burst pushes p95 latency past the ceiling. At about thirty seconds the HPA observes the custom metric and adds a replica. At forty-five seconds the new pod is ready and the mesh routes it an equal share of traffic — but its cache is empty, so every one of those requests is a miss and aggregate p95 rises further. By around two minutes the working set has warmed and p95 falls below where it started. The dip in between is what makes a latency-driven HPA oscillate unless the scale-up stabilization window covers the whole warm-up.t+0s · burstp95 breaches ceilingt+30s · HPA actsreplica addedt+45s · pod readycold cache takes traffict+75s · p95 peaksmisses fall throught+120s · warmp95 below baselineA stabilization window shorter than the warm-up makes the HPA scale up again at t+75s

yaml
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100          # may double pods per step
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # hold capacity 5m before shrinking
      policies:
        - type: Pods
          value: 2            # remove at most 2 pods per minute
          periodSeconds: 60

Verify: confirm the behavior block is registered on the HPA.

bash
kubectl get hpa tile-cache -n geospatial-mesh -o jsonpath='{.spec.behavior}' | jq

4. Scale on p95 latency instead of throughput (optional)

Where the SLA is expressed in latency, target the recorded p95 directly so the cache scales out before the latency ceiling is breached. Reuse the tile:cache:p95_5m series exposed through the adapter as an Object or external metric.

Which signal to scale each tier on, and what goes wrong when they share oneThree candidate scaling signals against the two tiers. CPU is a poor signal for the cache tier, which is memory-bound, and a reasonable one for the renderer. Cache hit ratio and request rate are the right signals for the cache tier and meaningless for the renderer. Query queue depth is the right signal for the renderer and irrelevant to the cache. The final row records the failure mode of coupling them: a burst of cache misses scales the cache, which does not help because the misses are the problem, while the renderer that is actually saturated stays at its original replica count.Cache tierRenderer tierCPU utilisationwrong — memory-boundusableCache hit ratioright signalmeaninglessQuery queue depthmeaninglessright signalCoupled to one signalscales uselesslystays saturated

yaml
  metrics:
    - type: Object
      object:
        metric:
          name: tile_cache_p95_seconds
        describedObject:
          apiVersion: v1
          kind: Service
          name: tile-cache
        target:
          type: Value
          value: "0.25"       # scale out before the 300ms objective

Verify: confirm the object metric resolves through the adapter.

bash
kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/geospatial-mesh/services/tile-cache/tile_cache_p95_seconds" \
  | jq '.items[0].value'

5. Observe a scaling event end to end

Drive load and watch the HPA move replicas, confirming it converges rather than oscillates.

bash
kubectl describe hpa tile-cache -n geospatial-mesh | sed -n '/Events/,$p'
kubectl get deployment tile-cache -n geospatial-mesh -w   # watch replica count settle

Configuration Reference

Field Scope Required value Effect
minReplicas HPA spec 3 Floor for availability under zero load
maxReplicas HPA spec 30 Ceiling bounding cost/blast radius
target.averageValue Pods metric 50 Per-pod req/s the HPA holds
scaleUp.stabilizationWindowSeconds behavior 30 Responsiveness to bursts
scaleDown.stabilizationWindowSeconds behavior 300 Damps flapping on short lulls
scaleDown.policies.value behavior 2 pods/60s Max shrink rate
metricsQuery rate window adapter [2m] Smooths the per-pod rate
Object metric value latency HPA 0.25 p95 seconds triggering scale-out

Common Failure Modes & Fixes

HPA shows targets: <unknown>. Root cause: the adapter is not serving the metric, usually a seriesQuery that matches no series or a missing pod/namespace resource override. Fix: query custom.metrics.k8s.io with kubectl get --raw to confirm the metric exists before blaming the HPA.

Replicas oscillate every scrape. Root cause: no scale-down stabilization window, so the HPA shrinks the instant a spike clears and grows again immediately. Fix: set scaleDown.stabilizationWindowSeconds: 300 and cap the per-step scale-down policy.

Cache scales on CPU and never reacts to latency. Root cause: a leftover Resource CPU metric dominates because the HPA takes the max recommendation across metrics. Fix: remove the CPU metric or keep it only as a floor, and drive scaling from the request-rate or p95 metric.

Scale-up lags a traffic burst. Root cause: the adapter’s [2m] rate window smooths too aggressively for a sharp spike. Fix: shorten the rate window to [1m] for the scaling series and lower the scale-up stabilization window, accepting slightly noisier decisions.

HPA hits maxReplicas and latency still breaches. Root cause: the cache tier is genuinely undersized or an upstream origin is the bottleneck, not the cache. Fix: raise maxReplicas with a cost review and confirm origin capacity; a cache that is always cold cannot absorb the load a larger fleet would.

FAQ

Why not just scale the tile cache on CPU?

Because a tile cache is I/O- and concurrency-bound, not CPU-bound. It saturates on in-flight request count and tail latency while CPU stays low, so a CPU-targeted HPA reacts far too late. Scaling on requests-per-second or p95 latency ties the replica count to the signal consumers actually feel, which is what the product’s SLA is written against.

How do the stabilization windows stop flapping?

The scale-down stabilization window makes the HPA use the highest recommendation over the trailing window when deciding to shrink, so a brief lull in map traffic does not immediately trigger a scale-in that a returning burst would reverse. Pairing a long (300s) scale-down window with a short (30s) scale-up window gives fast response to load and slow, deliberate release of capacity.

Can this HPA react to the SLO burn-rate alerts directly?

Not directly — an HPA reads a metric value, not an alert state. Instead, drive the HPA from the same underlying series the burn-rate rules watch (request rate, p95), so the cache scales out on the leading indicator before the availability or latency SLO actually burns. The alert then remains the backstop that pages if scaling cannot keep pace.

What min/max replica bounds make sense for a cache tier?

Set minReplicas high enough to serve baseline traffic and survive a node loss with headroom — three is a common floor. Set maxReplicas from a cost-and-capacity review of the origin behind the cache: scaling the cache past the origin’s ability to fill it only shifts the bottleneck. Revisit both bounds whenever the traffic profile or the origin capacity changes.

Why does a cold replica make latency worse before it makes it better?

Because a tile cache’s value is its warm working set, and a new replica has none. When the autoscaler adds a pod, the service mesh begins routing a share of traffic to it immediately, and every one of those requests is a miss that falls through to the renderer — so for the first minutes after a scale-up, aggregate p95 latency can rise even though capacity has increased. This is the single most common reason a latency-driven HPA oscillates: it scales up, latency briefly worsens, and it scales up again. The fixes are to give the scale-up a stabilization window long enough to cover the warm-up, to pre-warm new replicas against the most-requested tile ranges before they receive traffic, and to weight routing toward warm replicas during the warm-up period rather than splitting evenly.

Should the tile cache and the renderer scale on the same signal?

No — they are constrained by different resources and scaling them together wastes one of them. The cache tier is memory-bound and its useful signal is hit ratio and request rate; the renderer is CPU- and IO-bound against PostGIS and its useful signal is queue depth or query latency. Coupling them means a burst of cache misses scales the cache — which does not help, because the misses are the problem — while the renderer, which is the actual bottleneck, stays at its original replica count. Scale each tier on the resource that limits it, and let the cache’s hit ratio be the signal that tells you which tier is under pressure.

Related in SLA Monitoring for Spatial Data Products