# Monitoring Vector Query p95 Latency

Measuring the p95 latency of PostGIS and vector-tile queries requires histogram instrumentation, not a naive average, because spatial query cost is heavy-tailed: a handful of large-bbox or high-vertex requests dominate the experience that consumers actually feel. This operation wires bucketed latency histograms into a Prometheus SLI, computes the 95th percentile with histogram_quantile, and pages when a per-domain p95 breaches its published ceiling. It implements the latency indicator specified in SLA Monitoring for Spatial Data Products within the Spatial Pipeline Orchestration & Observability practice, and sits beside Prometheus Alerting Rules for Tile Availability so a domain covers both latency and availability of its serving layer.

# Prerequisites

# Step-by-Step Implementation

Each step yields a working fragment and a command that proves it before the next.

# 1. Instrument the query path with a histogram

A histogram pre-buckets observations so a percentile can be estimated server-side. Choose le boundaries that bracket the SLA ceiling tightly — clustered buckets around 300ms give an accurate p95 where it matters.

# vector_metrics.py — histogram instrumentation on the spatial query path.
from prometheus_client import Histogram
import time

VECTOR_QUERY = Histogram(
    "vector_query_duration_seconds",
    "PostGIS/vector-tile query latency",
    labelnames=("domain", "layer"),
    # Buckets clustered around the 300ms SLA so the p95 estimate is sharp.
    buckets=(0.005, 0.025, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 1.0, 2.5, 5.0),
)

def run_vector_query(domain: str, layer: str, execute) -> object:
    start = time.perf_counter()
    try:
        return execute()             # runs the PostGIS / MVT query
    finally:
        # Idempotent observation: one sample per query, always recorded.
        VECTOR_QUERY.labels(domain=domain, layer=layer).observe(
            time.perf_counter() - start
        )

Verify: confirm the bucket series is exposed on the metrics endpoint.

curl -s http://localhost:8000/metrics | grep vector_query_duration_seconds_bucket | head

# 2. Record the p95 as an SLI

Compute the 95th percentile from the bucket rates, grouped by domain and preserving the le label the quantile function needs. Recording it keeps the alert expression cheap.

# vector-latency-rules.yaml
groups:
  - name: vector_latency_recording
    interval: 30s
    rules:
      - record: vector:query_latency:p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum(rate(vector_query_duration_seconds_bucket[5m])) by (le, domain)
          )

Verify: parse and validate the rule file.

promtool check rules vector-latency-rules.yaml

# 3. Alert when p95 breaches the ceiling

Page when the recorded p95 exceeds 500ms for a sustained window, keeping the objective (300ms) and the alert ceiling (500ms) distinct so normal variance does not page.

  - name: vector_latency_alerts
    rules:
      - alert: VectorQueryP95Breach
        expr: vector:query_latency:p95_5m > 0.5
        for: 5m
        labels:
          severity: page
          slo: vector_p95
        annotations:
          summary: "p95 latency breach on {{ $labels.domain }} vector queries"
          description: "p95 above 500ms ceiling (300ms objective) for 5m."

Verify: confirm the combined file still validates.

promtool check rules vector-latency-rules.yaml

# 4. Prove the alert with a unit test

Drive a synthetic bucket series whose mass sits above the ceiling and assert the page fires.

# vector-latency-tests.yaml
rule_files:
  - vector-latency-rules.yaml
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'vector_query_duration_seconds_bucket{le="0.5",domain="parcels"}'
        values: '0+2x40'
      - series: 'vector_query_duration_seconds_bucket{le="+Inf",domain="parcels"}'
        values: '0+20x40'     # most mass beyond 500ms → p95 > 0.5
    alert_rule_test:
      - eval_time: 6m
        alertname: VectorQueryP95Breach
        exp_alerts:
          - exp_labels:
              severity: page
              slo: vector_p95
              domain: parcels

Verify: run the behavioural test.

promtool test rules vector-latency-tests.yaml

# 5. Confirm the SLI in the query console

Evaluate the recorded percentile directly so you see the live per-domain p95, not just the alert state.

curl -s 'http://localhost:9090/api/v1/query?query=vector:query_latency:p95_5m' \
  | jq '.data.result[] | {domain: .metric.domain, p95: .value[1]}'

# Configuration Reference

# Common Failure Modes & Fixes

histogram_quantile returns NaN. Root cause: the le label was dropped in the by () aggregation, so the function has no buckets to interpolate. Fix: always include le in the recording rule’s by (le, domain) clause.

p95 reads suspiciously round, like exactly 0.3. Root cause: the estimate is being clamped to a bucket boundary because too few buckets straddle the ceiling. Fix: add le boundaries around 300ms (for example 0.2, 0.3, 0.4) so interpolation has resolution where the SLA lives.

p95 looks healthy but consumers report slowness. Root cause: a heavy-tail of large-bbox queries lives beyond your largest finite bucket, collapsing into +Inf and dragging the true p99 out of view. Fix: extend the top buckets and consider alerting on p99 as well; large-bbox jobs may belong on the async path from Optimizing Async Execution for Spatial Joins.

Alert flaps around the ceiling. Root cause: p95 hovers near 500ms and the for duration is too short. Fix: widen for to 5m and confirm the objective/ceiling gap is wide enough to absorb normal variance.

Per-domain series missing after a deploy. Root cause: the instrumentation was called without the domain/layer labels on one code path. Fix: make the labelled observe mandatory in the query wrapper so no path emits an unlabelled sample.

# FAQ

# Related

• Up to the parent: SLA Monitoring for Spatial Data Products
• Prometheus Alerting Rules for Tile Availability — the availability half of the serving-layer contract
• Distributed Tracing for Spatial Request Flows — the per-request view that explains a breached p95
• Spatial Pipeline Orchestration & Observability — up to the section overview