# SLA Monitoring for Spatial Data Products

Turning tile availability, vector query latency, and dataset freshness into measured, alertable service-level objectives with explicit error budgets that hold every domain accountable for its published contract.

A spatial data product is only as trustworthy as the evidence that it is meeting its promises, and in a federated mesh those promises are made per domain rather than centrally. This page sits within Spatial Pipeline Orchestration & Observability and treats monitoring not as a dashboard afterthought but as the enforcement surface for the SLAs a domain commits to when it publishes a product. Where Orchestrating Spatial Pipelines in Python guarantees that a dataset is produced idempotently, and Distributed Tracing for Spatial Request Flows explains why a single request was slow, this concern answers a different question: is the product, in aggregate and over its measurement window, honouring the availability, latency, and freshness targets its consumers depend on? The answer must be a number with an alert attached, not a subjective judgement.

Figure — Tile and vector services emit metrics through exporters into Prometheus, which computes SLO burn rate and pages on-call while an error-budget gauge tracks remaining tolerance.

# Architectural Boundaries & Design Rationale

SLA monitoring for a data mesh is a domain-owned concern, not a platform-owned one. The platform provides the metric pipeline — exporters, a scrape target convention, a Prometheus estate, and Alertmanager routing — but each domain writes the recording and alerting rules that encode its product’s promises, because only the owning team knows what tile z-levels are hot, which vector layers carry a sub-second contract, and how stale a freshness feed is allowed to become before consumers are misled. Centralising those rules would recreate the monolithic bottleneck that Spatial Domain Boundary Design exists to dissolve. The monitoring layer therefore inherits the same boundary: a domain exposes a stable metric contract at its edge, and the SLO rules that consume it live in the domain’s own repository under the same review gate as its data contracts.

The design rests on the SLI–SLO–error-budget triad applied to spatial signals. A service-level indicator is a ratio computed from raw telemetry — the fraction of tile requests that returned 200 within the freshness window, or the fraction of vector queries under the latency ceiling. A service-level objective is the target that indicator must hold over a rolling window, for example 99.5% tile availability over 30 days. The error budget is the arithmetic complement: 0.5% of requests may fail before the objective is breached. Expressing the objective as a budget is what makes the signal actionable, because it converts an abstract percentage into a depletable quantity that a burn-rate alert can watch draining in real time.

Three spatial-specific failure modes justify treating this as first-class architecture rather than generic uptime monitoring. First, freshness decoupled from availability: a tile endpoint can return 200 for every request while serving hours-old data because an upstream reprojection job stalled, so availability alone is a lie and freshness must be a separate indicator. Second, hot-key skew: tile traffic follows a Zipfian distribution across z/x/y, so a global p95 hides a specific metropolitan tile pyramid that is timing out — labels must preserve enough dimensionality to isolate it without exploding cardinality. Third, cross-domain budget contamination: when one domain’s product is composed from another’s, a naive alert pages the wrong team. Every indicator therefore carries a domain label, and burn-rate alerts route on it so the page reaches the team that owns the breaching product. Corrected thresholds and objective changes propagate through the same review path as the rules themselves, keeping the on-call contract auditable rather than mutated by hand under incident pressure.

# Specification & Contract Reference

Every SLO a domain publishes pins the same surface: the indicator expression, the objective and its window, the burn-rate windows that arm paging versus ticketing, and the labels that scope the alert to a product. The table below is the minimum a production SLO definition must declare before it is allowed to route a page.

The burn-rate figures are not arbitrary. A burn rate of 1 means the budget is being consumed exactly at the pace that would exhaust it precisely at the end of the window; a burn rate of 14.4 over a 30-day window would exhaust the entire budget in roughly two days, which is why it warrants an immediate page. Pairing a fast 5-minute window with a 1-hour confirmation window suppresses the single-scrape spikes that a deploy produces while still catching a genuine fast burn within minutes. These conventions are the same ones the child pages apply concretely — the availability rules in Prometheus Alerting Rules for Tile Availability, the histogram-based latency signal in Monitoring Vector Query p95 Latency, and the autoscaling response wired in HPA Configuration for Spatial Tile Cache Nodes.

# Production Implementation

The source of truth for a product’s SLOs is a Prometheus rule file, versioned in the domain’s repository and validated in CI with promtool before it can be applied. The recording rules pre-compute the SLI ratios so that alert expressions stay cheap and readable, and the alerting rules implement the multi-window multi-burn-rate pattern. Idempotency is inherent: applying the same rule file twice is a no-op, and because the rules are pure functions of scraped series they never mutate state. Zero-trust applies to the metric pipeline itself — exporters are scraped over mTLS and every rule file is signed and admitted only from the owning domain’s namespace.

# spatial-slo-rules.yaml — recording + multi-window burn-rate alerting.
# Validate: promtool check rules spatial-slo-rules.yaml
groups:
  - name: spatial_sli_recording
    interval: 30s
    rules:
      # Tile availability: fraction of tile requests that returned 200.
      - record: tile:availability:ratio_rate5m
        expr: |
          sum(rate(tile_requests_total{code="200"}[5m])) by (domain)
          /
          sum(rate(tile_requests_total[5m])) by (domain)
      # Same ratio over a 1h window for the confirmation leg of the alert.
      - record: tile:availability:ratio_rate1h
        expr: |
          sum(rate(tile_requests_total{code="200"}[1h])) by (domain)
          /
          sum(rate(tile_requests_total[1h])) by (domain)

  - name: spatial_slo_alerts
    rules:
      # Fast burn: budget for a 0.995 SLO drains 14.4x too fast over 5m AND 1h.
      - alert: TileAvailabilityFastBurn
        expr: |
          (1 - tile:availability:ratio_rate5m) > (14.4 * 0.005)
          and
          (1 - tile:availability:ratio_rate1h) > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
          slo: tile_availability
        annotations:
          summary: "Fast error-budget burn on {{ $labels.domain }} tiles"
          description: "Availability SLI breaching 99.5% budget at >14.4x burn."
      # Slow burn: opens a ticket, does not page.
      - alert: TileAvailabilitySlowBurn
        expr: (1 - tile:availability:ratio_rate1h) > (1 * 0.005)
        for: 15m
        labels:
          severity: ticket
          slo: tile_availability
        annotations:
          summary: "Slow error-budget burn on {{ $labels.domain }} tiles"

CI gates the file on both static validity and behavioural correctness. promtool check rules parses every expression and rejects a malformed PromQL or a label typo before it can reach the Prometheus estate, and promtool test rules replays a synthetic time series through the group to assert that a crafted outage actually fires TileAvailabilityFastBurn and a slow steady leak fires only the ticket. Gating on the behavioural test is what stops a silently-broken alert — an expression that parses but never evaluates true — from shipping.

# CI step: reject the rule file unless it is both valid and behaviourally correct.
promtool check rules spatial-slo-rules.yaml
promtool test rules spatial-slo-tests.yaml   # asserts fast-burn fires, slow-burn tickets

Alertmanager routes on the severity and domain labels so a page reaches the owning team’s on-call rotation while a ticket is filed without waking anyone. Because the routing key is the domain label the indicator already carries, a product composed across domains still pages the team that owns the breaching leg rather than the aggregator.

# Diagnostic Runbook

When an SLO fires — or worse, fails to fire during a known incident — the fault is almost always in indicator definition, label cardinality, rule evaluation, or budget arithmetic rather than the raw exporter. Work the steps in order; each isolates one layer of the pipeline.

1. Confirm the indicator is populated. Query tile:availability:ratio_rate5m directly. An empty result means the recording rule never matched — usually a label selector drift between the exporter’s emitted labels and the rule’s by (domain) grouping.
2. Separate availability from freshness. If availability reads healthy during a consumer complaint, evaluate the freshness indicator independently. A stalled upstream job returns 200 on stale tiles, so a green availability SLI with a red freshness SLI localises the fault to production, not serving.
3. Check burn-rate window alignment. Verify the short and long windows are both breaching before expecting a page. A fast window alone that clears within a scrape interval is deploy noise the confirmation window is designed to suppress.
4. Audit label cardinality. Inspect the series count for the underlying counter. A hot-key label such as raw z/x/y can explode cardinality and stall rule evaluation; aggregate to a bounded label set and keep the tile pyramid identity in a separate low-cardinality dimension.
5. Reconcile the error-budget arithmetic. Confirm the objective in the alert threshold (14.4 * budget) matches the published slo.objective. A stale objective left after a target change is the classic cause of an alert that pages too eagerly or never at all.
6. Validate rule provenance. Confirm the deployed rule file’s revision matches the Git SHA in source control; a partial apply can leave one domain running last quarter’s objective.
7. Replay against a synthetic outage. If the cause is still unclear, run promtool test rules with a fixture that reproduces the reported symptom and confirm the group fires as expected, then trace the divergence between fixture and production series.

# SLA Targets & Performance Baselines

The monitoring layer itself carries an SLA: an alert that arrives late is as useless as no alert. The budget below is what the observability domain must hold so downstream product SLAs stay meaningful.

Holding these baselines depends on keeping rule groups small and label-bounded, evaluating latency indicators on a tighter interval than availability, and treating the burn-rate windows as event-driven thresholds rather than fixed cron so a fast burn is caught within minutes of onset.

# Governance & Compliance Notes

SLO definitions are governance artifacts because they encode a public promise. Objectives, windows, and burn-rate thresholds are committed to source control, reviewed, and promoted only after CI validates the rules with promtool check rules and asserts their behaviour with promtool test rules. This keeps every alertable objective mapped to a catalog record described in Metadata Cataloging for Raster/Vector, so a consumer reading a product’s advertised SLA sees the same objective the on-call rotation is paged against.

Audit requirements are concrete. Every error-budget consumption event and every objective change is written to an append-only, domain-partitioned record, so a reviewer can reconstruct exactly how much budget a product had spent when a decision — a freeze, a rollback, a capacity request — was taken. Where a product carries data-residency constraints, its metrics inherit them: exporter series for a region-bound tenant are scraped and retained in-region, and burn-rate alerts route to the in-region rotation rather than crossing a residency boundary. Regularly exercise the alerts in game-day drills that inject synthetic outages, so failover of the paging path, correctness of the burn-rate arithmetic, and the freshness-versus-availability separation are all validated before a real incident depends on them.

# Related

• Up to the parent: Spatial Pipeline Orchestration & Observability
• Prometheus Alerting Rules for Tile Availability — the concrete multi-window burn-rate rules for tile 200-rate
• Monitoring Vector Query p95 Latency — histogram instrumentation of PostGIS query latency
• HPA Configuration for Spatial Tile Cache Nodes — autoscaling the tile cache on a custom Prometheus metric
• Distributed Tracing for Spatial Request Flows — the per-request view that explains a breached latency SLO