# Prometheus Alerting Rules for Tile Availability

This procedure builds the multi-window, multi-burn-rate Prometheus rules that turn the raw 200-response rate of an OGC API Tiles or MVT endpoint into a paging-grade availability SLO, with recording rules that keep the alert expressions cheap and a promtool gate that proves they fire before they ship. It is the concrete rule set behind the availability objective defined in SLA Monitoring for Spatial Data Products, under the broader Spatial Pipeline Orchestration & Observability practice, and it pairs naturally with Monitoring Vector Query p95 Latency so a domain covers both the availability and the latency halves of its tile contract. Get the burn-rate windows right and a genuine outage pages within minutes while a deploy blip stays silent.

# Prerequisites

Figure — Two burn-rate windows must both breach before a page fires; a slow single-window breach opens a ticket instead.

# Step-by-Step Implementation

Each step below produces a fragment of the rule file and a command that proves the fragment is correct before you build on it.

# 1. Record the availability ratio at two windows

The alert expressions stay cheap and readable only if the SLI ratio is pre-computed. Record the 200-rate over both the fast and the confirming window, grouped by domain and layer so a single hot layer can be isolated.

# tile-availability-rules.yaml
groups:
  - name: tile_availability_recording
    interval: 30s
    rules:
      - record: tile:availability:ratio_rate5m
        expr: |
          sum(rate(tile_requests_total{code="200"}[5m])) by (domain, layer)
          /
          sum(rate(tile_requests_total[5m])) by (domain, layer)
      - record: tile:availability:ratio_rate1h
        expr: |
          sum(rate(tile_requests_total{code="200"}[1h])) by (domain, layer)
          /
          sum(rate(tile_requests_total[1h])) by (domain, layer)

Verify: confirm the rule file parses and the recording rules are well-formed.

promtool check rules tile-availability-rules.yaml

# 2. Add the fast-burn page rule

The page fires only when the 5-minute and 1-hour windows both breach 14.4 times the 0.005 budget, which suppresses single-scrape spikes while still catching a fast burn within minutes.

  - name: tile_availability_alerts
    rules:
      - alert: TileAvailabilityFastBurn
        expr: |
          (1 - tile:availability:ratio_rate5m) > (14.4 * 0.005)
          and
          (1 - tile:availability:ratio_rate1h) > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
          slo: tile_availability
        annotations:
          summary: "Fast burn on {{ $labels.domain }}/{{ $labels.layer }} tiles"
          description: "Tile 200-rate breaching 99.5% budget at >14.4x."

Verify: re-check the combined file after adding the alert group.

promtool check rules tile-availability-rules.yaml

# 3. Add the slow-burn ticket rule

A slow leak that would still exhaust the budget over the window should open a ticket, not wake anyone. Use a wider window at a burn rate of 1.

      - alert: TileAvailabilitySlowBurn
        expr: |
          (1 - avg_over_time(tile:availability:ratio_rate1h[6h])) > (1 * 0.005)
        for: 15m
        labels:
          severity: ticket
          slo: tile_availability
        annotations:
          summary: "Slow burn on {{ $labels.domain }}/{{ $labels.layer }} tiles"

Verify: confirm both alerts are registered in the group.

promtool check rules tile-availability-rules.yaml | grep -E 'SUCCESS|FAILED'

# 4. Prove the rules fire with a unit test

A rule that parses but never evaluates true is worse than no rule. Drive a synthetic series through promtool test rules to assert the page fires on a crafted outage and the slow burn tickets on a steady leak.

# tile-availability-tests.yaml
rule_files:
  - tile-availability-rules.yaml
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'tile_requests_total{code="200",domain="basemap",layer="roads"}'
        values: '0+5x120'          # steady successes then...
      - series: 'tile_requests_total{code="503",domain="basemap",layer="roads"}'
        values: '0+95x120'         # ...a heavy 503 flood
    alert_rule_test:
      - eval_time: 10m
        alertname: TileAvailabilityFastBurn
        exp_alerts:
          - exp_labels:
              severity: page
              slo: tile_availability
              domain: basemap
              layer: roads

Verify: run the behavioural test; a non-zero exit fails CI.

promtool test rules tile-availability-tests.yaml

# 5. Apply and confirm the rules are live

Load the file into Prometheus and confirm the alert is registered and evaluating, not just present on disk.

# after reload (SIGHUP or /-/reload), confirm the alert is loaded
curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name=="TileAvailabilityFastBurn") | {name, state}'

# Configuration Reference

# Common Failure Modes & Fixes

The page never fires during a real outage. Root cause: only the fast window is breaching because the outage is shorter than the 1-hour confirmation leg, or the by (domain, layer) grouping in the alert does not match the recorded series labels. Fix: confirm the recording rule and alert share identical grouping, and verify with promtool test rules using a fixture that spans both windows.

The page fires on every deploy. Root cause: the alert is watching only the 5-minute window without the 1-hour and leg, so a brief restart blip trips it. Fix: restore the two-window and condition and keep for: 2m so a single scrape cannot page.

No series returned from the recording rule. Root cause: the exporter emits status rather than code, or omits the domain label the rule groups by. Fix: align the selector to the exporter’s actual label set — promtool check metrics against a scrape sample reveals the true names.

Alert fires but never routes to on-call. Root cause: the severity: page label is present but Alertmanager’s route tree matches on a different key. Fix: confirm the receiver’s match block keys on severity and domain, and reload Alertmanager after the change.

Cardinality explosion stalls rule evaluation. Root cause: a raw z/x/y tile-coordinate label multiplies series into the millions. Fix: drop the per-tile label in the recording rule’s by () clause and keep tile identity in a separate, sampled diagnostic metric.

# FAQ

# Related

• Up to the parent: SLA Monitoring for Spatial Data Products
• Monitoring Vector Query p95 Latency — the latency half of the tile contract
• HPA Configuration for Spatial Tile Cache Nodes — scaling the cache when availability burns
• Spatial Pipeline Orchestration & Observability — up to the section overview