Measuring Cost per Thousand Tiles in Prometheus

Total spend grows when an estate grows, which makes it a poor alert signal and an excellent way to have the wrong conversation. Cost per thousand tiles does not: it stays flat while traffic doubles and moves sharply when something is actually wrong — a cache key regression, a costlier new version, a consumer scraping a tile port. This guide builds that ratio as a recording rule, alerts on its movement rather than its value, and shows how to read it when it fires. It implements the headline indicator from Cost Observability for Spatial Workloads within Spatial Pipeline Orchestration & Observability, and it consumes the labels wired up in Attributing Tile Serving Cost to Consumers.

Prerequisites

Requirement Value / Assumption Notes
Tools Prometheus ≥ 2.45, promtool, Alertmanager Rules are unit-tested before they ship
Metrics spatial_serving_cost_micros_total, spatial_tiles_served_total Both labelled by domain, product, category
History ≥ 30 days The baseline is a 30-day median
Rate card Version label on the cost metric A ratio across a repricing is not comparable
Access roles prometheus-rules-editor Rules are governed
Environment PROM, PRODUCT Exported before running

Step-by-Step Implementation

1. Record the ratio, not the totals

yaml
# cost-per-tile-rules.yaml
groups:
  - name: spatial_cost_recording
    interval: 60s
    rules:
      # Micros per tile × 1000 = micros per thousand tiles. Recorded so the alert
      # expression stays cheap and the dashboard reads exactly what the alert does.
      - record: spatial:cost_per_1k_tiles:30m
        expr: |
          1000 *
            sum(rate(spatial_serving_cost_micros_total{port="tiles"}[30m])) by (domain, product)
          / sum(rate(spatial_tiles_served_total{port="tiles"}[30m])) by (domain, product)

      # The same ratio split by category, which is where a regression is diagnosed.
      - record: spatial:cost_per_1k_tiles_by_category:30m
        expr: |
          1000 *
            sum(rate(spatial_serving_cost_micros_total{port="tiles"}[30m])) by (domain, product, category)
          / sum(rate(spatial_tiles_served_total{port="tiles"}[30m])) by (domain, product)

      # Cache saving ratio — the dominant driver, recorded alongside so a responder
      # never has to correlate two dashboards during an incident.
      - record: spatial:cache_hit_ratio:30m
        expr: |
          sum(rate(spatial_tiles_served_total{category="cache"}[30m])) by (domain, product)
          / sum(rate(spatial_tiles_served_total[30m])) by (domain, product)

Why the ratio alerts and the total does not: four months of one productTwo series over four months. Total monthly spend rises steadily from 100 to 190 units as the product grows, which is indistinguishable from a regression. Cost per thousand tiles stays flat near four units through months one to three and then doubles to eight in month four, when a cache key change fragmented the cache. Only the ratio separates growth from regression, which is why the alert is written against it and the total is reported rather than alerted on.total, month 1100total, month 4190growth, not regressionratio, months 1–34flat through growthratio, month 48cache key fragmented

Verify the rules evaluate and produce a plausible figure:

bash
promtool check rules cost-per-tile-rules.yaml
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=spatial:cost_per_1k_tiles:30m{product="'"$PRODUCT"'"}' \
  | jq -r '.data.result[] | "\(.metric.domain)/\(.metric.product): \(.value[1]) micros/1k"'

2. Alert on movement against the trailing median

A fixed threshold is wrong for every product except the one it was tuned on. The median over 30 days is per-product and self-maintaining.

yaml
  - name: spatial_cost_alerts
    rules:
      - alert: TileCostPerThousandRegression
        expr: |
          spatial:cost_per_1k_tiles:30m
            > 2 * quantile_over_time(0.5, spatial:cost_per_1k_tiles:30m[30d])
        for: 30m
        labels: { severity: ticket }
        annotations:
          summary: "cost/1k tiles doubled on {{ $labels.domain }}/{{ $labels.product }}"
          description: >-
            Check spatial:cache_hit_ratio:30m first — a fall there explains most
            regressions. If the ratio is unchanged, compare the version label.

      # The leading indicator: cache ratio falls before cost per tile rises.
      - alert: TileCacheRatioDegraded
        expr: spatial:cache_hit_ratio:30m < 0.6
        for: 20m
        labels: { severity: ticket }
        annotations:
          summary: "cache hit ratio {{ $value | humanizePercentage }} on {{ $labels.product }}"
          description: "Check cache key cardinality and the Vary header before capacity."

Verify the rules fire on synthetic data rather than waiting for a real regression:

yaml
# cost-rules-tests.yaml
rule_files: [cost-per-tile-rules.yaml]
evaluation_interval: 60s
tests:
  - interval: 60s
    input_series:
      # Steady traffic, cost tripling partway through the window.
      - series: 'spatial_tiles_served_total{port="tiles",domain="d",product="p",category="cache"}'
        values: '0+600x120'
      - series: 'spatial_serving_cost_micros_total{port="tiles",domain="d",product="p",category="render"}'
        values: '0+40x60 2400+120x60'
    alert_rule_test:
      - eval_time: 110m
        alertname: TileCostPerThousandRegression
        exp_alerts:
          - exp_labels: { severity: ticket, domain: d, product: p }
bash
promtool test rules cost-rules-tests.yaml

3. Read the ratio when it fires

The diagnosis order is fixed, because the causes have very different fixes and only one of them is a capacity problem.

The fixed diagnosis order when the ratio alerts, and why capacity is lastA cost-per-tile alert is diagnosed in a fixed order. The cache hit ratio is checked first because a fall there explains most regressions and is never fixed by adding capacity. The version label is checked second, since a cost step at a version boundary is usually a resolution or zoom-range change. Consumer concentration is checked third, because one caller access pattern can move a whole product ratio. Only when all three are unchanged is this a capacity or pricing question. Each of the first three drops onto a rail naming its actual fix.1 · Cache ratiofell?2 · Versionstep at a boundary?3 · Consumersone caller dominating?4 · Capacityonly if all unchangedunchangedunchangedunchangedfix the cache keyexpected — median absorbs itoffer the right portResolved before capacitythree of four times

bash
# 1. Cache ratio. A fall here explains most regressions and is never fixed by capacity.
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=spatial:cache_hit_ratio:30m{product="'"$PRODUCT"'"}' | jq -r '.data.result[0].value[1]'

# 2. Version. A cost step at a version boundary is a resolution or zoom-range change.
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=sum by (version) (rate(spatial_serving_cost_micros_total{product="'"$PRODUCT"'"}[30m]))' \
  | jq -r '.data.result[] | "\(.metric.version): \(.value[1])"'

# 3. Consumer concentration. One caller's access pattern can move a product's ratio.
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=topk(3, spatial:consumer_cost_share:1h{product="'"$PRODUCT"'"})' \
  | jq -r '.data.result[] | "\(.metric.caller): \(.value[1])"'

# 4. Only if all three are unchanged is this a capacity or pricing question.

4. Put the ratio where the team already looks

bash
# The ratio belongs on the product's own dashboard, beside latency and freshness —
# not on a separate cost dashboard nobody opens between incidents.
curl -sS -X POST "$GRAFANA/api/dashboards/db" -H 'content-type: application/json' -d '{
  "dashboard": {
    "title": "Product health",
    "panels": [
      {"title": "p95 latency",        "targets": [{"expr": "vector:query_latency:p95_5m"}]},
      {"title": "freshness lag",      "targets": [{"expr": "spatial:freshness_lag_seconds"}]},
      {"title": "cost / 1k tiles",    "targets": [{"expr": "spatial:cost_per_1k_tiles:30m"}]},
      {"title": "cache hit ratio",    "targets": [{"expr": "spatial:cache_hit_ratio:30m"}]}
    ]
  }, "overwrite": true}'

Configuration Reference

Element Value Rationale
Rate window 30m Long enough to smooth traffic bursts, short enough to be current
Baseline quantile_over_time(0.5, …[30d]) Median resists a single bad day
Alert factor 2× the median Below this is normal variation for most products
for 30m Cost regressions are sustained, not spiky
Severity ticket, never page Nothing is down; nobody should wake up
Cache-ratio floor 0.6 The leading indicator, alerted separately
Units micros per 1k tiles Integer-friendly; avoids float noise in the metric

Severity deserves the note. A cost regression is a ticket by definition: the product is serving correctly, consumers are unaffected, and the fix is a considered change rather than an urgent one. Paging on cost trains people to dismiss pages.

Why a cost regression is a ticket and never a pageFour properties compared between a cost regression and an availability breach. A cost regression leaves the product serving correctly, affects no consumer, is fixed by a considered change, and can wait until morning. An availability breach stops the product serving, affects every consumer, needs an immediate rollback, and cannot. The final row records the consequence of getting the severity wrong: paging on cost trains responders to dismiss pages, which degrades the response to the breaches that matter.Cost regressionAvailability breachProduct still servingyesnoConsumers affectednoneallFixconsidered changeimmediate rollbackSeverityticketpageIf paged anywaypages get dismissed—

Common Failure Modes & Fixes

The ratio is NaN for a product. Root cause: division by zero — no tiles served in the window, usually because the product is idle or the port label does not match. Fix: guard the recording rule with a > 0 filter on the denominator, or accept NaN for idle products and exclude them from alerting.

The ratio steps up on a single day across every product. Root cause: a rate-card change, not a behaviour change. Fix: the rate_card label distinguishes them; alerting expressions can exclude a window around a known repricing rather than being retuned.

Alert fires and the cache ratio is unchanged. Root cause: usually a new version with a wider zoom range or finer resolution, which costs more per tile legitimately. Fix: confirm from the version label; if intended, the median absorbs it within a few weeks and the alert self-resolves.

Ratio looks healthy and the bill grows. Root cause: correct behaviour — the ratio is flat and volume is up, which is growth rather than regression. Fix: this is what the ratio is for. If the growth itself is the concern, that is a capacity-planning conversation, not a cost-regression one.

A single consumer moves the whole product’s ratio. Root cause: a caller with a very low cache hit ratio scraping tiles sequentially. Fix: the per-consumer breakdown identifies them; the remedy is offering the correct port rather than throttling.

FAQ

Why per thousand tiles rather than per request or per byte?

Because a tile is the unit the product actually publishes, so the ratio is comparable across products in a way per-request or per-byte figures are not. A request may fetch one tile or a batch; bytes vary by feature density and encoding, so a dense urban tile and a sparse rural one differ by an order of magnitude without anything being wrong. Tiles served is the denominator a domain team already understands, and multiplying by a thousand keeps the value in a range where integer micros avoid floating-point noise in the metric itself.

Should the ratio include pipeline cost as well as serving cost?

No — keep them separate, because they answer different questions and move for different reasons. Serving cost per tile responds to caching, access patterns and traffic mix; pipeline cost per partition responds to resolution, zoom range and cadence. Combining them produces a number that moves for six reasons at once and diagnoses none of them. Where a single figure is genuinely wanted for a budget conversation, sum the two on the dashboard rather than in the ratio.

How long should the baseline window be?

Thirty days, which is long enough to span a full traffic cycle including monthly patterns and short enough to follow deliberate changes without a manual reset. A shorter window makes the baseline chase a regression upward, so a slow degradation never triggers; a longer one makes an intended change — a new zoom range, a deliberate resolution increase — alert for months after everyone agreed to it. The median rather than the mean is what makes thirty days workable, since one anomalous day cannot drag it.

Does this replace looking at the actual bill?

No, and the two should disagree occasionally. The bill is authoritative for what was spent and useless for saying why; the ratio is diagnostic and approximate, because unit costs are modelled rather than billed. A persistent gap between the two means the rate card has drifted from reality and should be refreshed. Treating the ratio as the operational signal and the bill as the monthly reconciliation gives each the job it is good at.

Related in Cost Observability for Spatial Workloads