# Cost Observability for Spatial Workloads

Spatial infrastructure fails on cost long before it fails on capacity, and the reason is almost always that nobody could see which product, consumer, or query was responsible.

A federated estate distributes spend the same way it distributes ownership: twelve domains each running pipelines, stores, renderers and caches, on shared infrastructure, billed to one account. Without attribution, the result is a bill that grows faster than usage and cannot be reasoned about — and the response is usually an estate-wide efficiency drive that penalises the frugal domains alongside the wasteful ones. This topic, inside Spatial Pipeline Orchestration & Observability, specifies how spatial cost is measured, attributed to the product and consumer that caused it, and monitored with the same discipline as latency. It is the economic counterpart to SLA Monitoring for Spatial Data Products: one measures whether a product meets its contract, the other what meeting it costs.

# Architectural Boundaries & Design Rationale

Cost attribution in a spatial estate has to answer three different questions, and a system designed for one answers the others badly.

Which product costs what to produce? This is pipeline cost — compute for reprojection and tiling, storage for artifacts and intermediates, the database capacity a domain’s validation queries consume. It is attributable at the DAG run, because every run already carries the product and version it materialized. This is the number that tells a domain whether a product is worth publishing at its current cadence and resolution.

Which product costs what to serve? This is serving cost — renderer compute, cache capacity, egress. It is attributable per request, because every request already resolves to a product. This is the number that tells a domain whether a port’s access pattern matches its design.

Which consumer causes what cost? This is the question that changes behaviour, and it is the hardest, because it requires serving cost to be attributable not merely to a product but to the caller identity that requested it. That identity exists — the zero-trust layer establishes it on every request — so the requirement is that the cost record carries it through, which is a plumbing decision made early or not at all.

The boundary that matters is between attribution and chargeback. Attribution is a measurement: this product, this consumer, this many currency units. Chargeback is a policy: whose budget it comes out of. Attribution is nearly always worth doing and rarely controversial; chargeback is an organisational decision that frequently is. Building attribution without committing to chargeback is the right sequence, because the measurement changes behaviour on its own — a team that can see it is issuing forty thousand tile requests a night to build a nightly extract will ask for the extract, whether or not anyone bills them for it.

A second boundary: cost is not the same as efficiency. A product that costs a great deal because it serves an enormous amount of valuable traffic is not a problem. The signals worth alerting on are ratios rather than totals — cost per thousand tiles, cost per published partition, cost per unique consumer — because those are the numbers that move when something is genuinely wrong and stay flat when the estate is simply growing.

# Specification & Contract Reference

Every cost record carries the dimensions needed to answer all three questions. Emitting a total without them makes the record unattributable after the fact, and cost data cannot be retroactively enriched.

The derived indicators are what get monitored. Absolute spend is reported; ratios are alerted on.

Two of these are specific to spatial work and rarely appear in generic cost tooling. cache_saving_ratio matters because tile serving is the dominant cost line and caching is the dominant lever on it — a ratio below 0.6 usually means a cache key problem rather than a capacity problem, and no amount of extra renderer capacity will fix it. idle_intermediate_ratio catches the accumulation of reprojected and validated intermediates that pipelines produce, result-persist, and never read again — cheap individually, and a substantial fraction of storage after two years of daily partitions.

# Production Implementation

Cost is emitted as ordinary Prometheus metrics from the points that incur it, carrying the attribution dimensions. Deriving it in a separate billing pipeline is possible but loses the ability to alert on it alongside every other signal.

# cost_metrics.py — attribute spend at the point it is incurred.
from prometheus_client import Counter

# Serving: incremented per request, with the caller identity the zero-trust layer resolved.
SERVE_COST = Counter(
    "spatial_serving_cost_micros_total",
    "Serving cost in millionths of a currency unit",
    labelnames=("domain", "product", "version", "port", "caller", "category"),
)

# Pipeline: incremented per DAG run, with the partition it materialized.
PIPELINE_COST = Counter(
    "spatial_pipeline_cost_micros_total",
    "Pipeline cost in millionths of a currency unit",
    labelnames=("domain", "product", "version", "partition", "category"),
)

# Unit costs come from the infrastructure's published rates, refreshed daily.
UNIT = {"renderer_cpu_ms": 0.42, "egress_byte": 0.00009, "cache_hit": 0.0004}


def record_tile_serve(ctx, cpu_ms: float, bytes_out: int, from_cache: bool) -> None:
    """One tile response. A cache hit costs the cache lookup only — which is the
    whole argument for the cache, made in the same units as everything else."""
    micros = (
        UNIT["cache_hit"] if from_cache
        else cpu_ms * UNIT["renderer_cpu_ms"]
    ) + bytes_out * UNIT["egress_byte"]

    SERVE_COST.labels(
        domain=ctx.domain, product=ctx.product, version=ctx.version,
        port="tiles", caller=ctx.caller_identity,
        category="cache" if from_cache else "render",
    ).inc(micros)

The ratios are recording rules, so alerts evaluate cheaply and dashboards read the same numbers the alerts do.

# cost-rules.yaml — ratios, not totals. Totals grow with the estate; ratios do not.
groups:
  - name: spatial_cost_recording
    interval: 60s
    rules:
      - record: spatial:cost_per_1k_tiles:30m
        expr: |
          1000 * sum(rate(spatial_serving_cost_micros_total{port="tiles"}[30m])) by (domain, product)
              / sum(rate(spatial_tiles_served_total[30m])) by (domain, product)

      - record: spatial:cache_saving_ratio:30m
        expr: |
          sum(rate(spatial_serving_cost_micros_total{category="cache"}[30m])) by (domain, product)
          / sum(rate(spatial_serving_cost_micros_total[30m])) by (domain, product)

      # A single consumer's share of one product's serving cost.
      - record: spatial:consumer_cost_share:1h
        expr: |
          sum(rate(spatial_serving_cost_micros_total[1h])) by (domain, product, caller)
          / ignoring(caller) group_left
            sum(rate(spatial_serving_cost_micros_total[1h])) by (domain, product)

  - name: spatial_cost_alerts
    rules:
      - alert: SpatialCostPerTileRegression
        expr: |
          spatial:cost_per_1k_tiles:30m
            > 2 * quantile_over_time(0.5, spatial:cost_per_1k_tiles:30m[30d])
        for: 30m
        labels: { severity: ticket }
        annotations:
          summary: "cost per 1k tiles doubled on {{ $labels.domain }}/{{ $labels.product }}"
          description: "Check cache hit ratio and recent version changes before capacity."

      - alert: SpatialConsumerDominatesCost
        expr: spatial:consumer_cost_share:1h > 0.4
        for: 2h
        labels: { severity: ticket }
        annotations:
          summary: "{{ $labels.caller }} is {{ $value | humanizePercentage }} of {{ $labels.product }} cost"
          description: "Likely an access-pattern mismatch — check whether they need a bulk port."

Confirm attribution is complete, which is the check that matters most because an unattributed record is unrecoverable:

# Every currency unit must carry a domain, product and category. Any series missing
# one is a plumbing gap, and no amount of later analysis will fill it in.
curl -s "$PROM/api/v1/query" --data-urlencode \
  'query=sum(spatial_serving_cost_micros_total) - sum(sum(spatial_serving_cost_micros_total) by (domain, product, category))' \
  | jq -r '.data.result[0].value[1] // "0"'
# expected: 0

# Diagnostic Runbook

1. Start from the ratio that alerted, not from the bill. A cost alert names a domain, product and indicator. Opening the infrastructure bill instead means starting from a total that mixes twelve domains and cannot be decomposed after the fact.
2. For a cost_per_1k_tiles regression, check the cache saving ratio first. A rise in cost per tile with a fall in cache saving is a cache problem — usually a key change or a Vary header multiplying the cache — and adding renderer capacity makes it more expensive rather than less.
3. If cost rose at a version boundary, compare the two versions’ resolution and zoom range. A resolution increase or an extra zoom level multiplies both pipeline and serving cost, often by more than the team expected, and it is visible immediately in the version label.
4. When one consumer dominates, look at their access shape before their volume. A caller issuing sequential requests for adjacent tiles is building a bulk extract through a tile port; the fix is to give them the right port, not to throttle them.
5. For rising storage with flat reads, audit intermediates. Reprojected and validated intermediates from result persistence accumulate silently. Query which artifacts have not been read in 90 days before considering a storage tier change.
6. A high egress share usually means a cross-region path, not a chatty consumer. Check whether the serving edge and the origin are in the same region for the affected product; residency constraints sometimes force a split that nobody costed.
7. If attribution is incomplete, fix the emitter before analysing anything. Cost records cannot be enriched retroactively, so every hour spent analysing partly-attributed data is an hour producing conclusions that a complete dataset would contradict.

# SLA Targets & Performance Baselines

# The Four Levers, in the Order They Are Worth Pulling

Cost work goes wrong when it starts with the largest line on the bill rather than the largest addressable one. In a spatial estate the levers have a fairly consistent ordering by effect per unit of effort, and following it avoids a great deal of wasted engineering.

Caching comes first, because tile serving dominates and a cache hit is roughly two orders of magnitude cheaper than a render. A layer at a 70% hit ratio moved to 92% cuts renderer load by nearly three quarters, and the work is usually configuration — a cache key with unnecessary cardinality, a Vary header multiplying the cache, a missing stale-while-revalidate causing synchronised expiries. No other lever comes close for effort spent.

Precompute scope comes second. Rendering a full tile pyramid to maximum zoom across a national extent produces enormous numbers of tiles that are never requested. Measuring the observed zoom distribution and precomputing only where demand exists, rendering the rest on first request, cuts pipeline compute and storage together, with no consumer-visible change.

Storage tiering comes third, and is where most teams start because storage is easy to see on a bill. It is genuinely worth doing — cold raster archives and never-read pipeline intermediates accumulate steadily — but the saving is linear and modest compared with the first two, and it carries a latency risk that has to be checked against the product’s SLO before anything is moved.

Compute right-sizing comes last, not because it does not matter but because it is the most effort for the least durable gain. Worker classes, instance types and concurrency limits all drift back as workloads change, so the saving needs continuous attention, and a mis-sized reprojection worker that runs out of memory costs far more in failed runs than the instance saved.

One anti-lever deserves naming: reducing published resolution or cadence to save money. It is the fastest way to cut cost and the fastest way to make a product worse for every consumer, and it is almost always reached for before the four above have been exhausted. If it is genuinely the right call, it is a contract change with a version bump and a consumer conversation — not a cost optimisation applied quietly.

A final practical note: cost data is only actionable while it is current. A monthly report tells a domain what they spent five weeks ago on a version they have since replaced, which is history rather than feedback. Emitting cost as metrics on the same scrape interval as everything else — and putting the ratios on the same dashboards as latency and freshness — is what makes cost a property a team notices while they can still do something about it, rather than a number that arrives after the decisions that caused it.

One organisational pattern is worth adopting alongside the metrics: a standing, low-ceremony review where each domain looks at its own ratios once a month, in the same meeting where it reviews its SLOs. Cost treated as a separate exercise, owned by a central team, produces recommendations domains did not ask for and rarely act on. Cost treated as one more property of a product — reviewed by the team that owns it, against the same medians every other domain is measured by — produces the small continuous adjustments that keep an estate affordable without anyone running a programme. The platform team’s contribution to that review is the comparison baseline and the outlier list, not the remediation plan.

# Governance & Compliance Notes

Cost attribution has one property that makes it politically delicate and worth handling deliberately: it makes every domain’s spend visible to every other. That transparency is the point — it is what lets a platform team point at a genuine outlier rather than run an estate-wide efficiency programme — but it works only if the comparison is fair. Comparing absolute spend across domains with wildly different consumer volumes produces resentment and no insight; comparing ratios produces a conversation. Publishing ratios by default and totals on request is the arrangement that survives contact with an organisation.

Attribution data is also access-pattern data, and access patterns are sensitive in their own right. A record showing which consumer queried which product over which extent, at what volume, describes what that consumer is working on. Where domains are separated for confidentiality rather than merely for ownership, cost records need the same access controls as the access logs they derive from — a domain should see its own products’ cost by consumer, and the platform team should see the estate, but one domain browsing another’s consumer breakdown is an information leak dressed as a dashboard.

Finally, unit costs change. Infrastructure rates are renegotiated, regions are repriced, and a ratio computed against last year’s rates is not comparable with one computed today. Recording the rate card version alongside the cost record is what keeps a 30-day median meaningful across a pricing change, and its absence is why cost time series so often show a step change that everyone eventually learns to ignore.

# Related

• SLA Monitoring for Spatial Data Products — the reliability counterpart to these indicators
• Edge Caching and Tile Delivery Topology — the dominant lever on serving cost
• Product Thinking for GIS Datasets — why the right output port is a cost decision
• Orchestrating Spatial Pipelines in Python — where pipeline cost is incurred and attributed
• Spatial Pipeline Orchestration & Observability — up to the section overview