# Distributed Tracing for Spatial Request Flows

Reconstructing the full journey of a single tile or feature request as it crosses ingress, routing, serving backends, and PostGIS — with the spatial parameters of the request attached to every span — so a slow or wrong answer can be diagnosed at the hop that caused it rather than guessed at from aggregate metrics.

In a federated spatial mesh a single map interaction fans out across services that no one team owns end to end, and when one turns slow the aggregate dashboards described in SLA Monitoring for Spatial Data Products tell you that a latency SLO is burning but never where. This page sits within Spatial Pipeline Orchestration & Observability and establishes distributed tracing as the per-request counterpart to those metrics: an OpenTelemetry trace that stitches ingress, cross-domain routing, tile and vector backends, and the PostGIS query into one causally-ordered waterfall, with spatial attributes — bounding box, CRS, tile z/x/y — recorded on the spans so a responder can see that the cost lived in a large-bbox scan against an unindexed geometry column rather than in the network. Tracing is what converts a burning error budget into a specific, fixable span.

Figure — A trace waterfall for one tile request: nested spans across ingress, routing, tile backend, and PostGIS, each carrying spatial attributes.

# Architectural Boundaries & Design Rationale

Tracing is a cross-cutting concern that must respect the same domain boundaries as everything else in the mesh. The platform owns the trace backbone — the context-propagation convention, the collector, and the sampling policy — while each domain owns the spans it emits and the spatial attributes it attaches, because only the domain team knows which of its operations are worth a dedicated span and which attributes make a trace diagnosable. This mirrors the ownership model of Spatial Domain Boundary Design: a domain exposes a stable tracing contract at its edge — it will continue a trace it receives and propagate context to its callees — but it is free to instrument its internals as it sees fit.

A trace is built from spans arranged in a causal tree. The root span is opened at ingress when a request arrives without an upstream context; every subsequent hop opens a child span whose parent is carried in the incoming request, so the collector can reassemble the tree even though no single service saw the whole path. The identifier that threads the tree is the W3C traceparent, and its faithful propagation across every hop — including the asynchronous ones — is the single hardest part of the pattern, which is why it has its own procedure in Propagating Trace Context Through Tile Pipelines. Without unbroken propagation the tree fractures into disconnected fragments and the diagnostic value collapses.

What makes a spatial trace worth more than a generic one is the attributes. A span that records only http.status_code tells you a request was slow; a span that also records crs, bbox, tile z/x/y, and the row count returned tells you the request was slow because it asked for a continent-scale bounding box at a low zoom against a layer whose geometry column was missing a spatial index. Three spatial-specific concerns shape the design. First, attribute discipline: bbox and CRS belong on spans as low-cardinality, queryable attributes, never baked into the span name, or the backend’s cardinality explodes. Second, sampling that preserves the interesting traces: uniform head sampling discards exactly the rare slow large-bbox requests you most need, so the policy combines a low baseline head-sampling rate with tail-based rules that keep every errored or over-budget trace. Third, the async boundary: tile generation frequently hands off through a queue, and a naive instrumentation loses the parent context at that hop, orphaning the backend spans — closing that gap is the whole point of context propagation through the pipeline. Sampling and attribute conventions are versioned alongside the domain’s other contracts so a change to what a trace records is reviewed, not mutated live.

# Specification & Contract Reference

Every domain that participates in a trace pins the same surface: how it names spans, which spatial attributes it must attach, how it propagates context, and how it samples. The table below is the minimum a production tracing setup declares before it is trusted for diagnosis.

Span names must stay low-cardinality: tile.render is a good name, tile.render z=12 bbox=… is a cardinality bomb that belongs in attributes. The attribute keys align with the OpenTelemetry semantic conventions so a general-purpose backend can query them, while the spatial keys (crs, bbox, tile) are the domain’s stable extension. Because the same crs and bbox attributes are recorded on spans and can be carried as baggage, a trace becomes joinable against the SLI series behind Monitoring Vector Query p95 Latency — a p95 breach and the traces that caused it share a vocabulary.

# Production Implementation

Instrumentation is done with the OpenTelemetry SDK and versioned in the domain’s repository. The reference below sets up a tracer, opens a child span for the spatial query, and attaches the spatial attributes that make the span diagnosable. Idempotency is inherent — a span is a pure record of work that already happened, so re-emitting or retrying export never mutates domain state — and zero-trust applies to the export path: spans leave the process only over mTLS to the collector, and no attribute carries a secret or a raw user identifier.

# tracing.py — OpenTelemetry instrumentation with spatial span attributes.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Exporter reaches the collector over mTLS (zero-trust transport).
provider = TracerProvider(resource=Resource.create({"service.name": "tile-backend"}))
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="https://otel-collector.mesh.internal:4317"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("tile-backend")

def render_tile(z: int, x: int, y: int, bbox: str, crs: str, conn) -> bytes:
    # Child span; parent context is extracted from the inbound request upstream.
    with tracer.start_as_current_span("tile.render") as span:
        # Spatial attributes: low-cardinality, queryable, no bbox in the span name.
        span.set_attribute("crs", crs)
        span.set_attribute("bbox", bbox)
        span.set_attribute("tile", f"{z}/{x}/{y}")
        with tracer.start_as_current_span("postgis.ST_AsMVT") as q:
            rows = conn.execute(
                "SELECT ST_AsMVT(t) FROM mvt_features(%s,%s,%s,%s) AS t",
                (z, x, y, crs),
            )
            q.set_attribute("rows", rows.rowcount)   # spot runaway scans
            return rows.fetchone()[0]

Sampling is configured at the collector so the decision is centralised and the interesting traces survive. A cheap head-sampling floor keeps a representative 5% of all traffic, and a tail-based policy retains every trace that errored or exceeded the latency budget, so the rare slow large-bbox request — the one worth diagnosing — is never discarded.

# otel-collector.yaml (processors) — keep the interesting traces.
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}   # every over-budget trace is retained
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

The parent context must survive every hop for this to work, including the asynchronous tile-generation handoff. The mechanics of injecting and extracting traceparent across HTTP and a queue boundary — and carrying crs/bbox as baggage — are covered in Propagating Trace Context Through Tile Pipelines, which this instrumentation assumes is already in place upstream.

# Diagnostic Runbook

When a trace is unhelpful the fault is almost always broken propagation, missing attributes, or a sampling policy that discarded the evidence — rarely the backend itself. Work the steps in order.

1. Confirm the trace is whole. Look up the trace_id and count distinct service.name spans. Fewer services than the request touched means propagation broke at a hop — the async boundary is the usual culprit.
2. Find the dominant span. Sort the tree by self-time, not total time. The span with the largest self-time is where the latency actually lived; a long parent with a longer child means the child is the target, not the parent.
3. Read the spatial attributes. On the dominant span, inspect bbox, crs, and rows. A continent-scale bbox at low zoom or a row count in the millions localises the fault to the query shape, not the infrastructure.
4. Check for a CRS mismatch. Compare the span’s crs attribute against the layer’s storage CRS. An on-the-fly reprojection inside the query is a common hidden cost that a crs attribute makes visible.
5. Verify the sample was kept for a reason. Confirm whether the trace survived on the error or latency tail rule or merely the baseline. A slow trace that was kept only by chance means the tail policy is not catching the class of request you care about.
6. Correlate with the SLI. Join the trace’s crs/bbox against the p95 series for the same window; if the burning percentile shares the trace’s attributes, you have found the representative cause, not an outlier.
7. Reproduce with the same attributes. Replay a request with the offending bbox/z/x/y in staging and confirm the same span dominates, which proves the fix before it ships.

# SLA Targets & Performance Baselines

Tracing must be cheap enough to leave on and complete enough to trust. The budget below is what the tracing subsystem holds so it aids rather than distorts the request path.

Holding these baselines depends on batching span export so instrumentation never blocks the request, keeping span counts per request bounded, and centralising the sampling decision at the collector so the interesting traces are retained without every service carrying its own policy.

# Governance & Compliance Notes

Tracing conventions are governance artifacts because a trace can leak as much as it reveals. Span names, required spatial attributes, and sampling policy are committed to source control and reviewed, so a change to what a trace records — or retains — is auditable rather than silent. This keeps the tracing contract aligned with the catalog records in Metadata Cataloging for Raster/Vector: a traced service maps to a governed data product with a documented instrumentation surface.

Privacy and residency are concrete constraints on attributes. Spans must never carry a raw user identifier, an API key, or a full address string; a bounding box is spatial context, but a precise point tied to an identity is personal data, so the attribute convention forbids it and review enforces the ban. Where a tenant carries data-residency obligations, its traces inherit them — spans are exported to an in-region collector and retained in-region rather than crossing a boundary for centralised analysis. Exercise the propagation path in game-day drills that inject a broken traceparent at the async hop, so orphaned-span detection, the tail-sampling retention of errored traces, and the mTLS export path are all validated before an incident depends on the trace being whole.

# Related

• Up to the parent: Spatial Pipeline Orchestration & Observability
• Propagating Trace Context Through Tile Pipelines — carrying traceparent across HTTP and queue hops
• SLA Monitoring for Spatial Data Products — the aggregate SLOs a trace explains
• Monitoring Vector Query p95 Latency — the p95 signal traces join against
• Orchestrating Spatial Pipelines in Python — the pipelines whose async hops must propagate context