# Spatial Pipeline Orchestration & Observability

Federated spatial domains do not fail at the boundary; they fail in the pipelines behind it — a reprojection that ran twice, a tile write that never completed, a topology check that was quietly skipped under load. Spatial Pipeline Orchestration & Observability is the discipline that makes those pipelines idempotent, reproducible, and measurable, so that every geospatial data product a domain publishes carries a provable lineage and a governed SLA. This is the third top-level section of the site, and it operationalizes what the other two define: the domain-ownership model set out in Geospatial Data Mesh Fundamentals and the request-time enforcement built in Federated Ownership & Routing Architecture. From the site root at /, this section is where architecture becomes runtime — Python-based orchestration with Prefect, Airflow, and Dagster on one side, and Prometheus, OpenTelemetry, and horizontal autoscaling on the other, joined so that a failed stage is retried safely and a breached target maps to a concrete domain, product, and DAG run.

Figure — Orchestrated DAG stages transform ingested spatial data into domain output ports while a metrics-and-traces sidecar feeds Prometheus and a tracing backend.

The topology encodes the core claim of this section: orchestration and observability are not two systems but two faces of one. The orchestrator owns what runs and in what order; observability owns what actually happened and whether it met contract. Because every stage is idempotent and every run is instrumented, a retried pipeline converges to the same published artifact, and a Prometheus alert or a distributed trace can be traced back to the exact DAG run, coordinate reference system, and bounding box that produced it. The remainder of this section works through three concerns in depth: orchestrating spatial pipelines in Python, SLA monitoring for spatial data products, and distributed tracing for spatial request flows.

# Domain-Driven Design & Boundary Definition

Orchestration in a data mesh is domain-scoped, not platform-scoped. Each domain owns its own pipelines end to end — the DAGs that ingest, reproject, validate, tile, and publish its spatial products — and exposes only the resulting output ports, never the internal task graph. This is the operational counterpart to the boundary discipline in Spatial Domain Boundary Design: a boundary that is drawn at design time must be honoured at runtime by a pipeline that cannot silently reach across it. A cadastral domain’s reprojection DAG has no business writing into the environmental domain’s tile cache, and the orchestrator’s isolation model enforces that with per-domain workspaces, credentials, and result stores.

The unit that a pipeline produces is the spatial data product, and the pipeline is the mechanism that keeps the product’s published contract true over time. When a domain advertises a product as available in EPSG:4326 at 10m resolution under the versioned identifier v1.2.0-crs:EPSG:4326-res:10m, it is the orchestration layer that guarantees the artifact behind that identifier was actually reprojected to EPSG:4326, topologically validated, and tiled to the declared resolution — and that re-running the pipeline never mutates the artifact a consumer has already pinned. Reproducibility is therefore a boundary property: the same inputs and the same pipeline version must yield a byte-identical output, or the product’s version contract is a fiction.

Observability boundaries mirror ownership boundaries. Each domain emits telemetry tagged with its domain identifier, product identifier, and pipeline version, so that a platform-wide dashboard can be sliced back to a single owning team without cross-domain leakage. A breached availability target on the logistics tile port raises an alert routed to the logistics on-call rotation, not to a central operations desk that lacks the context to act. Governance councils set the baseline metrics and thresholds; domains retain autonomy over how their pipelines meet them.

# Core Concepts & Specifications

Orchestration and observability share a compact but precise vocabulary. The terms below recur across every DAG definition, every alert rule, and every trace on this site, and pinning their spatial meaning is what keeps a pipeline reproducible rather than merely runnable.

CRS handling is the single most common source of non-reproducibility, so it is pinned as a first-class pipeline concern. Every domain reprojects incoming data to the canonical geographic CRS EPSG:4326 at ingest, retains EPSG:3857 only for web-mercator tiling, and uses projected zones such as EPSG:32633 solely inside survey-grade domains. A pipeline that accepts data in an undeclared CRS must fail closed rather than guess. The reprojection contract, the topology contract, and the tiling contract are all versioned together in the product identifier, so a change to any one of them is a visible version bump rather than a silent drift — the mechanics of which are worked through in orchestrating spatial pipelines in Python.

Observability telemetry is spatial-specific by design. Generic request-count and latency metrics are necessary but insufficient; a spatial platform must also emit bounding-box coverage, CRS-transformation latency, tile-cache hit ratio, freshness lag, and topology-validation failure rate. These are the signals that let an SLA breach resolve to a concrete cause — a cold tile cache, a slow reprojection, a producer shipping invalid polygons — rather than an opaque platform alert. SLA monitoring for spatial data products specifies how these indicators become Prometheus recording rules, alert rules, and autoscaling triggers.

# Platform Engineering Patterns

The load-bearing pattern is idempotent, retry-safe orchestration. Spatial tasks are expensive and prone to transient failure — an object-store timeout mid-tiling, a PostGIS deadlock during a spatial join, a worker eviction under autoscaling — so every task must be safe to retry without producing a duplicate or corrupt artifact. The mechanism is a deterministic cache key computed from the task’s semantic inputs, most commonly SHA-256(bbox + crs + layer), combined with result persistence: before a task runs, the orchestrator checks whether an artifact already exists for that key and, if so, returns it instead of recomputing. This is exactly the idempotency discipline that the routing plane relies on in Async Execution for Heavy Spatial Queries, applied to batch pipelines rather than request handling.

Retries are bounded and observable. A task declares its retry count and backoff — for example retry_delay_seconds with exponential growth — and every retry carries the same idempotency key, so a provider or database that already materialized the result on the first partial attempt returns it rather than duplicating work. The three major Python orchestrators express this differently: Prefect through cache_key_fn and result persistence, covered in building idempotent spatial DAGs with Prefect; Airflow through deterministic output paths and task-level retries, covered in automating CRS reprojection in Airflow; and Dagster through software-defined assets and asset checks, covered in topology validation with Dagster asset checks.

Zero-trust extends into the pipeline, not just the request path. Each domain’s orchestration workers authenticate to PostGIS and object storage with scoped, rotated credentials; no pipeline holds a token that reaches another domain’s store. Result stores are encrypted and access-audited, and every DAG run is recorded with its inputs, its pipeline version, and the identity of the worker that produced it. This means a reproducibility claim is also an audit claim: given a product version, the platform can name the exact pipeline run, code revision, and CRS that produced it. Infrastructure is declarative — DAG definitions, alert rules, and autoscaler manifests are versioned as code and promoted through CI, never hand-edited in a console.

# Metadata, Cataloging & Federated Discovery

Pipelines are the producers of catalog metadata, not merely consumers of it. When a DAG run publishes a product, it writes back the artifact’s bounding box, native CRS, resolution, freshness timestamp, quality metrics, and pipeline version into the federated catalog described in Metadata Cataloging for Raster/Vector. The catalog entry is therefore a byproduct of a successful, validated run — a product cannot appear as available until its pipeline has actually materialized and validated the artifact behind it. This closes a gap that batch ETL systems routinely leave open, where a catalog advertises data that a failed job never produced.

Lineage is captured automatically because the orchestrator already holds the task graph. Each published artifact records the upstream assets it derived from — the raw ingest, the reprojected intermediate, the validated geometry — so a consumer or auditor can walk the derivation backwards from a tile to the source observation. Distributed tracing extends this lineage into the request path: a traceparent propagated from the ingress gateway through the tile cache to the renderer lets an operator correlate a slow consumer request with the specific pipeline run and cache state that served it, a linkage detailed in propagating trace context through tile pipelines.

Freshness is a discoverable, monitored property. Every product advertises an expected update cadence, and the pipeline emits a freshness-lag metric measuring the gap between the newest source observation and the newest published artifact. When that lag exceeds the product’s declared cadence, the monitoring layer raises a staleness alert against the owning domain — so discovery never returns a product that is silently stale.

# Governance, Lifecycle & SLA Baselines

Governance in this section is enforced by the pipeline and the monitoring layer acting together. A product’s lifecycle state governs how aggressively its pipeline runs and how tightly its SLA is enforced, and the orchestrator treats that state as a first-class input. The routing-side view of these states lives in Spatial Product Lifecycle Management; the pipeline-and-observability view is summarized below.

SLA baselines make orchestration and observability measurable rather than aspirational. The targets below apply to a production spatial pipeline and its published products; each has an alert threshold wired into Prometheus and a remediation owner on the platform or domain team.

These baselines only hold if the pipeline is idempotent and the telemetry is spatial-specific. A pipeline that duplicates tiles on retry inflates its own latency and corrupts its freshness signal; a monitoring stack that measures only generic HTTP latency cannot tell a cold cache from a slow reprojection. The three clusters that follow build each half of this: reproducible orchestration, and the SLA monitoring and distributed tracing that prove it.

# Conclusion

Spatial Pipeline Orchestration & Observability is where a geospatial data mesh earns its reliability claims. Domain ownership and contract routing define what each domain must publish; orchestration guarantees that the pipeline behind each product is idempotent and reproducible, so a retry or replay converges to the same versioned artifact; and observability — Prometheus SLOs, OpenTelemetry traces, and metric-driven autoscaling — proves it, mapping every breach to a concrete domain, product, and run. Build the pipeline to be retry-safe and CRS-strict, instrument every stage with spatial-specific telemetry, and govern lifecycle and SLA through code, and the mesh gains a runtime that is not merely automated but accountable.

# Related

• Geospatial Data Mesh Fundamentals — the domain-ownership and product model this section operationalizes
• Federated Ownership & Routing Architecture — request-time enforcement that these pipelines produce artifacts for
• Orchestrating Spatial Pipelines in Python — idempotent, retry-safe DAGs across Prefect, Airflow, and Dagster
• SLA Monitoring for Spatial Data Products — Prometheus SLOs, alerting, and autoscaling for spatial workloads
• Distributed Tracing for Spatial Request Flows — end-to-end trace correlation across tile and query paths