# Spatial Pipeline Orchestration & Observability

Federated spatial domains do not fail at the boundary; they fail in the pipelines behind it — a reprojection that ran twice, a tile write that never completed, a topology check that was quietly skipped under load. Spatial Pipeline Orchestration & Observability is the discipline that makes those pipelines idempotent, reproducible, and measurable, so that every geospatial data product a domain publishes carries a provable lineage and a governed SLA. This is the third top-level section of the site, and it operationalizes what the other two define: the domain-ownership model set out in Geospatial Data Mesh Fundamentals and the request-time enforcement built in Federated Ownership & Routing Architecture. From the site root at /, this section is where architecture becomes runtime — Python-based orchestration with Prefect, Airflow, and Dagster on one side, and Prometheus, OpenTelemetry, and horizontal autoscaling on the other, joined so that a failed stage is retried safely and a breached target maps to a concrete domain, product, and DAG run.

Figure — Orchestrated DAG stages transform ingested spatial data into domain output ports while a metrics-and-traces sidecar feeds Prometheus and a tracing backend.

The topology encodes the core claim of this section: orchestration and observability are not two systems but two faces of one. The orchestrator owns what runs and in what order; observability owns what actually happened and whether it met contract. Because every stage is idempotent and every run is instrumented, a retried pipeline converges to the same published artifact, and a Prometheus alert or a distributed trace can be traced back to the exact DAG run, coordinate reference system, and bounding box that produced it. The remainder of this section works through three concerns in depth: orchestrating spatial pipelines in Python, SLA monitoring for spatial data products, and distributed tracing for spatial request flows.

# Domain-Driven Design & Boundary Definition

Orchestration in a data mesh is domain-scoped, not platform-scoped. Each domain owns its own pipelines end to end — the DAGs that ingest, reproject, validate, tile, and publish its spatial products — and exposes only the resulting output ports, never the internal task graph. This is the operational counterpart to the boundary discipline in Spatial Domain Boundary Design: a boundary that is drawn at design time must be honoured at runtime by a pipeline that cannot silently reach across it. A cadastral domain’s reprojection DAG has no business writing into the environmental domain’s tile cache, and the orchestrator’s isolation model enforces that with per-domain workspaces, credentials, and result stores.

The unit that a pipeline produces is the spatial data product, and the pipeline is the mechanism that keeps the product’s published contract true over time. When a domain advertises a product as available in EPSG:4326 at 10m resolution under the versioned identifier v1.2.0-crs:EPSG:4326-res:10m, it is the orchestration layer that guarantees the artifact behind that identifier was actually reprojected to EPSG:4326, topologically validated, and tiled to the declared resolution — and that re-running the pipeline never mutates the artifact a consumer has already pinned. Reproducibility is therefore a boundary property: the same inputs and the same pipeline version must yield a byte-identical output, or the product’s version contract is a fiction.

Observability boundaries mirror ownership boundaries. Each domain emits telemetry tagged with its domain identifier, product identifier, and pipeline version, so that a platform-wide dashboard can be sliced back to a single owning team without cross-domain leakage. A breached availability target on the logistics tile port raises an alert routed to the logistics on-call rotation, not to a central operations desk that lacks the context to act. Governance councils set the baseline metrics and thresholds; domains retain autonomy over how their pipelines meet them.

# Core Concepts & Specifications

Orchestration and observability share a compact but precise vocabulary. The terms below recur across every DAG definition, every alert rule, and every trace on this site, and pinning their spatial meaning is what keeps a pipeline reproducible rather than merely runnable.

CRS handling is the single most common source of non-reproducibility, so it is pinned as a first-class pipeline concern. Every domain reprojects incoming data to the canonical geographic CRS EPSG:4326 at ingest, retains EPSG:3857 only for web-mercator tiling, and uses projected zones such as EPSG:32633 solely inside survey-grade domains. A pipeline that accepts data in an undeclared CRS must fail closed rather than guess. The reprojection contract, the topology contract, and the tiling contract are all versioned together in the product identifier, so a change to any one of them is a visible version bump rather than a silent drift — the mechanics of which are worked through in orchestrating spatial pipelines in Python.

Observability telemetry is spatial-specific by design. Generic request-count and latency metrics are necessary but insufficient; a spatial platform must also emit bounding-box coverage, CRS-transformation latency, tile-cache hit ratio, freshness lag, and topology-validation failure rate. These are the signals that let an SLA breach resolve to a concrete cause — a cold tile cache, a slow reprojection, a producer shipping invalid polygons — rather than an opaque platform alert. SLA monitoring for spatial data products specifies how these indicators become Prometheus recording rules, alert rules, and autoscaling triggers.

# Platform Engineering Patterns

The load-bearing pattern is idempotent, retry-safe orchestration. Spatial tasks are expensive and prone to transient failure — an object-store timeout mid-tiling, a PostGIS deadlock during a spatial join, a worker eviction under autoscaling — so every task must be safe to retry without producing a duplicate or corrupt artifact. The mechanism is a deterministic cache key computed from the task’s semantic inputs, most commonly SHA-256(bbox + crs + layer), combined with result persistence: before a task runs, the orchestrator checks whether an artifact already exists for that key and, if so, returns it instead of recomputing. This is exactly the idempotency discipline that the routing plane relies on in Async Execution for Heavy Spatial Queries, applied to batch pipelines rather than request handling.

Retries are bounded and observable. A task declares its retry count and backoff — for example retry_delay_seconds with exponential growth — and every retry carries the same idempotency key, so a provider or database that already materialized the result on the first partial attempt returns it rather than duplicating work. The three major Python orchestrators express this differently: Prefect through cache_key_fn and result persistence, covered in building idempotent spatial DAGs with Prefect; Airflow through deterministic output paths and task-level retries, covered in automating CRS reprojection in Airflow; and Dagster through software-defined assets and asset checks, covered in topology validation with Dagster asset checks.

Backfill and replay are the second load-bearing pattern, and they are where naive pipelines break first. A spatial domain is routinely asked to reprocess history: a datum correction lands, a source provider reissues a year of imagery, a topology rule tightens and every polygon published under the old rule must be revalidated. If the pipeline’s identity is when it ran rather than what it consumed, a backfill produces a second, conflicting artifact for a period that already has one, and consumers who pinned a version find the data underneath them changed. The fix is to make the partition — not the schedule — the unit of work. A DAG run is keyed by the spatial and temporal partition it materializes (bbox, crs, and an observation window), so re-running the pipeline for 2026-03 over the cadastral extent overwrites exactly that partition, deterministically, and leaves every other partition untouched. Replay then becomes an ordinary operation rather than an incident: select the affected partitions, re-run, and let result persistence skip the ones whose inputs did not actually change.

Concurrency control is the third. Spatial tasks contend for a small number of genuinely scarce resources — PostGIS connection slots, GDAL’s memory ceiling during warp operations, object-store write throughput — and an orchestrator that fans out a thousand tile tasks without a limit converts a healthy database into a queue of lock waits. Each domain therefore declares concurrency pools per resource rather than per DAG: a postgis-write pool sized to the connection budget, a reproject pool sized to worker memory, a tile-publish pool sized to the store’s write quota. Tasks acquire a slot in the pool their work actually consumes, so an expensive reprojection cannot starve a cheap catalog write, and a burst in one domain cannot exhaust the shared database on behalf of every other.

Zero-trust extends into the pipeline, not just the request path. Each domain’s orchestration workers authenticate to PostGIS and object storage with scoped, rotated credentials; no pipeline holds a token that reaches another domain’s store. Result stores are encrypted and access-audited, and every DAG run is recorded with its inputs, its pipeline version, and the identity of the worker that produced it. This means a reproducibility claim is also an audit claim: given a product version, the platform can name the exact pipeline run, code revision, and CRS that produced it. Infrastructure is declarative — DAG definitions, alert rules, and autoscaler manifests are versioned as code and promoted through CI, never hand-edited in a console.

# Metadata, Cataloging & Federated Discovery

Pipelines are the producers of catalog metadata, not merely consumers of it. When a DAG run publishes a product, it writes back the artifact’s bounding box, native CRS, resolution, freshness timestamp, quality metrics, and pipeline version into the federated catalog described in Metadata Cataloging for Raster/Vector. The catalog entry is therefore a byproduct of a successful, validated run — a product cannot appear as available until its pipeline has actually materialized and validated the artifact behind it. This closes a gap that batch ETL systems routinely leave open, where a catalog advertises data that a failed job never produced.

Lineage is captured automatically because the orchestrator already holds the task graph. Each published artifact records the upstream assets it derived from — the raw ingest, the reprojected intermediate, the validated geometry — so a consumer or auditor can walk the derivation backwards from a tile to the source observation. Distributed tracing extends this lineage into the request path: a traceparent propagated from the ingress gateway through the tile cache to the renderer lets an operator correlate a slow consumer request with the specific pipeline run and cache state that served it, a linkage detailed in propagating trace context through tile pipelines.

Quality metrics travel with the metadata rather than living in a separate dashboard. A published artifact carries the counts that a consumer needs in order to trust it: how many features were validated, how many were repaired, how many were rejected, what fraction of the declared bounding box the artifact actually covers, and what the topology-validation pass rate was for the run. A consumer choosing between two candidate products can then compare their measured quality directly, instead of inferring it from the age of the last successful run. This also gives governance a lever that does not require inspecting anyone’s pipeline: a domain whose repaired-feature ratio climbs month over month is visibly accumulating source-data debt, and the trend is legible from catalog metadata alone.

Freshness is a discoverable, monitored property. Every product advertises an expected update cadence, and the pipeline emits a freshness-lag metric measuring the gap between the newest source observation and the newest published artifact. When that lag exceeds the product’s declared cadence, the monitoring layer raises a staleness alert against the owning domain — so discovery never returns a product that is silently stale.

# Governance, Lifecycle & SLA Baselines

Governance in this section is enforced by the pipeline and the monitoring layer acting together. A product’s lifecycle state governs how aggressively its pipeline runs and how tightly its SLA is enforced, and the orchestrator treats that state as a first-class input. The routing-side view of these states lives in Spatial Product Lifecycle Management; the pipeline-and-observability view is summarized below.

SLA baselines make orchestration and observability measurable rather than aspirational. The targets below apply to a production spatial pipeline and its published products; each has an alert threshold wired into Prometheus and a remediation owner on the platform or domain team.

These baselines only hold if the pipeline is idempotent and the telemetry is spatial-specific. A pipeline that duplicates tiles on retry inflates its own latency and corrupts its freshness signal; a monitoring stack that measures only generic HTTP latency cannot tell a cold cache from a slow reprojection. The three clusters that follow build each half of this: reproducible orchestration, and the SLA monitoring and distributed tracing that prove it.

# Failure Modes & Recovery Semantics

Every pipeline in this section is designed backwards from the ways spatial work actually fails. Three categories matter, and they demand different recovery semantics rather than a single blanket retry.

Transient infrastructure failures — an object-store timeout, a PostGIS deadlock, a worker evicted mid-warp — are safe to retry because the work is unchanged and the idempotency key still resolves to the same artifact. These get bounded exponential retries and nothing more; escalating them to a human is noise. Data-quality failures — a source shipping self-intersecting polygons, a shapefile whose declared CRS contradicts its coordinates, a raster with a missing overview level — must not retry, because the input will not improve on a second attempt. They fail the run closed, hold the previously published artifact in place, and raise a producer-facing alert. Contract failures — the pipeline produced an artifact that no longer satisfies the product’s declared resolution, extent, or schema — are the most dangerous, because the artifact exists and looks plausible. These block publication at the asset-check gate and leave the catalog pointing at the last good version.

The distinction is operationally load-bearing: a platform that retries a data-quality failure eight times burns an hour of compute, delays the staleness alert by an hour, and still fails. The recovery ladder below is the default posture for a production domain.

The critical invariant across all five rows: a failed run never leaves a half-published product. Publication is the last step, it is atomic, and it is gated on every validation having passed. A consumer therefore only ever sees an artifact that a complete, validated run produced — the alternative, where a partially written tile set is discoverable, converts one domain’s pipeline failure into every downstream consumer’s data-quality incident.

# Choosing an Orchestrator for Spatial Work

Prefect, Airflow, and Dagster are all viable for spatial pipelines, and this section covers all three rather than prescribing one. They differ in where they put the abstraction, and that difference maps directly onto which spatial concern is hardest for a given domain.

Airflow models the task graph. Its scheduler, backfill machinery, and pool-based concurrency are the most mature of the three, which matters when a domain’s dominant problem is reprocessing years of imagery on a shared, contended database. Its weakness for spatial work is that data assets are implicit: nothing in a bare Airflow DAG knows that a task produced a reprojected extent, so lineage and asset-level validation have to be bolted on. Prefect models the flow, with idempotency and caching as first-class runtime features — cache_key_fn plus result persistence expresses “this reprojection has already been computed for this bbox and CRS” more directly than any of the others, which suits domains whose pain is expensive, frequently-retried computation. Dagster models the asset: a reprojected layer, a validated geometry set, and a tile pyramid are declared objects with checks attached, so topology validation becomes a property of the data rather than a step in a script — the best fit where correctness gates dominate.

None of this is a ranking, and a mesh does not need one orchestrator. Because domains own their pipelines end to end and expose only output ports, three domains may reasonably run three different orchestrators — provided all of them honour the same contract: idempotent tasks, deterministic partition identity, atomic publication, and telemetry tagged with domain, product, and pipeline version. The orchestrator is an implementation detail behind the output port; the contract is not.

# Conclusion

Spatial Pipeline Orchestration & Observability is where a geospatial data mesh earns its reliability claims. Domain ownership and contract routing define what each domain must publish; orchestration guarantees that the pipeline behind each product is idempotent and reproducible, so a retry or replay converges to the same versioned artifact; and observability — Prometheus SLOs, OpenTelemetry traces, and metric-driven autoscaling — proves it, mapping every breach to a concrete domain, product, and run. Build the pipeline to be retry-safe and CRS-strict, instrument every stage with spatial-specific telemetry, and govern lifecycle and SLA through code, and the mesh gains a runtime that is not merely automated but accountable.

# Related

• Geospatial Data Mesh Fundamentals — the domain-ownership and product model this section operationalizes
• Federated Ownership & Routing Architecture — request-time enforcement that these pipelines produce artifacts for
• Orchestrating Spatial Pipelines in Python — idempotent, retry-safe DAGs across Prefect, Airflow, and Dagster
• SLA Monitoring for Spatial Data Products — Prometheus SLOs, alerting, and autoscaling for spatial workloads
• Distributed Tracing for Spatial Request Flows — end-to-end trace correlation across tile and query paths