# Orchestrating Spatial Pipelines in Python

A spatial pipeline that reprojects, validates topology, and tiles is only trustworthy if running it twice produces exactly one set of artifacts — yet the default behaviour of most orchestrators is to happily recompute and overwrite, doubling a spatial join or corrupting a half-written tile set on every retry. Orchestrating spatial pipelines in Python is the practice of making those graphs idempotent and retry-safe across the three orchestrators enterprises actually run — Prefect, Airflow, and Dagster — so that a transient PostGIS deadlock or an evicted worker never leaves a domain’s output port inconsistent. This page sits within Spatial Pipeline Orchestration & Observability and pairs with SLA Monitoring for Spatial Data Products: orchestration guarantees the artifact is produced exactly once, and monitoring proves it stayed within contract.

Figure — A five-stage spatial DAG where each task is keyed by an idempotency hash and guarded by bounded retries with backoff.

# Architectural Boundaries & Design Rationale

The orchestration layer sits strictly between a domain’s raw ingest and its published output ports, and it owns exactly one responsibility: turning source data into a validated, versioned artifact exactly once. It must not reach across a domain boundary, and it must not expose its internal task graph to consumers — a discipline inherited from Spatial Domain Boundary Design. What a consumer sees is a product at v1.2.0-crs:EPSG:4326-res:10m; what the pipeline guarantees is that the artifact behind that identifier was reprojected, validated, and tiled by a specific, reproducible run.

Idempotency is the design centre, and it is enforced by keying every task on the semantic identity of its inputs rather than on wall-clock time or run count. The canonical key is SHA-256(bbox + crs + layer): two runs over the same bounding box, in the same CRS, for the same layer, produce the same key and therefore resolve to the same persisted artifact. This turns a retry from a hazard into a no-op — a task that already succeeded returns its cached result, and a task that failed halfway re-runs against inputs that have not changed. The same principle underlies the request-side idempotency in Async Execution for Heavy Spatial Queries; here it is applied to batch stages.

The three orchestrators express idempotency through different primitives, and choosing among them is a boundary decision rather than a matter of taste. Prefect models it with cache_key_fn and result persistence, which suits event-driven, Python-native flows — detailed in building idempotent spatial DAGs with Prefect. Airflow models it with deterministic output paths and task retries, which suits schedule-driven ETL where a canonical file path is the natural idempotency anchor — detailed in automating CRS reprojection in Airflow. Dagster models it with software-defined assets and asset checks, which suits asset-centric platforms where validation is a declared property of the data — detailed in topology validation with Dagster asset checks. All three must agree on one thing: an invalid topology blocks publication rather than reaching a consumer.

# Specification & Contract Reference

A production spatial pipeline pins the following parameters in version control before it is allowed to publish. They are the minimal surface that makes a run reproducible and a retry safe.

The load-bearing detail is that the idempotency key, the result store key, and the published version identifier all reference the same immutable artifact. Because the key is computed from the bounding box, CRS, and layer — not from a timestamp — a scheduled re-run over unchanged inputs is a cache hit, and only genuinely new or changed data triggers recomputation. This is what lets a pipeline run on a cadence without re-tiling the entire estate on every tick, and it is why the topology gate must sit before publication: a validated artifact is cached, so an invalid one must never be written under a valid key.

# Production Implementation

The reference below is a Prefect 2.x flow that reprojects, validates, and tiles a spatial layer with idempotent task caching. Each task computes its own cache key from the bounding box, CRS, and layer, so a retry or a scheduled re-run returns the persisted result instead of recomputing. Idempotency and domain-scoped credentials are called out explicitly because both are mandatory: a task that is not keyed on its inputs will duplicate work on retry, and a worker that holds a cross-domain credential violates the mesh’s zero-trust posture.

# spatial_pipeline.py — idempotent Prefect flow: reproject -> validate -> tile -> publish.
# Run: prefect deployment run spatial-pipeline/nightly
import hashlib
from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

def idem_key(context, arguments) -> str:
    # Deterministic cache key: identical bbox+crs+layer => identical key => cached result.
    bbox = arguments["bbox"]
    crs = arguments["crs"]
    layer = arguments["layer"]
    digest = hashlib.sha256(f"{bbox}|{crs}|{layer}".encode()).hexdigest()
    return f"spatial-{layer}-{digest}"

@task(cache_key_fn=idem_key, cache_expiration=timedelta(days=7),
      retries=3, retry_delay_seconds=[10, 30, 90], persist_result=True)
def reproject(bbox: str, crs: str, layer: str) -> str:
    # Reproject the source layer to the canonical CRS; write to a key-derived path.
    out = f"s3://cadastral-domain/reprojected/{layer}.parquet"
    run_ogr2ogr(src=layer, dst=out, t_srs="EPSG:4326")  # domain-scoped credential
    return out

@task(cache_key_fn=idem_key, retries=0, persist_result=True)
def validate_topology(bbox: str, crs: str, layer: str, src: str) -> str:
    # Hard gate: invalid geometry blocks publication rather than propagating downstream.
    invalid = count_invalid_geometries(src)  # PostGIS ST_IsValid over the reprojected set
    if invalid > 0:
        raise ValueError(f"topology gate failed: {invalid} invalid geometries in {layer}")
    return src

@task(cache_key_fn=idem_key, retries=3, retry_delay_seconds=[10, 30, 90], persist_result=True)
def tile(bbox: str, crs: str, layer: str, src: str) -> str:
    # Tile to MVT; idempotent because the output path is derived from the same key.
    out = f"s3://cadastral-domain/tiles/{layer}/"
    build_mvt(src=src, dst=out, minzoom=0, maxzoom=14)
    return out

@flow(name="spatial-pipeline")
def spatial_pipeline(bbox: str, crs: str, layer: str, version: str):
    reprojected = reproject(bbox, crs, layer)
    validated = validate_topology(bbox, crs, layer, reprojected)
    tiles = tile(bbox, crs, layer, validated)
    publish(layer, version=version, artifact=tiles)  # writes catalog entry on success only

if __name__ == "__main__":
    spatial_pipeline(bbox="-89.7,39.7,-89.6,39.8", crs="EPSG:4326",
                     layer="parcels", version="v1.2.0-crs:EPSG:4326-res:10m")

Heavy stages that would exhaust a worker — full-estate re-tiling, cross-domain spatial joins — must run on dedicated queues rather than blocking the scheduling loop, mirroring the backpressure discipline of Async Execution for Heavy Spatial Queries. The catalog write happens only on a successful, validated run, so a product never appears as available until its artifact genuinely exists — the linkage into Metadata Cataloging for Raster/Vector.

# Diagnostic Runbook

When a spatial pipeline misbehaves, the fault is almost always in cache keying, the topology gate, retry configuration, or credential scope. Work the steps in order.

1. Confirm the run and its inputs. Read the DAG run record and verify the bounding box, CRS, and layer match the intended product version. A run producing the wrong version points to a parameter drift, not a code fault.
2. Inspect the idempotency key. Recompute SHA-256(bbox + crs + layer) for the run and compare it to the persisted result-store key. A mismatch means a non-deterministic input (a timestamp, a run id) leaked into the key and defeated caching.
3. Check for duplicate artifacts. List the output path and confirm exactly one artifact exists per key. Duplicates indicate a task wrote to a path that is not derived from the key, so retries did not converge.
4. Audit the topology gate. Confirm the validate task ran and failed closed on invalid geometry. A published product with invalid polygons means the gate was skipped, mis-ordered after tiling, or its retry count was non-zero and it masked the failure.
5. Review retry and backoff behaviour. Read the task retry log; retries that fire back-to-back without backoff point to a mis-set retry_delay_seconds, which amplifies load on an already-degraded PostGIS.
6. Verify credential scope. Confirm the worker authenticated with a domain-scoped credential and did not reach another domain’s store. A cross-domain access is a zero-trust violation regardless of whether the run succeeded.
7. Replay against unchanged inputs. Re-run the flow with identical parameters and assert every task reports a cache hit and no new artifact is written — the definitive idempotency check.

# SLA Targets & Performance Baselines

Holding these baselines depends on keying every task deterministically, persisting results, and keeping the topology gate before publication with zero retries so a validation failure is never silently swallowed.

# Governance & Compliance Notes

Pipeline definitions are auditable artifacts. The DAG code, cache-key function, retry policy, and topology gate are committed to source control and promoted only after CI verifies that a replay over unchanged inputs produces zero recomputation and that the topology gate fails closed on a known-invalid fixture. This keeps the pipeline aligned with the product’s published contract and with the lifecycle rules in Spatial Product Lifecycle Management: only production-state products run on a guaranteed cadence with full result persistence.

Every run is recorded with its inputs, its code revision, its pipeline version, and the identity of the worker, written to an append-only audit stream. Because artifacts are immutable and keyed on their inputs, a compliance reviewer can name the exact run that produced any published product version and confirm it was topologically validated before publication. Credentials are domain-scoped and rotated, and cross-domain store access is denied by default, so a pipeline can never be used to exfiltrate or corrupt another domain’s data.

# Related

• Up to the parent: Spatial Pipeline Orchestration & Observability
• Building Idempotent Spatial DAGs with Prefect — cache_key_fn and result persistence in depth
• Automating CRS Reprojection in Airflow — deterministic output paths and ogr2ogr reprojection
• Topology Validation with Dagster Asset Checks — @asset_check gating on ST_IsValid
• SLA Monitoring for Spatial Data Products — proving the produced artifact stays within contract
• Async Execution for Heavy Spatial Queries — where heavy joins and full re-tiles are offloaded