# Async Execution for Heavy Spatial Queries

The broker tier that lifts long-running spatial compute off the request path and runs it inside isolated, domain-owned worker pools.

Multi-polygon intersections, raster-vector overlays, and network-topology traversals routinely exceed the synchronous request budget that an interactive API can hold open. Inside the Federated Ownership & Routing Architecture, these workloads cannot be allowed to occupy gateway connection slots or share a thread pool with feature queries — a single multi-minute mosaicking job would otherwise stall an entire domain. This page specifies the broker pattern that decouples heavy spatial execution from the request lifecycle: a deterministic job state machine, spatially-partitioned worker pools, exactly-once result materialization, and the diagnostic and SLA discipline that keeps the whole thing observable. It is the backend contract behind the async routes defined in API Gateway Mapping for GIS Services, and it consumes only the validated payloads that pass Schema Contracts for Vector/Tile Data at the edge.

Figure — Heavy queries are decoupled from the request path: submit, enqueue, compute, commit, then poll for the materialized result.

# Architectural Boundaries & Design Rationale

Async spatial execution exists to enforce a hard separation between the acceptance of a query and its computation. The gateway accepts a job, validates it, and returns 202 Accepted in milliseconds; the actual ST_Intersection or raster overlay runs minutes later inside a worker pool that has no connection to the ingress thread that submitted it. This decoupling is what protects the broader mesh: a saturated mosaicking queue cannot exhaust the connection pool serving cadastral feature reads, because the two never share a runtime. The same domain-ownership model that drives Spatial Domain Boundary Design is projected down onto compute here — each worker pool is bound to one domain, reads only datasets that domain has explicitly published, and writes only into that domain’s storage path.

Treating heavy execution as a brokered, isolated tier prevents four recurring failure modes:

1. Thread starvation. A synchronous polygon-intersection that runs for 90 seconds holds a worker thread the whole time; a few concurrent ones drain the pool and every fast query queues behind them. Brokering moves these slow, multi-minute jobs off the request path entirely.
2. Duplicate computation. Network partitions and queue redeliveries mean a job may be dispatched more than once. Without an idempotency key, a retried submission enqueues a second multi-minute job and two workers race to write the same output path.
3. Partial writes. A worker that crashes mid-write leaves a truncated GeoParquet file or a half-populated tile pyramid that a consumer may read as if it were complete.
4. Cross-domain leakage. A worker that can reach any storage bucket can exfiltrate a neighbouring domain’s cadastral geometries; isolation must be enforced at the network and credential layer, not by convention.

The orchestration layer answers all four with a bounded job state machine. Every job moves through a small, append-only set of states, and the terminal commit is the only transition that makes a result visible. Submission writes QUEUED; a worker that claims the job moves it to IN_PROGRESS and stamps claimed_at; successful materialization moves it to COMPLETED; an unrecoverable error moves it to FAILED and parks the payload in a dead-letter queue. Stale IN_PROGRESS jobs whose claimed_at exceeds the SLO deadline are reaped back to QUEUED for one bounded retry. Because the ledger is append-only and keyed on the idempotency hash, replays are idempotent by construction — a duplicate submission resolves to the existing job record rather than spawning a new one.

Cross-domain dependencies are resolved through explicit contract negotiation rather than implicit coupling. Workers never reach across a boundary to pull a dataset on demand; they consume only what an owning domain has published, and cache-invalidation signals propagate through the Domain Sync Protocols for Spatial Data so that a worker never computes against a stale or partially replicated geometry. When a query must touch multiple domains, the fan-out is governed by the Cross-Domain Routing Strategies that pin each partition to a deterministic worker.

# Specification & Contract Reference

Every async job is governed by an explicit submission contract: the fields the broker validates, the spatial constraints it enforces, and the idempotency semantics it guarantees. The gateway rejects a malformed submission synchronously at ingress so that poison messages never reach the queue. The surface below is the minimum the broker enforces before a job is enqueued.

The submission contract is not duplicated guesswork — geometry topology, CRS, and property types are validated against the same versioned definitions the producer side enforced in CI, so a payload that passed Schema Contracts for Vector/Tile Data also passes broker ingress. Validated jobs are routed to domain-specific compute pools using the partition_key: consistent hashing on H3 or S2 cells gives deterministic worker affinity, which keeps a spatial index hot in node-local cache and minimizes inter-node shuffle. Job state lives in a distributed, append-only ledger (DynamoDB, PostgreSQL with logical replication, or etcd), and idempotency is enforced at the queue consumer by hashing job_idempotency_key and discarding duplicate claims.

# Production Implementation

Worker pools are provisioned as infrastructure-as-code so that parallelism, resource ceilings, and queue endpoints are versioned alongside the contracts they serve. The Kubernetes Job below runs a fixed-parallelism spatial worker pool with explicit memory and CPU limits — heavy raster overlays are memory-bound, and an unbounded pod will OOM-kill mid-write. Zero-trust is non-negotiable here: each pod runs in a domain-scoped subnet, presents a short-lived credential, and is admitted only after its readiness probe confirms the queue and spatial-index dependencies are reachable.

# Domain-scoped spatial worker pool. Memory limits are sized for the worst-case
# raster overlay; restartPolicy: Never so a crashed pod surfaces in the ledger
# rather than silently retrying and racing on the same output path.
apiVersion: batch/v1
kind: Job
metadata:
  name: spatial-worker-pool
  namespace: geospatial-mesh
spec:
  parallelism: 4
  completions: 4
  backoffLimit: 2          # bounded retries; exhausted jobs land in the DLQ
  template:
    spec:
      containers:
      - name: spatial-compute
        image: registry.platform.io/spatial-worker:v2.4
        env:
        - name: OTEL_SERVICE_NAME
          value: "async-spatial-worker"
        - name: QUEUE_ENDPOINT
          value: "amqp://rabbitmq.geospatial-mesh.svc.cluster.local:5672"
        resources:
          requests:
            memory: "8Gi"
            cpu: "4"
          limits:
            memory: "16Gi"   # hard ceiling stops a runaway overlay starving the node
            cpu: "8"
        readinessProbe:
          httpGet:           # idiomatic: surfaces HTTP status to Kubernetes
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      restartPolicy: Never

The readinessProbe uses httpGet rather than exec: curl. The curl pattern requires the curl binary in the image and does not surface HTTP status codes to Kubernetes, so a worker that returns 503 dependency_unavailable would still be marked ready. The httpGet probe reads the status code directly and is the idiomatic approach.

Idempotency and exactly-once materialization are enforced with a two-phase commit. The worker computes into a staging path keyed by the idempotency hash, validates the result, and only then performs an atomic move into the domain-owned final path — the ledger transitions to COMPLETED after the storage operation succeeds, never before. A retried or duplicated job that finds the final path already present short-circuits to the existing result.

# Two-phase commit for a spatial job. Phase 1 writes to a staging key; phase 2 is an
# atomic rename into the domain path. The ledger is the source of truth — it flips to
# COMPLETED only once the object store confirms the move.
import hashlib

def execute_spatial_job(job, store, ledger, compute):
    key = hashlib.sha256(job["job_idempotency_key"].encode()).hexdigest()
    final_path = f"s3://{job['domain']}/results/{key}.{job['output_format']}"

    # Idempotent short-circuit: a prior run already materialized this exact job.
    if store.exists(final_path):
        ledger.upsert(key, status="COMPLETED", result=final_path)
        return final_path

    # Zero-trust: the worker's JWT spatial_scope must contain the job extent.
    if not within_scope(job["bbox"], job["spatial_scope"]):
        ledger.upsert(key, status="FAILED", reason="scope_denied")
        raise PermissionError("job extent outside authorized spatial_scope")

    # Phase 1 — compute into staging keyed by the idempotency hash.
    staging_path = f"s3://{job['domain']}/staging/{key}.part"
    ledger.upsert(key, status="IN_PROGRESS")
    result = compute(job)                       # ST_Intersection / overlay / traversal
    store.write(staging_path, result)

    if not validate_geometry(result, job["input_crs"]):
        ledger.upsert(key, status="FAILED", reason="invalid_geometry")
        raise ValueError("post-compute geometry validation failed")

    # Phase 2 — atomic move makes the result visible; commit the ledger last.
    store.rename(staging_path, final_path)      # single atomic operation
    ledger.upsert(key, status="COMPLETED", result=final_path)
    return final_path

Worker-to-storage communication runs under mutual TLS with short-lived IAM credentials, and access is gated by attribute-based access control scoped to the job’s bounding box and data_classification tier. A worker must present a signed JWT carrying spatial_scope and data_classification claims before it can touch a domain bucket — there is no ambient credential that grants cross-domain reach. Cold-start latency is held down by keeping pools warm with a Horizontal Pod Autoscaler triggered on queue depth, and by pre-fetching frequently accessed PostGIS GiST indexes or GeoParquet spatial partitions into node-local NVMe. When partitioning, prefer spatial locality over uniform distribution so that a join’s inputs land on the same node — the partition-aware strategy is detailed in Optimizing async execution for spatial joins.

# Diagnostic Runbook

When an async spatial job misbehaves, the failure is almost always in idempotency state, header propagation, worker readiness, registry sync, or a tripped SLO — rarely the network. Work the steps in order; each isolates one boundary the broker enforces.

1. Recover the job trace. Read the X-Request-Trace-ID returned with the 202 and query the tracing backend for the job_id span. Attach trace_id, span_id, and job_id to every worker log line so a stalled job correlates cleanly with its compute span. A missing span means the job never left ingress — jump to step 3.
2. Inspect ledger state. Query the ledger for status=IN_PROGRESS AND claimed_at < (now() - slo_deadline_ms). These are stalled jobs; the span from step 1 will show whether the bottleneck is raster I/O wait, a topology-validation timeout, or an OOM kill.
3. Confirm contract and header propagation. Verify job_idempotency_key, spatial_scope, and data_classification survived the gateway hop. A dropped scope claim collapses access checks and surfaces as a spurious 403 or, worse, a cross-domain read.
4. Check worker readiness and queue lag. Read the consumer lag on the ingress queue and the readiness-probe state of the pool. Jobs stuck in QUEUED with healthy lag usually mean failed readiness probes (often expired mTLS certs) keeping pods out of rotation.
5. Validate idempotency before any replay. Before re-enqueuing a dead-lettered job, hash its job_idempotency_key and check the ledger. Re-enqueue only if no COMPLETED record exists — otherwise a replay duplicates a multi-minute computation.
6. Reconcile staging against final paths. For jobs marked COMPLETED but missing in object storage (a crash between rename and ledger commit is impossible by ordering, but a regional failover can split them), re-materialize from the worker staging cache rather than recomputing.
7. Refresh topology on suspected desync. If a worker computed against a stale geometry, force a sync via the domain sync controller and re-run step 1 to confirm the corrected dataset version resolves.

When a heavy operation exceeds its SLO ceiling, the broker must degrade rather than fail the client — routing the payload to a simplified path (a bounding-box approximation instead of an exact polygon intersection) so a partial answer returns within budget. That health-aware degradation mirrors the Fallback Chains for Geocoding Services pattern, applied to compute instead of resolvers.

# SLA Targets & Performance Baselines

The broker carries thin acceptance overhead; the heavy work belongs to the worker pools behind it. The targets below are the budget the async tier must hold so that domain SLAs stay meaningful.

Holding these baselines relies on warm pools, spatial-locality partitioning, and node-local index caches. For disaster recovery, the job-state ledger is replicated asynchronously across availability zones; on regional failure, promote the secondary queue cluster, redirect ingress, and reconcile in-flight jobs by comparing primary and secondary ledger states — jobs marked COMPLETED but absent from storage are re-materialized from worker staging caches, and lineage is verified by replaying audit logs through a dedicated reconciliation pipeline.

# Governance & Compliance Notes

Async execution decisions are governance decisions, so the broker treats them as auditable artifacts. Worker pool manifests, ABAC policies, and SLO ceilings are committed to source control as policy-as-code, reviewed, and promoted only after a CI pipeline validates idempotency replay behaviour and simulates SLO-breach degradation. Every job carries a contract fingerprint and a data-classification claim, so a result object can always be traced back to the governed data product it derived from — the catalog linkage that Metadata Cataloging for Raster/Vector records on the source side.

Audit requirements are concrete. The ledger captures every state transition, idempotency-cache hit, and policy denial in append-only, domain-partitioned streams, and workers inject W3C Trace-Context plus OpenTelemetry baggage so each compute decision correlates with downstream spatial-engine logs; instrumentation follows the OpenTelemetry specification. Where jurisdictional constraints apply — data-residency rules binding a tenant’s vector features to a region — the broker enforces them at admission, refusing to dispatch a job whose bbox or spatial_scope falls outside the permitted extent, and pinning its worker pool to an in-region subnet. By keeping state transitions idempotent, routing boundaries cryptographic, and degradation paths deterministic, platform teams can safely lift heavy spatial compute off the synchronous API surface while preserving domain data sovereignty and predictable SLAs.

# Related

• Up to the parent: Federated Ownership & Routing Architecture
• API Gateway Mapping for GIS Services — the ingress contract that brokers these async routes
• Schema Contracts for Vector/Tile Data — the validated payloads workers consume
• Domain Sync Protocols for Spatial Data — how workers avoid stale or partial geometries
• Cross-Domain Routing Strategies — deterministic fan-out across domains
• Optimizing async execution for spatial joins — partition-aware join strategies behind this broker