# Best practices for spatial metadata catalogs

In a domain-driven geospatial data mesh, catalog ingestion failures rarely originate from storage capacity or network latency — they stem from unvalidated spatial metadata propagation. A single misaligned coordinate reference system (CRS) or inverted bounding box in a raster or vector asset cascades into cross-domain spatial join failures, directly violating product discoverability SLAs. This page is a focused operational procedure for making catalog registration deterministic: contract enforcement at the ingestion gate, idempotent normalization to EPSG:4326, structured diagnostic logging, and zero-downtime rollback. It sits under the Metadata Cataloging for Raster/Vector reference within the broader Geospatial Data Mesh Fundamentals reference set, and it assumes the ownership and SLO model established in Product Thinking for GIS Datasets. Get the validation gate and idempotency semantics right and a non-conforming asset is rejected before it can pollute the discovery layer.

# Prerequisites

# Step-by-Step Implementation

Validation is enforced at the ingestion boundary, never post-hoc inside the discovery layer, so a malformed asset is shed before any consumer can resolve it. Each step below is verifiable with a diagnostic command before you proceed to the next.

# 1. Enforce the ingestion contract at the catalog gate

Traditional monolithic GIS architectures rely on centralized metadata silos where validation happens after registration. A federated mesh shifts ownership to domain teams, so catalog registration must reject non-conforming assets at the boundary. The spatial-extent-validator config below codifies the explicit domain boundaries required by Scoping Rules for Spatial Products: only assets with verifiable extents, authoritative CRS declarations, and explicit ownership tags pass the gate.

apiVersion: v1
kind: ConfigMap
metadata:
  name: spatial-catalog-validator-config
  namespace: data-mesh-ingest
data:
  validation_policy.yaml: |
    crs_normalization:
      target_epsg: 4326
      fallback_strategy: reject
      allowed_source_epsgs: [3857, 32610, 32611, 4269, 4326]
    extent_validation:
      enforce_wkt_polygon: true
      max_degenerate_area_sqm: 0.0
      invert_bbox_on_negative: true
      min_coverage_threshold: 0.01
    schema_enforcement:
      required_fields: ["product_id", "domain_owner", "spatial_extent", "crs_authority"]
      json_schema_path: "/schemas/spatial-catalog-v2.schema.json"

Verify the config is mounted and parseable before the worker starts:

kubectl -n data-mesh-ingest get configmap spatial-catalog-validator-config \
  -o jsonpath='{.data.validation_policy\.yaml}' | \
  python3 -c 'import sys,yaml; yaml.safe_load(sys.stdin); print("policy OK")'

# 2. Run the idempotent extraction & normalization pipeline

The ingestion worker computes a unified EPSG:4326 WKT envelope, validates it against the catalog schema, and commits atomically. The trap-based workspace teardown prevents resource leaks, and the min/max coordinate ordering guards against inverted bounding boxes — the single most common cause of empty cross-domain join results.

#!/usr/bin/env bash
set -euo pipefail

# Idempotent workspace setup
WORK_DIR=$(mktemp -d /tmp/spatial-validate-XXXXXX)
trap 'rm -rf "$WORK_DIR"' EXIT

SOURCE_ASSET_PATH="${1:?Missing source asset path}"
OUTPUT_CATALOG_ENTRY="${2:?Missing output catalog entry path}"
SCHEMA_PATH="/schemas/spatial-catalog-v2.schema.json"

# 1. Extract raw spatial extent (vector/raster agnostic)
ogrinfo -json -al "${SOURCE_ASSET_PATH}" 2>/dev/null | \
  jq -r '.layers[0].geometryFields[0].extent' > "${WORK_DIR}/raw_extent.json" || \
  { echo "FATAL: Failed to parse OGR extent"; exit 1; }

# 2. Normalize CRS to EPSG:4326 and compute WKT envelope
#    For raster assets: reproject a copy in memory, then read back extent
gdalwarp -t_srs EPSG:4326 -of GTiff \
  "${SOURCE_ASSET_PATH}" /vsimem/normalized.tif 2>/dev/null

EXTENT_RAW=$(gdalinfo -json /vsimem/normalized.tif | \
  jq -r '.cornerCoordinates | "\(.upperLeft[0]),\(.lowerRight[1]),\(.lowerRight[0]),\(.upperLeft[1])"')

# Clean up virtual memory filesystem
rm -f /vsimem/normalized.tif 2>/dev/null || true

# 3. Compute WKT polygon with explicit coordinate ordering using shapely
python3 - "${EXTENT_RAW}" > "${WORK_DIR}/normalized_extent.wkt" <<'PYEOF'
import sys
from shapely.geometry import box

coords = [float(x) for x in sys.argv[1].split(',')]
# Enforce min/max ordering to prevent inverted bboxes
xmin, ymin = min(coords[0], coords[2]), min(coords[1], coords[3])
xmax, ymax = max(coords[0], coords[2]), max(coords[1], coords[3])
poly = box(xmin, ymin, xmax, ymax)
if poly.is_empty or poly.area == 0:
    sys.exit('ERROR: Degenerate spatial extent detected')
print(poly.wkt)
PYEOF

# 4. Schema validation using jsonschema
python3 -m jsonschema \
  --instance "${WORK_DIR}/normalized_extent.wkt" \
  "${SCHEMA_PATH}" > "${WORK_DIR}/validated_entry.json" 2>&1 || {
  echo "FATAL: Schema validation failed"
  cat "${WORK_DIR}/validated_entry.json"
  exit 1
}

# Atomic swap prevents partial catalog states
mv "${WORK_DIR}/validated_entry.json" "${OUTPUT_CATALOG_ENTRY}"
echo "SUCCESS: Catalog entry committed idempotently"

This sequence guarantees repeatable execution regardless of retry count. For the band-level raster metadata that must accompany the extent contract, align with the parent Metadata Cataloging for Raster/Vector reference.

Verify idempotency by running the worker twice and confirming the committed entry is byte-identical:

./validate.sh asset.tif out_a.json && ./validate.sh asset.tif out_b.json && \
  diff -q out_a.json out_b.json && echo "idempotent: identical entry"

# 3. Wire structured diagnostic logging

When ingestion fails or downstream spatial queries return empty result sets, the failure vector must be isolatable from logs alone. Emit every pipeline log as structured JSON carrying trace_id, asset_hash, and validation_stage fields. Detect CRS drift by comparing the detected projection against the allowlist:

gdalsrsinfo -o proj "${SOURCE_ASSET_PATH}"   # compare against allowed_source_epsgs

Verify that rejected assets are queryable by stage — this query surfaces every CRS reject in one pass:

grep -E '"validation_stage":"crs_normalization","status":"reject"' /var/log/ingest/*.json | \
  jq -r '.asset_path + " -> " + .detected_crs + " (expected: EPSG:4326)"'

# 4. Configure zero-downtime rollback & state recovery

Catalog corruption from bad metadata must be reversible without service interruption. Maintain a rolling 7-day snapshot of the catalog index in an immutable object store and recover with a single database transaction. Versioning of catalog entries follows the discipline in Spatial Product Lifecycle Management: semantic versions where minor bumps reflect extent corrections and major bumps indicate CRS migrations.

#!/usr/bin/env bash
set -euo pipefail
SNAPSHOT_URI="${1:?Missing snapshot URI}"
CATALOG_DB="${2:?Missing catalog database URI}"

# 1. Verify snapshot integrity
aws s3 cp "${SNAPSHOT_URI}/catalog-checksum.sha256" /tmp/checksum
sha256sum -c /tmp/checksum || { echo "FATAL: Snapshot checksum mismatch"; exit 1; }

# 2. Download snapshot to local temp file
aws s3 cp "${SNAPSHOT_URI}/catalog_snapshot.csv" /tmp/catalog_snapshot.csv

# 3. Atomic swap via database transaction
psql "${CATALOG_DB}" <<'SQL'
BEGIN;
  CREATE TEMP TABLE catalog_backup AS TABLE spatial_catalog;
  TRUNCATE TABLE spatial_catalog;
  \copy spatial_catalog FROM '/tmp/catalog_snapshot.csv' CSV HEADER
  UPDATE spatial_catalog SET last_rollback_ts = NOW();
COMMIT;
SQL

# 4. Invalidate downstream cache
curl -sf -X POST http://mesh-cache.internal/v1/invalidate \
  -H "Content-Type: application/json" \
  -d '{"scope":"spatial_catalog"}'
echo "ROLLBACK_COMPLETE: Catalog restored to snapshot state"

The transaction guarantees consumers never observe a partial catalog state. Verify the restore by confirming row counts match the snapshot header:

psql "${CATALOG_DB}" -tAc 'SELECT count(*) FROM spatial_catalog' && \
  tail -n +2 /tmp/catalog_snapshot.csv | wc -l

Post-rollback, trigger a full re-validation sweep through Step 2 to quarantine any asset that failed the original ingestion but was cached by consumers.

# Configuration Reference

# Common Failure Modes & Fixes

Empty cross-domain join despite valid-looking assets. Symptom: a spatial join returns zero rows even though both products registered successfully. Root cause: an inverted bounding box (xmax < xmin) slipped through because invert_bbox_on_negative was disabled. Fix: set invert_bbox_on_negative: true and re-run Step 2; the shapely min/max ordering normalizes the envelope on commit.

CRS_DRIFT_DETECTED on a previously-passing producer. Symptom: assets from one domain begin rejecting at the crs_normalization stage. Root cause: the producer changed its source projection to an EPSG absent from allowed_source_epsgs. Fix: either reproject upstream to an allowlisted CRS, or — if the new projection is enterprise-approved — add it to the allowlist and bump schema_version; never silently widen the allowlist without a contract review.

Degenerate extent halts the pipeline. Symptom: ERROR: Degenerate spatial extent detected aborts ingestion. Root cause: the source geometry is a zero-area point or collapsed line, which breaks R-tree indexing. Fix: inspect with ogrinfo -al -so asset.gpkg; buffer the geometry upstream or exclude it from the product extent — do not lower max_degenerate_area_sqm to force it through.

Forked product_id after a retry storm. Symptom: two catalog rows share one logical product. Root cause: a non-idempotent commit path created duplicates under retry. Fix: enforce the atomic mv swap from Step 2, dedupe on product_id, and bump schema_version so the dedup is auditable.

Consumers serve stale geometry after a rollback. Symptom: clients return pre-rollback extents. Root cause: the downstream cache was not invalidated. Fix: confirm the POST /v1/invalidate call in Step 4 returned 2xx; re-issue with scope: spatial_catalog and verify the cache cleared before re-opening the discovery layer.

# FAQ

# Related

• Metadata Cataloging for Raster/Vector — parent reference for raster/vector catalog registration
• Spatial Domain Boundary Design — the boundary contracts and boundary_priority the catalog enforces
• Spatial Product Lifecycle Management — versioning, deprecation, and the 90-day retention window
• Scoping Rules for Spatial Products — what an asset must declare to pass the ingestion gate