# Detecting Attribute Drift in Published Layers

Geometry defects announce themselves; attribute drift does not. A producer’s export tool changes a field width, a code list gains a value nobody documented, a required attribute starts arriving null in three percent of features — and every geometry check passes, the layer publishes, and consumers discover it weeks later as an unexplained gap in a join. This guide builds the detector: fingerprint a layer’s attribute shape and distribution at publish time, compare each publish against the trailing baseline, and raise a producer-facing alert when the shape moves. It implements the drift tier of Spatial Data Quality and Validation Standards within Geospatial Data Mesh Fundamentals, and complements Measuring Positional Accuracy Against Control Points, which covers the geometric half.

# Prerequisites

Drift detection needs history, which means the first five publishes of a new layer produce a baseline rather than an alert. That is correct: a layer with no history has no normal to deviate from, and alerting against a fixed expectation on day one produces noise that teaches everyone to ignore the detector.

# Step-by-Step Implementation

# 1. Fingerprint the attribute shape

The shape is the structural facts: which fields exist, their types, and whether they are nullable in practice. A change here is almost always a producer-side export change rather than a data change.

-- attr_shape.sql — the structural fingerprint of one partition's attributes.
-- Parameters: :product (text), :partition (text)
SELECT
    key                                              AS field,
    jsonb_typeof(value)                              AS observed_type,
    count(*)                                         AS present,
    count(*) FILTER (WHERE value = 'null'::jsonb)    AS nulls,
    -- Distinct-value count is the cheapest signal that a code list changed.
    count(DISTINCT value)                            AS distinct_values,
    max(length(value::text))                         AS max_len
FROM product_features f,
     LATERAL jsonb_each(f.attrs)
WHERE f.product = :'product' AND f.partition_key = :'partition'
GROUP BY key, jsonb_typeof(value)
ORDER BY key;

Verify that the field set matches the product’s declared contract before looking at any distribution:

psql --csv -f attr_shape.sql -v product="$PRODUCT_ID" -v partition="$PARTITION" \
  | cut -d, -f1 | tail -n +2 | sort -u > /tmp/observed_fields
curl -sS "$CATALOG_API/products/$PRODUCT_ID/contract" \
  | jq -r '.attributes[].name' | sort -u > /tmp/declared_fields
diff /tmp/declared_fields /tmp/observed_fields && echo "shape matches the contract"

# 2. Fingerprint the value distribution

Shape catches structural change; distribution catches the subtler case where the shape is unchanged and the content has moved — a code list that has quietly gained a category, a numeric field whose range has shifted.

# attr_fingerprint.py — a compact, comparable summary of one publish.
import csv
import hashlib
import json
import sys


def fingerprint(rows: list[dict]) -> dict:
    """One row per field. Everything here is comparable across publishes without
    retaining the data itself, which keeps history cheap and avoids duplicating
    a domain's attributes into the quality store."""
    out = {}
    for r in rows:
        total = int(r["present"])
        out[r["field"]] = {
            "type": r["observed_type"],
            "null_ratio": round(int(r["nulls"]) / total, 6) if total else 1.0,
            "distinct": int(r["distinct_values"]),
            # Cardinality ratio separates an identifier from a code list.
            "cardinality_ratio": round(int(r["distinct_values"]) / total, 6) if total else 0.0,
            "max_len": int(r["max_len"]),
        }
    digest = hashlib.sha256(
        json.dumps(sorted(out.keys()), separators=(",", ":")).encode()
    ).hexdigest()[:16]
    return {"fields": out, "shape_digest": digest}


if __name__ == "__main__":
    rows = list(csv.DictReader(open(sys.argv[1], newline="")))
    print(json.dumps(fingerprint(rows), indent=2))

Verify the fingerprint is stable across two runs over the same partition — a fingerprint that changes without the data changing is useless as a drift signal:

psql --csv -f attr_shape.sql -v product="$PRODUCT_ID" -v partition="$PARTITION" > /tmp/a.csv
psql --csv -f attr_shape.sql -v product="$PRODUCT_ID" -v partition="$PARTITION" > /tmp/b.csv
diff <(python3 attr_fingerprint.py /tmp/a.csv) <(python3 attr_fingerprint.py /tmp/b.csv) \
  && echo "fingerprint is deterministic"

# 3. Compare against the trailing baseline

The baseline is a median over recent publishes, not the previous one. Comparing against a single prior publish makes every measurement a comparison against whatever noise that publish happened to carry.

# drift_check.py — compare one fingerprint against the trailing median.
import json
import statistics
import sys

# A field's null ratio moving by more than this, in absolute terms, is drift.
NULL_RATIO_DELTA = 0.02
# A distinct-value count moving by more than this factor is drift.
DISTINCT_FACTOR = 1.5


def baseline(history: list[dict]) -> dict:
    """Trailing median per field per statistic. Median rather than mean, so one
    bad publish in the history does not move the baseline it is compared against."""
    fields = {}
    for entry in history:
        for name, stats in entry["fields"].items():
            fields.setdefault(name, {"null_ratio": [], "distinct": []})
            fields[name]["null_ratio"].append(stats["null_ratio"])
            fields[name]["distinct"].append(stats["distinct"])
    return {
        name: {k: statistics.median(v) for k, v in vals.items()}
        for name, vals in fields.items()
    }


def drift(current: dict, base: dict) -> list[str]:
    findings = []
    for name, stats in current["fields"].items():
        if name not in base:
            findings.append(f"{name}: new field, absent from the baseline")
            continue
        b = base[name]
        if abs(stats["null_ratio"] - b["null_ratio"]) > NULL_RATIO_DELTA:
            findings.append(
                f"{name}: null ratio {b['null_ratio']:.3f} -> {stats['null_ratio']:.3f}")
        if b["distinct"] and stats["distinct"] > b["distinct"] * DISTINCT_FACTOR:
            findings.append(
                f"{name}: distinct values {b['distinct']:.0f} -> {stats['distinct']}")
    for name in base:
        if name not in current["fields"]:
            findings.append(f"{name}: field disappeared")
    return findings


if __name__ == "__main__":
    current = json.load(open(sys.argv[1]))
    history = [json.loads(line) for line in open(sys.argv[2])]
    if len(history) < 5:
        print("baseline building — no drift check until 5 publishes exist")
        raise SystemExit(0)
    findings = drift(current, baseline(history))
    for f in findings:
        print(f"DRIFT: {f}")
    # Drift is producer-facing, not blocking: exit 0 so publication proceeds.
    raise SystemExit(0)

Verify the detector fires by feeding it a synthetic change:

python3 - <<'PY' > /tmp/drifted.json
import json
fp = json.load(open('/tmp/current.json'))
fp["fields"]["owner_ref"]["null_ratio"] = 0.09      # baseline near 0.00
json.dump(fp, open('/tmp/drifted.json', 'w'))
PY
python3 drift_check.py /tmp/drifted.json quality_history.jsonl   # expect a DRIFT line

# 4. Route the finding to the producing domain

Drift is a signal about an upstream source, so it belongs with the team that owns that source — not on a platform dashboard where nobody reads it.

# Findings become an annotation on the publish, and a notification to the owning team.
python3 drift_check.py /tmp/current.json quality_history.jsonl \
  | jq -Rsc '{drift: (split("\n") | map(select(length > 0)))}' \
  | curl -sS -X PATCH "$CATALOG_API/products/$PRODUCT_ID/versions/$VERSION/quality" \
      -H 'content-type: application/json' -d @-

# Configuration Reference

# Common Failure Modes & Fixes

Every publish reports drift on the same field. Root cause: a genuinely high-variance field — a timestamp, a running identifier — being compared as though it were a code list. Fix: exclude high-cardinality fields whose cardinality_ratio approaches 1.0; they carry no drift signal by construction.

Drift appears immediately after a contract change and never clears. Root cause: the baseline still contains publishes from before the change, so the median straddles two regimes. Fix: reset the baseline at a MAJOR contract version boundary. A deliberate change should not be reported as drift forever.

A field disappears with no alert. Root cause: the detector compares fields present in the current publish and never iterates the baseline. Fix: the reverse pass in drift() above; missing fields are the highest-value finding and the easiest to omit.

Null ratio jumps from 0.00 to 0.03 and the team dismisses it. Root cause: three percent looks small. It is not — it means one feature in thirty now fails any join that depends on the field. Fix: express the finding in features rather than ratio when reporting to a producer; “4,200 features now have no owner reference” lands where “0.03” does not.

The detector is deterministic locally and noisy in CI. Root cause: the fingerprint query has no ORDER BY and the aggregation is being computed over a partition still receiving writes. Fix: run the fingerprint against the immutable published artifact rather than against the live table.

# FAQ

# Related

• Spatial Data Quality and Validation Standards — the parent topic and the full indicator set
• Measuring Positional Accuracy Against Control Points — the geometric half of quality measurement
• Data Contracts for Spatial Products — the clauses a shape change may violate outright