# Tuning Redis Idempotency Cache TTL for Spatial Jobs

A retried or rebalanced spatial job that finds no dedup record re-runs an expensive ST_Intersection or tile-build it has already paid for — but a dedup record that lingers forever quietly leaks memory until the Redis node evicts live keys under pressure. This operation, part of Async Execution for Heavy Spatial Queries within the Federated Ownership & Routing Architecture, sizes the TTL on the idempotency store keyed by SHA-256(bbox+crs+params) so it comfortably covers the retry and fallback windows without hoarding stale fingerprints. It shares the same x-idempotency-key convention exercised in Building Fallback Chains for Routing APIs, so a job started synchronously and a job replayed asynchronously collapse onto one key.

Figure — the idempotency record must outlive the job’s worst-case duration plus the retry and fallback windows, yet expire before it becomes dead memory.

# Prerequisites

# Step-by-Step Implementation

Every job computes its fingerprint the same way, writes the record atomically, and lets Redis expire it. The TTL is derived, not hard-coded by taste: it is the p99 job duration plus the full retry ladder plus the fallback replay window plus a safety margin.

# 1. Normalize inputs and derive the fingerprint

The hash must be stable across producers, so normalize the bounding box precision, uppercase the CRS token, and sort the parameter map before hashing. An unsorted params dict silently produces two keys for one logical job.

import hashlib, json

def idem_key(bbox, crs, params):
    # bbox rounded to 6 dp (~0.1 m at the equator); crs and params canonical.
    norm = {
        "bbox": [round(float(c), 6) for c in bbox],   # EPSG:4326 lon/lat
        "crs": crs.upper(),                            # e.g. EPSG:4326
        "params": {k: params[k] for k in sorted(params)},
    }
    digest = hashlib.sha256(
        json.dumps(norm, separators=(",", ":"), sort_keys=True).encode()
    ).hexdigest()
    return f"idem:{digest}"

Verify two logically identical jobs collapse to one key:

python3 -c "from job import idem_key; \
print(idem_key([-74.5,40.1,-73.9,40.9],'epsg:4326',{'res':'10m','op':'intersects'})); \
print(idem_key([-74.500000,40.100000,-73.900000,40.900000],'EPSG:4326',{'op':'intersects','res':'10m'}))"
# both lines must print the identical idem:<hash>

# 2. Derive the TTL from measured job duration

Bind the TTL to the observed p99, never a round number pulled from the air. The retry ladder here is three attempts of exponential backoff capped at 30s; the fallback window matches the secondary tier’s replay budget.

import os

def derive_ttl():
    p99 = int(os.environ["JOB_P99_SECONDS"])   # measured, e.g. 900
    retry_window = 2 + 4 + 8 + 16 + 30          # backoff ladder, ~60s
    fallback_window = 300                        # secondary replay budget
    safety_margin = int(0.25 * (p99 + retry_window + fallback_window))
    return p99 + retry_window + fallback_window + safety_margin

Verify the derived value matches the deployed env var:

JOB_P99_SECONDS=900 python3 -c "from job import derive_ttl; print(derive_ttl())"
# compare against IDEMPOTENCY_TTL_SECONDS in the worker manifest

# 3. Claim the job atomically with SET NX EX

A single SET key value NX EX ttl is the whole idempotency guarantee: NX makes the write win only if no fingerprint exists, EX arms expiry in the same round trip so there is never a window where the key exists without a TTL.

import redis, os
r = redis.from_url(os.environ["REDIS_URL"])

def claim(key: str, trace_id: str, ttl: int) -> bool:
    # Returns True if THIS worker owns the job; False if already claimed.
    return bool(r.set(key, trace_id, nx=True, ex=ttl))

Verify a claimed key carries a bounded TTL — never -1 (no expiry):

redis-cli -u "$REDIS_URL" TTL "idem:$HASH"
# expect a positive integer close to IDEMPOTENCY_TTL_SECONDS; -1 is a bug

# 4. Choose fixed TTL over sliding for dedup correctness

Do not refresh the TTL on read. A sliding TTL that renews on every retry can keep a hot fingerprint alive indefinitely and mask a genuinely stuck job. Fixed expiry from the moment of claim gives a deterministic dedup horizon.

def owns(key: str) -> bool:
    # Read-only check; deliberately does NOT reset the TTL (no sliding window).
    return r.exists(key) == 1

Verify the TTL decreases monotonically across two reads:

redis-cli -u "$REDIS_URL" TTL "idem:$HASH"; sleep 5; \
redis-cli -u "$REDIS_URL" TTL "idem:$HASH"
# second value must be ~5 lower than the first

# 5. Bound total memory with volatile-ttl eviction

Because every dedup key has a TTL, volatile-ttl lets Redis shed the closest-to-expiry fingerprints first if it ever approaches maxmemory, protecting live claims. Confirm the used-memory ceiling holds under load.

redis-cli -u "$REDIS_URL" CONFIG SET maxmemory 512mb
redis-cli -u "$REDIS_URL" CONFIG SET maxmemory-policy volatile-ttl
redis-cli -u "$REDIS_URL" INFO memory | grep -E 'used_memory_human|maxmemory_human|evicted_keys'

A steadily climbing evicted_keys under a fixed job rate means the TTL is too long for the throughput — shorten it or raise maxmemory.

# Configuration Reference

# Common Failure Modes & Fixes

• Symptom: the same heavy join runs twice under retry. Root cause: params was hashed unsorted, so a retry produced a second key. Fix: sort and canonicalize params before hashing; add the idem_key equality assertion from Step 1 to CI.
• Symptom: redis-cli TTL returns -1. Root cause: the key was written with SET then a separate EXPIRE that never ran, or EX was dropped. Fix: always use one SET ... NX EX; audit with redis-cli TTL and alert on any -1 in the idem: namespace.
• Symptom: evicted_keys climbs and live jobs re-run. Root cause: TTL too long for the job arrival rate, so the working set exceeds maxmemory. Fix: recompute TTL from the current p99 and raise maxmemory, or shorten the fallback window if the secondary tier no longer replays that far back.
• Symptom: a stuck job never re-runs even after it truly failed. Root cause: a sliding TTL kept the fingerprint alive past the real job lifetime. Fix: switch to fixed TTL; never refresh on read, as shown in Step 4.
• Symptom: dedup misses during a Redis failover. Root cause: the claim landed on a replica that had not yet received the write. Fix: issue claims against the primary with WAIT 1 100 for the critical path, and treat a missed dedup as a retry, not a duplicate materialization — the downstream ON CONFLICT write is the second line of defense.

# FAQ

# Related

• Async Execution for Heavy Spatial Queries — parent reference for the async worker and dedup pattern.
• Configuring Celery Queues for Spatial Workloads — the queues whose tasks write these idempotency keys.
• Building Fallback Chains for Routing APIs — the fallback hops that replay against this same store.
• Optimizing Async Execution for Spatial Joins — the ON CONFLICT write that backstops a dedup miss.
• Federated Ownership & Routing Architecture — up one level to the routing architecture overview.