Tiering Cold Raster Storage Without Breaking SLAs

Raster archives are the largest and least-read thing in most spatial estates, which makes cold tiering an obvious saving and a quiet way to breach a latency commitment. A Cloud Optimized GeoTIFF in an archival tier does not merely respond slowly — its range-read access pattern, the property that made it cloud-native in the first place, stops working entirely when the object must be restored before any byte can be read. This guide decides what is safe to tier, measures the retrieval latency against the product’s own SLO before moving anything, and keeps the tier visible to consumers. It applies the storage lever from Cost Observability for Spatial Workloads within Spatial Pipeline Orchestration & Observability.

Prerequisites

Requirement Value / Assumption Notes
Tools Object store CLI, gdalinfo (GDAL ≥ 3.6), curl, Prometheus Retrieval latency must be measured, not assumed
Access telemetry Per-object read history, ≥ 90 days Tiering without it is guessing
SLO The product’s declared latency commitment The bound the tier must respect
Format COG with internal tiling and overviews Range reads are the pattern at risk
Access roles domain-owner decides; platform-engineer applies Tiering is a product decision
Environment BUCKET, PRODUCT, PROM Exported before running

Step-by-Step Implementation

1. Separate never-read from rarely-read

These need different treatment, and the distinction is the whole decision. Never-read objects can go to the coldest tier available; rarely-read objects are the ones where a restore latency lands on a real consumer.

What is safe to tier, decided by who reads it rather than how oftenFour content classes against their reader, their tolerance for retrieval latency, and the verdict. Superseded scenes are read by auditors and reproducers who tolerate seconds, and are the clean tiering case. Current scenes for an active product are read by live consumers with a latency expectation, however rarely an individual tile is touched, and must stay hot. Pipeline intermediates are read by nothing after their retention window and should expire rather than tier. Published products of any age must never expire, because reproducibility requires they remain addressable.Read byToleratesVerdictSuperseded scenesauditors, reproducerssecondstier freelyCurrent sceneslive consumersmillisecondskeep hotPipeline intermediatesnothing, after 30d—expireAny published versionreproducerssecondsnever expire

bash
# Read history per object over 90 days, bucketed. The two tails behave differently.
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=count_values("reads", sum by (object) (increase(raster_object_reads_total{product="'"$PRODUCT"'"}[90d])))' \
  | jq -r '.data.result[] | "\(.metric.reads) reads: \(.value[1]) object(s)"' | sort -n | head -20

Verify the shape before choosing a policy. A distribution where 80% of objects were never read and the rest are read weekly is a clean tiering case; a smooth distribution is not:

bash
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "raster/${PRODUCT}/" \
  --query 'sum(Contents[].Size)' --output text | numfmt --to=iec

2. Measure retrieval latency against the SLO — before moving anything

Every tier’s published retrieval time is a guarantee about the first byte after restore, not about a range read. Measure what the access pattern actually experiences.

First-byte latency for a COG header range read, by storage classMeasured first-byte latency in milliseconds for a sixteen-kilobyte header range read against four storage classes, with a three hundred millisecond product SLO drawn as a dashed line. Standard storage answers in about forty milliseconds and infrequent-access in about sixty, both comfortably inside the SLO. Instant-retrieval archival answers in around 380 milliseconds, which breaches it. Deep archive does not answer at all without a restore, which does not merely slow the range-read pattern down but breaks it entirely.STANDARD40 msSTANDARD_IA60 msGLACIER_IR380 msbreaches this SLODEEP_ARCHIVE640 msrange reads breakproduct SLO

bash
#!/usr/bin/env bash
# tier_probe.sh — measure a real range read from each candidate tier.
set -euo pipefail
OBJ="${1:?object key}"

for tier in STANDARD STANDARD_IA GLACIER_IR; do
  cp_key="probe/${tier}/$(basename "$OBJ")"
  aws s3 cp "s3://${BUCKET}/${OBJ}" "s3://${BUCKET}/${cp_key}" \
      --storage-class "$tier" --quiet

  # The COG header, then one overview tile — the actual access pattern.
  t0=$(date +%s.%N)
  aws s3api get-object --bucket "$BUCKET" --key "$cp_key" \
      --range "bytes=0-16383" /dev/null >/dev/null
  t1=$(date +%s.%N)
  printf '%-14s header range-read: %.3fs\n' "$tier" "$(echo "$t1 - $t0" | bc)"

  aws s3 rm "s3://${BUCKET}/${cp_key}" --quiet
done

Verify the measured latency against the product’s own commitment rather than against the tier’s marketing figure:

bash
SLO_MS=$(curl -sS "$CATALOG_API/products/$PRODUCT" | jq -r '.slo.latency_p95_ms')
echo "product SLO p95: ${SLO_MS}ms — any tier above this is not a candidate"

3. Apply the policy by prefix, with the products’ SLOs respected

Lifecycle rules are prefix-based, so the layout has to separate tierable content from the rest. Retrofitting that is a rewrite, which is why it belongs in the provisioning layout.

The order the tiering decision is made, and the two checks that stop itA candidate prefix passes four checks before any lifecycle rule is applied. Read history over ninety days establishes that it is genuinely cold. Object size must exceed the per-object overhead threshold, or tiering costs more than it saves. Measured first-byte latency must sit inside the product SLO. The minimum storage duration of the target class must be shorter than how long objects will actually stay. Only then is the rule applied. Two checks drop onto a rail leading to leaving the prefix hot, which is the correct outcome far more often than teams expect.Cold in 90dread historySize > 128KBper-object overheadLatency < SLOmeasured, not publishedRule appliedby prefixcoldlarge enoughwithin SLOmany small objectsbreaches the SLOLeave it hotthe right answer, often

json
{
  "Rules": [
    {
      "ID": "archive-superseded-scenes",
      "Filter": { "Prefix": "raster/archive/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 180, "StorageClass": "GLACIER_IR" }
      ]
    },
    {
      "ID": "keep-current-scenes-hot",
      "Filter": { "Prefix": "raster/current/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 90, "StorageClass": "STANDARD_IA" }
      ]
    },
    {
      "ID": "expire-pipeline-intermediates",
      "Filter": { "Prefix": "intermediate/" },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    }
  ]
}

Note what is absent: no expiration on any published raster. A published version stays addressable because reproducibility requires it, and tiering is about where it lives rather than whether it exists.

Verify the rules match the prefixes that actually exist — a misprefixed rule silently applies to nothing:

bash
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" \
  | jq -r '.Rules[] | "\(.ID): \(.Filter.Prefix)"' \
  | while read -r line; do
      prefix="${line#*: }"
      n=$(aws s3 ls "s3://${BUCKET}/${prefix}" --recursive --summarize \
          | awk '/Total Objects/{print $3}')
      echo "${line} -> ${n:-0} object(s)"
    done

4. Record the tier in the catalog so consumers can see it

A consumer who does not know an object is cold cannot plan around it, and will experience the restore as an unexplained outage.

bash
# The tier is a property of the artifact, published like any other.
curl -sS -X PATCH "$CATALOG_API/products/$PRODUCT/artifacts" \
  -H 'content-type: application/json' \
  -d '{"storage_tier": "GLACIER_IR",
       "first_byte_p95_ms": 380,
       "note": "archived scenes; current scenes remain in STANDARD"}'

# And it is monitored, because a tier change alters the latency distribution.
curl -sS "$PROM/api/v1/query" --data-urlencode \
  'query=histogram_quantile(0.95, sum(rate(raster_first_byte_seconds_bucket{product="'"$PRODUCT"'"}[30m])) by (le, storage_tier))' \
  | jq -r '.data.result[] | "\(.metric.storage_tier): \(.value[1])s"'

Configuration Reference

Setting Value Effect Risk
Archive transition 180d to instant-retrieval archival Largest saving on superseded scenes Restore latency if wrongly classified
Current transition 90d to infrequent-access Moderate saving, millisecond retrieval Per-request retrieval fee
Intermediate expiry 30d Removes result-persistence residue None if products are excluded by prefix
Published expiry None, ever Reproducibility —
Minimum object size > 128KB Below it, tiering costs more than it saves Small-object overhead
Minimum duration Respect the tier’s minimum Early deletion incurs the full minimum charge Churn costs more than storage
Catalog storage_tier Published Consumers can plan Silent latency surprise

The minimum-duration row catches more teams than the latency one. Most cold tiers bill a minimum storage period regardless of how long the object actually stays, so an object tiered and then deleted or re-tiered a week later costs the full minimum — and a policy that churns objects between tiers can cost more than leaving everything hot.

Common Failure Modes & Fixes

Range reads stop working after tiering. Root cause: an archival tier requiring an explicit restore before any byte is readable. A COG’s whole access pattern depends on reading the header first, and that read fails. Fix: use an instant-retrieval archival class for anything a consumer may range-read; deep archive is only for content nobody reads without notice.

Storage cost rises after enabling tiering. Root cause: many small objects, each paying a per-object overhead and a minimum duration that exceeds the saving. Fix: apply a minimum object size to the rule; tiering a million 40 KB objects is reliably worse than leaving them.

A product’s latency SLO breaches intermittently after tiering. Root cause: the tier was chosen from published retrieval figures rather than measured against the product’s own commitment. Fix: the probe above, run against a real object with the real access pattern, before the rule is applied.

The lifecycle rule appears to do nothing. Root cause: the prefix does not match the actual key layout — usually a leading slash or a missing segment. Fix: the verification loop above counts objects per rule prefix, which makes a zero obvious.

Consumers report slow reads and the tier is not the cause. Root cause: the object was rewritten, resetting its transition clock, so it is hot and something else is slow. Fix: check the object’s current storage class directly rather than inferring it from the rule.

FAQ

Which raster content is genuinely safe to tier?

Superseded scenes are the clean case: a scene replaced by a newer acquisition of the same tile, retained for reproducibility rather than for use. It is read rarely, its readers are auditors and reproducers rather than interactive consumers, and its retrieval latency is acceptable to both. Current scenes for an active product are the clean counter-case, however rarely an individual tile is read, because the reader is a live consumer with a latency expectation. The distinction is about who reads it, not how often.

Does tiering interfere with a product’s reproducibility commitment?

No, provided nothing expires. Tiering changes where an artifact lives and how quickly it responds; reproducibility requires only that it remains addressable and unchanged. What breaks reproducibility is an expiration rule applied to published artifacts — which is why the policy above has none, and why intermediates and products live under separate prefixes so a rule written for one cannot reach the other. A reproducer waiting a few hundred milliseconds is fine; a reproducer finding nothing is not.

How do I choose between infrequent-access and instant-retrieval archival?

By read frequency against the retrieval fee, not by storage price alone. Infrequent-access tiers charge a per-retrieval fee, so an object read weekly can cost more there than in standard storage despite the lower storage rate. Instant-retrieval archival is cheaper still to store and more expensive again to read. The crossover is computable from the object’s measured read rate, and the useful discipline is to compute it per prefix rather than adopting one tier for everything — most estates have a hot prefix where tiering loses money and a cold one where it saves a great deal.

Should the tier be visible to consumers?

Yes, as a published property of the artifact. A consumer building an interactive application needs to know that a product’s archive is cold, because it changes what they can promise their own users; a consumer running a batch reprocessing job needs to know because it changes their runtime estimate. Publishing the tier and its measured first-byte latency costs one field and converts an unexplained slow read into an expected one — and it also creates pressure in the right direction, since a consumer who genuinely needs a product hot has somewhere to say so.

Related in Cost Observability for Spatial Workloads