ANN Index Staleness Runbook

Use this runbook when ANN index freshness degrades and retrieval quality is at risk.

When To Use

• ANN index age exceeds expected refresh interval.
• Retrieval quality drops with stale or missing embedding coverage.
• Alert indicates stale FAISS index age.

Signals

• RecSysStaleAnnIndex
• Secondary symptoms: higher fallback rates, lower retrieval diversity/coverage.

Immediate Actions (First 10 Minutes)

1. Freeze risky rollout steps until index freshness is restored.
2. Confirm current model run ID and index artifact timestamps.
3. Validate that embeddings for the current model are available.

Rebuild Procedure

1. Rebuild from the intended run:

uv run python -m scripts.backfill_ann \
  --run_id <TARGET_RUN_ID> \
  --output data/ann/index.faiss \
  --backup_dir data/ann/backups

2. Verify metadata and file timestamps for data/ann/index.faiss and sidecar metadata.
3. Restart serving workers/pods if index is loaded in-memory.

Fallback Procedure

1. If rebuild fails, roll back to last known good run ID:

uv run python -m scripts.backfill_ann \
  --run_id <PREVIOUS_GOOD_RUN_ID> \
  --output data/ann/index.faiss \
  --backup_dir data/ann/backups

2. Keep rollout paused until freshness and retrieval metrics recover.

Validation Checklist

1. ANN age metric returns below threshold.
2. No error spikes from ANN lookup path.
3. Fallback and retrieval-quality indicators return to baseline.

Related Runbooks

• Model Rollback Runbook
• Personalization Queue Lag Runbook