Operations Runbooks

This folder contains incident response runbooks for recommendation serving and personalization operations.

Runbook Index

• Model Rollback Runbook
  • Use for canary guardrail breaches, ranker failures, or bad model promotions.
• Personalization Queue Lag Runbook
  • Use when pending refresh age, queue depth, or worker failure alerts fire.
• ANN Index Staleness Runbook
  • Use when ANN index age exceeds threshold or retrieval quality degrades due to stale vectors.
• Incident Drill Checklist
  • Use for weekly/monthly operational readiness drills.
• Canary Dashboard Spec
  • Use to build/validate required rollout guardrail dashboards.
• Serving API V2 Performance Runbook
  • Use after staging deployments to validate functional API flows, latency gates, and bounded load.
• Rollbar Operations Runbook
  • Use to configure Rollbar tokens, environment gating, and safe test errors.
• Canonical Ingestion Canary Dashboard Spec
  • Use to validate Pegasus -> Tiger -> hh-lion event canary health in engineering and production.
• Retrain and Reindex Playbook
  • Use for scheduled refreshes and post-incident model/index regeneration.
• Legacy ClickHouse Table Cleanup
  • Use to remove decommissioned v1 feature-store tables and unused ClickHouse scaffolding tables.

Primary Alert Mappings

• RecSysCanaryLatencyAutoRollback -> model_rollback_runbook.md
• RecSysCanaryFallbackAutoRollback -> model_rollback_runbook.md
• RecSysCanaryPendingTimeoutAutoRollback -> personalization_queue_lag_runbook.md
• RecSysCanaryFeatureAvailabilityAutoRollback -> model_rollback_runbook.md
• RecSysPersonalizationQueueLagHigh -> personalization_queue_lag_runbook.md
• RecSysPendingRefreshAgeHigh -> personalization_queue_lag_runbook.md
• RecSysWorkerFailureRateHigh -> personalization_queue_lag_runbook.md
• RecSysStaleAnnIndex -> ann_index_staleness_runbook.md

Release Freeze

• Release Readiness Checklist