Canary Rollout and Rollback Runbook

This runbook defines the production response flow for homepage recommendation canary rollouts.

Related runbooks:

• Operations Runbook Index
• Model Rollback Runbook
• Personalization Queue Lag Runbook
• ANN Index Staleness Runbook
• Canary Dashboard Spec
• Retrain and Reindex Playbook

Scope

• Applies to /v2/homepage and /v2/homepage/refresh traffic.
• Uses alerts defined in k8s/shared/prometheus-alerts.yaml.
• Supports automatic and manual rollback decisions.

Guardrails and Triggers

• RecSysCanaryLatencyAutoRollback
  • Trigger: homepage p95 latency > 1.0s for 10m.
• RecSysCanaryFallbackAutoRollback
  • Trigger: homepage fallback ratio > 10% for 10m.
• RecSysCanaryPendingTimeoutAutoRollback
  • Trigger: refresh timeout_terminalized ratio > 2% for 10m.
• RecSysCanaryFeatureAvailabilityAutoRollback
  • Trigger: member feature availability < 85% for 10m.
• RecSysRankerDecisionFailureRateHigh
  • Trigger: ranker primary_failed|fallback_failed ratio > 5% for 10m.

Immediate Actions (First 10 Minutes)

1. Freeze rollout progression immediately.
2. Capture state from dashboards:
   • hh_lion_homepage_decision_events_total
   • hh_lion_refresh_outcome_events_total
   • hh_lion_ranker_decision_events_total
   • hh_lion_personalization_queue_depth
   • hh_lion_personalization_worker_events_total
3. Check readiness and runtime health:
   • curl -s http://<serving-host>/health/ready | jq
4. If any critical auto-rollback guardrail remains breached for 10 minutes, roll back canary.

Manual Rollback Procedure

1. Reduce traffic to last stable percentage (or 0% canary) in ingress/service routing.
2. Restore the last known-good registered model version through the reviewed rollback path:

uv run python -m scripts.rollback_model \
  --model-name homepage_two_tower \
  --rollback-model-version <PREVIOUS_GOOD_VERSION> \
  --execute

3. Rebuild ANN index from the same known-good run if the index artifact must also be restored:

uv run python -m scripts.backfill_ann \
  --run_id <PREVIOUS_GOOD_RUN_ID> \
  --output data/ann/index.faiss \
  --backup_dir data/ann/backups

4. Restart serving pods/workers if model or index artifacts are cached in-memory.

Triage Decision Tree

• High latency + queue lag:
  • Check worker count/capacity and Redis health first.
• High fallback + ranker failures:
  • Check CatBoost model availability and ranker feature completeness.
• Refresh timeout-terminalized spikes:
  • Check worker failure rate and pending age growth.
• Feature availability drop:
  • Check Feast/feature store pipeline and source freshness before resuming rollout.

Exit Criteria Before Resuming Canary

1. All critical guardrails return to healthy for at least 30 minutes.
2. /health/ready reports ready and personalization runtime checks are not in error.
3. Fallback and timeout rates return to baseline (pre-incident window).
4. Incident timeline, root cause, and remediation are documented.

CatBoost Ranker Expansion Gate

Before expanding a CatBoost ranker canary beyond the current traffic step:

1. Export the CatBoost ranker business-metrics snapshot:

uv run python scripts/export_catboost_ranker_business_metrics_snapshot.py \
  --output artifacts/release/ranker-business-metrics.json

2. Validate the snapshot:

uv run python scripts/release_readiness.py \
  --skip-default-checks \
  --catboost-ranker-business-metrics-snapshot artifacts/release/ranker-business-metrics.json

3. Continue only when the gate passes. The default gate requires both candidate and baseline variants to have at least 1,000 top-slot impressions and 1,000 requests, at least 10 successes and 10 failures per variant for each CTR/CVR confidence calculation, candidate top-slot CTR point lift >= 0.001, candidate booking CVR point lift >= 0.001, positive 95% lower-confidence-bound lift for both metrics, and candidate strategy-match ratio >= 0.95.

For manual deploy or release workflows that expand CatBoost ranker traffic, set require_catboost_ranker_business_metrics=true and pass catboost_ranker_business_metrics_snapshot=artifacts/release/ranker-business-metrics.json. The workflow readiness job will fail before build/release if the required snapshot path is omitted. For promote-production-model runs where the resolved candidate is a CatBoost ranker by model name, model-version tag, or source-run tag, pass the same catboost_ranker_business_metrics_snapshot; promotion fails before alias movement if the snapshot is missing or the gate fails.

By default, the exporter evaluates a mature 24-hour exposure window ending 24 hours before the label-event cutoff and attributes clicks/bookings during the following 24 hours. The release gate requires source=clickhouse exporter metadata and collected_at after the label-event cutoff; do not expand on fresh exposures whose CTR/CVR attribution window has not fully elapsed.

The exporter attributes candidate metrics only to candidate_catboost_rerank, selected_items_catboost_rerank, and section_selected_items_catboost_rerank by default. Repeat --candidate-strategy-id when validating a canary that intentionally uses a renamed CatBoost strategy.

Use scripts/monitoring/homepage_ranking_guardrails.sql for manual inspection of the same ClickHouse metric keys.