Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Model Rollback Runbook

Use this runbook when a newly promoted model degrades serving reliability or quality.

When To Use

  • Canary guardrail breach sustained for 10 minutes.
  • Elevated ranker fallback or ranker decision failure alerts.
  • Significant quality regression from business or experimentation dashboards.

Signals

  • RecSysCanaryLatencyAutoRollback
  • RecSysCanaryFallbackAutoRollback
  • RecSysCanaryFeatureAvailabilityAutoRollback
  • RecSysRankerDecisionFailureRateHigh

Immediate Actions (First 10 Minutes)

  1. Freeze rollout progression immediately.
  2. Route traffic back to last known good percentage (or 0% canary).
  3. Capture snapshots for:
    • hh_lion_homepage_decision_events_total
    • hh_lion_ranker_decision_events_total
    • hh_lion_request_latency_seconds
  4. Record incident start time and suspect model run/version.

Rollback Procedure

Preferred path: use the dedicated GitHub Actions rollback workflow.

Official rollback workflow

Workflow:

  • .github/workflows/rollback-production-model.yml

Inputs:

  • rollback_model_version
  • model_name (defaults to homepage_two_tower)

The workflow will:

  1. resolve the exact registered rollback target version
  2. verify ANN lineage matches that rollback target
  3. move the production alias back to the requested version
  4. re-verify ANN lineage on @production
  5. verify MLflow alias, ANN lineage, and the live API /v2/model/info all resolve the expected rollback run/version

Manual fallback (emergency-only)

If GitHub Actions is unavailable, you can perform the same rollback logic manually against in-cluster MLflow:

uv run python -m scripts.rollback_model \
  --model-name homepage_two_tower \
  --rollback-model-version <PREVIOUS_GOOD_VERSION> \
  --execute

Then verify:

uv run python scripts/check_ann_lineage.py \
  --model-name homepage_two_tower \
  --alias production

and confirm the live API resolves the rollback target via /v2/model/info.

When using the official workflow, prefer its final summary as the release-proof artifact because it now checks registry alias, ANN lineage, and live serving together.

Validation Checklist

  1. /health/ready is healthy.
  2. Fallback ratio returns to baseline.
  3. p95 latency returns below canary threshold.
  4. No sustained primary_failed|fallback_failed spikes.

Escalation

  • If rollback fails, force traffic to stable legacy deployment target and page platform on-call.
  • Open a post-incident review with model owner, backend owner, and on-call responder.