Release Readiness Checklist

Use this checklist before full rollout (100%) or production freeze approval.

Scope

• Serving API (/v2/homepage, /v2/homepage/refresh)
• Retrieval/ranking pipelines
• Feature and personalization runtime dependencies

One-Command Gate Runner

Run the default release checks with:

uv run python scripts/release_readiness.py

Run with full test suite:

uv run python scripts/release_readiness.py --with-full-tests

Run Feast v2 release gates:

uv run python scripts/release_readiness.py --with-feature-store-v2-gates

Run Feast v2 rollout gates with capacity evidence from deployment variables and a time-based recommendation-quality artifact:

uv run python scripts/release_readiness.py \
  --only-feature-store-v2-gates \
  --feature-store-v2-capacity-from-env \
  --recommendation-quality-artifact artifacts/recommendation_quality/time/latest_vs_production.json \
  --recommendation-quality-candidate-model latest \
  --recommendation-quality-baseline production \
  --recommendation-quality-split time \
  --recommendation-quality-k 10 \
  --recommendation-results-artifact artifacts/recommendation_quality/served_results.jsonl

Run staging serving API v2 gates after deployment:

uv run python scripts/release_readiness.py \
  --skip-default-checks \
  --serving-api-base-url http://<staging-hh-lion-service> \
  --serving-api-member-user-id <known-featured-member-id> \
  --serving-api-load-requests 50 \
  --serving-api-load-concurrency 5

Run with canary metrics guardrail snapshot:

uv run python scripts/release_readiness.py --metrics-snapshot <path-to-metrics.json>

Run one-command readiness with live Prometheus collection:

uv run python scripts/release_readiness.py \
  --prometheus-url http://<prometheus-host>:9090 \
  --metrics-output artifacts/release/canary-metrics.json

Generate snapshot directly from Prometheus canary queries:

uv run python scripts/export_canary_metrics_snapshot.py \
  --prometheus-url http://<prometheus-host>:9090 \
  --output artifacts/release/canary-metrics.json

Accepted snapshot formats:

{
  "homepage_p95_latency_seconds": 0.82,
  "homepage_fallback_ratio": 0.06
}

{
  "collected_at": "2026-02-18T10:30:00Z",
  "metrics": {
    "homepage_p95_latency_seconds": 0.82,
    "homepage_fallback_ratio": 0.06
  }
}

Quality Gates

1. Static checks:
   • uv run ruff check .
   • uv run mypy recsys
2. Contract/integration tests pass for:
   • API contracts: uv run pytest --no-cov tests/contract/test_homepage_api.py
   • model signature contracts: uv run pytest --no-cov tests/contract/test_model_signatures.py
   • model promotion gates: uv run pytest --no-cov tests/unit/test_register_model_gates.py
   • alert/runbook linkage + guardrail threshold alignment checks: uv run pytest --no-cov tests/unit/test_prometheus_alert_runbooks.py
   • Grafana dashboard config sync: uv run pytest --no-cov tests/unit/test_grafana_dashboard_configmap.py
   • test spec hygiene (no stale bootstrap placeholders): uv run pytest --no-cov tests/unit/test_test_spec_hygiene.py
   • release readiness metric snapshot parsing/guardrails: uv run pytest --no-cov tests/unit/test_release_readiness.py
   • canary metrics snapshot exporter script: uv run pytest --no-cov tests/unit/test_export_canary_metrics_snapshot.py
   • CatBoost ranker business metrics exporter script: uv run pytest --no-cov tests/unit/test_export_catboost_ranker_business_metrics_snapshot.py
3. No /v1 routes are registered:
   • uv run pytest --no-cov tests/integration/test_api_endpoints.py::TestRouteVersioning::test_no_v1_routes_are_registered
4. Core /v2 integration paths pass:
   • readiness failure-mode: uv run pytest --no-cov tests/integration/test_api_endpoints.py::TestHealthEndpoint::test_readiness_returns_503_when_personalization_queue_unhealthy
   • API error security behavior: uv run pytest --no-cov tests/integration/test_api_endpoints.py::TestErrorHandling
   • guest cohort path: uv run pytest --no-cov tests/integration/test_guest_cohort.py
   • established member path: uv run pytest --no-cov tests/integration/test_established_member.py
   • progressive enhancement path: uv run pytest --no-cov tests/integration/test_progressive_enhancement.py
   • refresh ownership isolation: uv run pytest --no-cov tests/integration/test_refresh_ownership.py

Business Override Removal Rollout

1. Confirm no application producers, Kafka fixtures, or API clients emit section_type=business_override.
2. Treat scripts/migrations/026_remove_business_override_section_type.sql as a manual ClickHouse operation. Count affected production rows first, run the cleanup mutation during a maintenance window, monitor system.mutations, then apply the enum narrowing after cleanup succeeds.
3. Communicate the intentional interaction-event Avro enum break to downstream consumers before schemas generated from this branch are promoted.

Feast V2 Gates

1. Static Feast contracts pass:
   • uv run pytest --no-cov tests/contract/test_clickhouse_feature_store_v2_schema.py tests/contract/test_feast_repo_v2.py
2. Feast v2 unit and workflow guardrails pass:
   • uv run pytest --no-cov tests/unit/test_feature_store_contracts_v2.py tests/unit/test_feature_store_config.py tests/unit/test_feature_store_clients.py tests/unit/test_feature_store_publisher.py tests/unit/test_feature_store_health_v2.py tests/unit/test_feature_quality_checks_v2.py tests/unit/test_feature_store_store.py tests/unit/test_feature_jobs_v2.py tests/unit/test_feature_store_v2_operations.py tests/unit/test_validate_feature_store_v2.py
3. Feast v2 integration tests pass with Docker-backed ClickHouse/Redis or production-like local services:
   • uv run pytest --no-cov -m "integration and feature_store_v2" -q
4. Feast v2 performance regression tests and benchmark smoke gates pass:
   • uv run pytest --no-cov tests/performance/test_feature_store_online_v2.py tests/performance/test_feature_store_offline_v2.py -m performance -q
   • uv run python scripts/benchmark_feature_store_v2.py --iterations 30 --offline-rows 50000 --publication-rows 10000 --fail-on-regression --output text
5. Capacity validation runs with explicit production-like inputs:
   • uv run python scripts/release_readiness.py --skip-default-checks --feature-store-v2-capacity-from-env
6. Recommendation-quality validation runs against the time-based production baseline artifact:
   • uv run python scripts/release_readiness.py --skip-default-checks --recommendation-quality-artifact artifacts/recommendation_quality/time/latest_vs_production.json --recommendation-quality-candidate-model latest --recommendation-quality-baseline production --recommendation-quality-split time --recommendation-quality-k 10
7. Served result validation enforces zero duplicate, invalid, unavailable, and untraceable exposures:
   • uv run python scripts/release_readiness.py --skip-default-checks --recommendation-results-artifact artifacts/recommendation_quality/served_results.jsonl
8. CatBoost ranker rollout expansion requires live business-metric evidence from the canary/A/B window:
   • uv run python scripts/export_catboost_ranker_business_metrics_snapshot.py --output artifacts/release/ranker-business-metrics.json
   • uv run python scripts/release_readiness.py --skip-default-checks --catboost-ranker-business-metrics-snapshot artifacts/release/ranker-business-metrics.json
   • for manual deploy/release workflows that expand CatBoost ranker traffic, set require_catboost_ranker_business_metrics=true and provide catboost_ranker_business_metrics_snapshot=artifacts/release/ranker-business-metrics.json; the workflow fails before build/release if the required snapshot path is omitted
   • for promote-production-model runs where the resolved candidate is a CatBoost ranker by model name, model-version tag, or source-run tag, provide catboost_ranker_business_metrics_snapshot=artifacts/release/ranker-business-metrics.json; production promotion fails before alias movement if the snapshot path is omitted or the gate fails
   • defaults require at least 1,000 top-slot impressions and 1,000 requests per variant, at least 10 successes and 10 failures per variant for each CTR/CVR confidence calculation, >= 0.001 absolute point lift on top-slot CTR and booking CVR versus baseline, a positive 95% lower-confidence-bound lift for both metrics, and >= 0.95 of candidate requests served by known CatBoost strategy IDs
   • tune --catboost-ranker-min-online-z-score only with experiment-owner approval; the default 1.96 is the normal-approximation multiplier for a 95% confidence interval
   • CatBoost registry promotion also requires non-negative 95% lower-confidence-bound offline lift for NDCG, request-level top-slot click-rate, and request-level top-slot booking-rate proxy metrics; it requires bounded model and baseline NDCG@10, MAP@10, MRR@10, and recall@10 payloads plus bounded deltas, and rejects impossible rates/deltas, inconsistent holdout counts, and sparse or rate-inconsistent top-slot success/failure counts, so do not promote runs that only have positive point-estimate deltas
   • by default, the exporter evaluates a mature 24-hour exposure window ending 24 hours before --end-time and attributes clicks/bookings through --end-time; keep --label-attribution-window-hours aligned with the training extractor before comparing CTR/CVR
   • the release gate requires the exporter snapshot metadata (source=clickhouse, metric_family, collected_at, distinct non-empty baseline/candidate assignments, non-empty candidate_strategy_ids, window.hours, window.end_time, window.label_event_end_time, and window.label_attribution_window_hours) and rejects CatBoost business snapshots that do not prove the default 24-hour mature exposure and 24-hour attribution windows or were collected before the label-event cutoff; override --catboost-ranker-min-business-window-hours or --catboost-ranker-min-label-attribution-window-hours only for an experiment with an explicitly different attribution design
   • the exporter proves assignment and CatBoost serving from homepage impression request sets, attributes top-slot clicks back to the exposed (request_id, item_id), and attributes bookings back by request_id; top-slot CTR counts one clicked/not-clicked outcome per exposed top-slot item request rather than raw duplicate click events; downstream click/booking events do not need to repeat strategy_id, but bookings without request_id cannot support CVR evidence
   • candidate serving coverage is matched only to candidate_catboost_rerank, selected_items_catboost_rerank, and section_selected_items_catboost_rerank; repeat --candidate-strategy-id to override this set for a canary with a renamed strategy
   • use scripts/monitoring/homepage_ranking_guardrails.sql for manual inspection or ad hoc ClickHouse validation of the same metric keys

Serving API V2 Gates

1. Functional staging smoke checks pass for health, readiness, model info, cohort features, item features, homepage, refresh, homepage decision, and direct recommendations:
   • uv run python scripts/benchmark_serving_api_v2.py --base-url <staging-base-url> --member-user-id <known-featured-member-id> --iterations 20 --warmup-iterations 5 --fail-on-regression --output text
2. Bounded in-cluster staging load passes before traffic promotion:
   • uv run python scripts/benchmark_serving_api_v2.py --base-url <staging-base-url> --member-user-id <known-featured-member-id> --iterations 20 --warmup-iterations 5 --load-requests 50 --load-concurrency 5 --fail-on-regression --output text
3. Staging API latency evidence must be collected from in-cluster execution or a dedicated load-test runner. Local port-forward runs are acceptable only for functional reachability diagnostics.
4. The checked-in serving API baseline is recsys/serving/baselines/api_v2.json.

Data Quality Gates

1. Feature availability ratio (member) >= 0.85.
2. Ranker feature completeness ratio >= 0.99.
3. No prolonged stale ANN index (<= 24h p95 age).
4. No sustained fallback spikes beyond canary thresholds.
5. Canary metric guardrail thresholds (must all pass):
   • homepage p95 latency <= 1.0s
   • homepage fallback ratio <= 0.10
   • homepage decision p95 latency <= 0.25s
   • homepage decision fallback ratio <= 0.02
   • refresh timeout-terminalized ratio <= 0.02
   • member feature availability ratio >= 0.85
   • ranker decision failure ratio <= 0.05
   • personalization queue depth max <= 100
   • pending refresh age seconds max <= 45
   • worker failure ratio <= 0.10
   • ANN index age p95 <= 24h
6. CatBoost ranker business guardrails pass before canary expansion:
   • candidate top-slot CTR beats baseline by >= 0.001 absolute
   • candidate booking CVR beats baseline by >= 0.001 absolute
   • candidate and baseline sample counts meet the configured minimums
7. Restaurant cutover validation passes:
   • uv run python scripts/validate_restaurant_cutover.py --sample-query buffet --sample-query omakase --sample-query "rooftop bar"
8. If snapshot evidence is available, validate guardrails with:
   • uv run python scripts/export_canary_metrics_snapshot.py --prometheus-url http://<prometheus-host>:9090 --output artifacts/release/canary-metrics.json
   • uv run python scripts/release_readiness.py --metrics-snapshot artifacts/release/canary-metrics.json
9. For one-step execution with live Prometheus metrics, use:
   • uv run python scripts/release_readiness.py --prometheus-url http://<prometheus-host>:9090 --metrics-output artifacts/release/canary-metrics.json

Performance Gates

1. Homepage p95 latency under release threshold (1.0s absolute max).
2. Ranker fallback latency and regular ranking latency are within existing test budgets.
3. Queue depth and pending refresh age are stable during canary windows.
4. Feast v2 online lookup p95/p99, historical retrieval throughput, publication throughput, and payload-size regression gates pass through --with-feature-store-v2-gates.
5. Serving API v2 functional, latency, and bounded-load gates pass through scripts/benchmark_serving_api_v2.py.
6. Redis and ClickHouse capacity validation uses measured FEATURE_STORE_V2_* cardinality and payload-size inputs before serving is enabled.

Required capacity variables for --feature-store-v2-capacity-from-env:

• FEATURE_STORE_V2_ACTIVE_MEMBERS
• FEATURE_STORE_V2_ACTIVE_ITEMS
• FEATURE_STORE_V2_ACTIVE_COHORTS
• FEATURE_STORE_V2_MEMBER_PAYLOAD_BYTES
• FEATURE_STORE_V2_ITEM_PAYLOAD_BYTES
• FEATURE_STORE_V2_COHORT_PAYLOAD_BYTES
• FEATURE_STORE_V2_DAILY_MEMBER_ROWS
• FEATURE_STORE_V2_DAILY_ITEM_POPULARITY_ROWS
• FEATURE_STORE_V2_DAILY_ITEM_METADATA_ROWS
• FEATURE_STORE_V2_DAILY_COHORT_ROWS
• FEATURE_STORE_V2_RETENTION_DAYS (optional; defaults to 730)
• FEATURE_STORE_V2_REDIS_REPLICATION_FACTOR (optional; defaults to the capacity planner default)

Security/Operational Gates

1. Required secrets present for target environment:
   • CACHE_HMAC_SECRET (production-safe mode)
   • PROMETHEUS_BEARER_TOKEN (if release workflow exports canary snapshot)
2. Required GitHub Actions variables for release workflow:
   • PROMETHEUS_URL (optional; enables automatic canary snapshot export + artifact upload)
3. No mock/synthetic production paths enabled.
4. Runbooks are linked in alerts and accessible to on-call.

Rollout Confirmation

1. Canary progression completed: 5% -> 25% -> 50% -> 100%.
2. No critical auto-rollback alerts triggered during each stage.
3. Post-rollout observation window (minimum 24h) is stable.

Evidence To Attach

1. CI run URL and commit SHA.
2. Canary dashboard snapshots.
3. MLflow run ID/model version promoted.
4. ANN rebuild run ID and timestamp.

Related Docs

• docs/canary_rollout_runbook.md
• docs/runbooks/canary_dashboard_spec.md
• docs/runbooks/model_rollback_runbook.md
• docs/runbooks/retrain_reindex_playbook.md