Retrain and Reindex Playbook

Use this playbook when model quality regresses, feature behavior drifts, or ANN retrieval quality needs refresh.

When To Run

• Weekly/bi-weekly scheduled refresh.
• Post-incident recovery after bad rollout or stale ANN index.
• Significant quality delta in offline/online metrics.

Preconditions

1. ClickHouse ingestion is healthy.
2. Feature pipelines are fresh enough for training window.
3. MLflow and artifact store are available.
4. Restaurant OpenSearch cutover validation passes.

Procedure

0) Validate Restaurant Cutover Inputs

uv run python scripts/validate_restaurant_cutover.py \
  --sample-query buffet \
  --sample-query omakase \
  --sample-query "rooftop bar"

This validates:

• the configured restaurant alias/index resolves
• canonical restaurant mapping fields exist
• intent retrieval returns hits through the new field contract
• ClickHouse Feast v2 item latest tables contain canonical restaurant metadata

Stop here if the script exits non-zero.

1) Train

TRAINING_DATA_SOURCE=clickhouse \
uv run python -m scripts.train --config homepage_personalization

Capture RUN_ID from output.

2) Evaluate

uv run python -m scripts.evaluate --run_id <RUN_ID> --holdout_path <temporal_holdout.{csv,parquet}>

Verify required metrics (for promotion gate): ndcg_at_10, recall_at_50, hitrate_at_10, coverage_at_50, diversity_at_10.

3) Register to Engineering

uv run python -m scripts.register_model \
  --run_id <RUN_ID> \
  --name homepage_two_tower \
  --stage Engineering

If gates fail, stop and investigate feature/data/model quality.

4) Build ANN from Same Run

uv run python -m scripts.backfill_ann \
  --run_id <RUN_ID> \
  --output data/ann/index.faiss \
  --backup_dir data/ann/backups

5) Canary Rollout

Use canary runbook rollout schedule:

1. 5% for 24h
2. 25% for 24h
3. 50% for 24h
4. 100% if all guardrails pass

Reference:

• docs/canary_rollout_runbook.md
• docs/runbooks/canary_dashboard_spec.md

6) Promote to Production

Use the reviewed production promotion workflow, not direct registration to Production. For manual fallback, resolve and promote the reviewed engineering alias with scripts.promote_model; CatBoost ranker promotion must include a validated live business-metrics snapshot.

uv run python -m scripts.promote_model \
  --model-name homepage_two_tower \
  --run-id <RUN_ID> \
  --source-alias engineering \
  --execute

For CatBoost rankers, add:

  --catboost-ranker-business-metrics-snapshot artifacts/release/ranker-business-metrics.json

Ensure ANN index in serving nodes matches the promoted run.

CatBoost Ranker Retraining

1. Extract request-level ranker choice sets: uv run python -m scripts.extract_ranker_training_data --output data/training/ranker --days 30 --max-missing-required-feature-rate 0.0 --label-attribution-window-hours 24 The extractor queries downstream label events through the end of the extraction period, but only trains on impressions whose attribution window has fully matured. Keep the attribution window aligned with the live CTR/CVR measurement window; a wider window may recover delayed bookings, while an overly wide window can credit behavior that is less attributable to the ranked exposure. Do not pass --limit for production retraining; it is only a development sampling knob because truncating the event window can drop candidates or post-exposure label events and bias offline CTR/CVR proxy evidence. The extractor records extraction_limited and interaction_limit, and training rejects limited artifacts before offline promotion gates run. A real extraction filters to events with non-empty request_id and fails when the source window returns no ranker interactions; do not substitute session-level or minute-level group IDs because CatBoost query groups, offline CTR/CVR proxies, and live canary attribution are all request-scoped. Use --dry-run only to inspect the planned window without writing artifacts.

2. Confirm dataset artifacts exist: features.npy, labels.npy, click_labels.npy, groups.npy, timestamps.npy, item_ids.npy, positions.npy, baseline_scores.npy, matching evaluation_*.npy arrays including evaluation_click_labels.npy, feature_metadata.json, and dataset_metadata.json. Missing artifacts are fatal; the training command exits with an error instead of silently skipping model creation. The primary arrays contain comparable groups for CatBoost training; the evaluation_*.npy arrays contain all mature exposure groups for request-level offline rate metrics, including no-click/no-booking groups. dataset_metadata.json includes input_group_count, evaluation_group_count, offline_rate_denominator_scope=all_mature_exposure_groups, comparable_group_count, dropped_group_count, dropped_group_rate, extraction_limited=false, interaction_limit=null, position_coverage, baseline_score_source, label_attribution_window_hours, exposure_start_timestamp, exposure_end_timestamp, and label_event_end_timestamp; review these because the extractor drops one-item/no-label-variation groups from training only, excludes immature impressions, and uses logged exposure order as the offline baseline for any request whose candidate set has incomplete retrieval scores. Training rejects missing or invalid attribution metadata and requires label_event_end_timestamp >= exposure_end_timestamp + label_attribution_window_hours, so offline CTR/CVR proxies cannot be computed from immature labels. Extraction/training reject limited extraction artifacts, malformed labels, non-1D or row-count-mismatched labels/groups, non-string/non-integer or empty group IDs, duplicate logged positions, and feature matrices that do not match the production ranker feature schema because they make ranking targets, the baseline order, or the CatBoost feature-name contract ambiguous. The interaction-event extractor does not map post-ranking strategy_id into the CatBoost source_type feature. If a pre-rank candidate source is unavailable in the source table, training uses the serving-compatible default source type to avoid treatment leakage and train/serve skew.

3. Train the ranker with validation, test metrics, and the offline promotion gate: uv run python -m scripts.train_catboost --data-path data/training/ranker --output models/ranker --loss-function 'YetiRank:mode=NDCG;top=10;dcg_type=Base;dcg_denominator=LogPosition' --iterations 2000 --learning-rate 0.05 Use an optimizable CatBoost ranking objective for --loss-function, such as YetiRank:mode=NDCG;top=10;dcg_type=Base;dcg_denominator=LogPosition, YetiRank, or LambdaMart:metric=NDCG. Metric-only names such as NDCG, MAP, and MRR are evaluation metrics, not accepted training objectives. Keep the default non-Classic YetiRank NDCG path on CPU and keep its DCG type/denominator aligned with the explicit NDCG:top=10;type=Base;denominator=LogPosition evaluation metric; CatBoost documents non-Classic YetiRank modes and LambdaMart as CPU-only. Do not use QueryCrossEntropy with the default graded 0..4 ranker labels; CatBoost defines that objective for labels in [0, 1], so training fails unless labels are normalized first. Training passes explicit unit group_weight values and requires sampling_unit=Group for CatBoost so each homepage ranking opportunity has equal query weight and is sampled as one ranking unit under CatBoost’s GroupWeight and sampling semantics. Do not use inverse group-size group_weight values; request-level equalization belongs in offline metrics, while CatBoost requires all GroupWeight values inside a group to be equal. Training also fails when more than 95% of extracted request groups are dropped before CatBoost training. Override --max-comparable-group-drop-rate only after inspecting label sparsity and confirming the offline holdout remains representative enough for promotion. Training, evaluation, and online inference CatBoost Pools attach the explicit ranker feature_names list from recsys.ranking.feature_schema; treat any feature-order change as a model-contract change that requires retraining. The saved model.cbm embeds CatBoost metadata for the feature schema fingerprint, feature names, and categorical feature indices, and metadata.json repeats the same contract. metadata.json also includes offline_promotion_gate.status, registration_allowed, and the exact offline NDCG/CTR/CVR proxy thresholds used for the run; skipped or unrecorded gates are not production-promotion evidence. Serving requires this compatible metadata.json sidecar by default and falls back instead of loading a stale, mismatched, or gate-skipped model. In production with PRODUCTION_SAFE_MODE=true, startup fails instead of silently falling back when CATBOOST_RANKER_PATH is configured but the ranker does not load; set RANKER_REQUIRE_CATBOOST=true to enforce the same requirement even before a path is injected. Only disable metadata validation with RANKER_REQUIRE_MODEL_METADATA=false for local experiments.

4. The training command fails before saving/registering unless offline evidence passes. Defaults require delta_vs_baseline_ndcg_at_10 >= 0, delta_vs_baseline_click_rate_at_1 >= 0.001, delta_vs_baseline_booking_rate_at_1 >= 0.001, non-negative 95% lower confidence bounds for NDCG, request-level top-slot click-rate, and request-level top-slot booking-rate deltas over all holdout groups, at least 1,000 click-positive and 100 booking-positive holdout groups, at least 10 top-slot successes and 10 top-slot failures for model and baseline click/booking proxy rates, and non-empty feature_metadata. Use --skip-offline-promotion-gate only for local experiments that must not be registered or promoted. The offline gate also rejects incomplete or impossible metric payloads, including missing holdout num_groups/num_samples, group counts larger than the total holdout groups, top-slot success/failure counts that do not add up to num_groups, top-slot counts that disagree with their reported rates, bounded metrics outside [0, 1], or baseline deltas outside [-1, 1]. The training script rejects combining --skip-offline-promotion-gate with --register-model; rerun without the skip flag after the offline gate passes before registration. Offline metric helpers reject mismatched, non-1D, or non-finite prediction, baseline, label, click-label, and group inputs before scoring. Model rankings break tied CatBoost scores with the logged baseline score to match serving behavior, while baseline rankings use logged exposure position to break tied baseline scores; graded labels never break ties or inflate NDCG, MAP, MRR, recall, or booking metrics. Request-level top-slot CTR proxies use explicit binary click labels so view/favorite/checkout/booking label upgrades do not count as clicks.

5. Register only after the offline gate passes: uv run python -m scripts.train_catboost --data-path data/training/ranker --output models/ranker --register-model Registration requires --mlflow-uri or MLFLOW_TRACKING_URI and fails the command if MLflow logging or model registration fails. The MLflow run logs feature_metadata.json, dataset_metadata.json, split_metadata.json, and offline_promotion_gate.json alongside ranker metrics so registry promotion can be audited against the same data coverage, temporal holdout evidence, and gate thresholds as the saved model sidecar. CatBoost registry gates recheck NDCG, request-level top-slot click-rate, and request-level top-slot booking-rate lower confidence bounds, so a run with positive point-estimate lift but noisy or missing lower-bound evidence is not promotable. They also require bounded ranking-quality payloads to include model and baseline NDCG@10, MAP@10, MRR@10, and recall@10 values plus bounded deltas, and reject impossible or incomplete offline metric payloads, including bounded rates outside [0, 1], deltas outside [-1, 1], missing holdout num_groups/num_samples, non-whole counts, sparse or rate-inconsistent top-slot success/failure counts, and positive-group counts larger than the holdout group count.

6. During canary, export and validate the CatBoost ranker business snapshot:

uv run python scripts/export_catboost_ranker_business_metrics_snapshot.py \
  --output artifacts/release/ranker-business-metrics.json

uv run python scripts/release_readiness.py \
  --skip-default-checks \
  --catboost-ranker-business-metrics-snapshot artifacts/release/ranker-business-metrics.json

The default exporter window is a mature 24-hour exposure window ending 24 hours before --end-time; clicks and bookings are attributed through --end-time when they occur within --label-attribution-window-hours after the request exposure. Wait for the attribution delay before using canary data for CTR/CVR expansion, and keep the live attribution window aligned with the training extractor.

7. Expand only if the live gate passes. Defaults require >= 0.001 absolute point lift on top-slot CTR and booking CVR versus baseline with at least 1,000 top-slot impressions and 1,000 requests per variant, at least 10 successes and 10 failures per variant for each CTR/CVR confidence calculation, positive 95% lower-confidence-bound lift for both metrics, plus >= 0.95 candidate strategy-match ratio. Candidate metrics are attributed only to candidate_catboost_rerank, selected_items_catboost_rerank, and section_selected_items_catboost_rerank by default; repeat --candidate-strategy-id for an intentionally renamed CatBoost strategy. When using manual deploy/release workflows for CatBoost ranker traffic expansion, set require_catboost_ranker_business_metrics=true and catboost_ranker_business_metrics_snapshot=artifacts/release/ranker-business-metrics.json so the workflow readiness job enforces the same live gate before build/release. For promote-production-model runs where the resolved candidate is a CatBoost ranker by model name, model-version tag, or source-run tag, pass the same snapshot path; promotion fails before alias movement if the live gate is missing or failing. The release gate requires exporter metadata proving source=clickhouse, post-cutoff collected_at, and the mature exposure and attribution windows; do not hand-author a flat metric JSON for CatBoost expansion. It rejects missing or identical baseline/candidate assignment names, missing candidate strategy IDs, and recomputes CatBoost strategy-match and assignment-overlap ratios from the raw request counts in the snapshot, so ratio-only or internally inconsistent evidence is rejected. The exporter determines experiment assignment and CatBoost serving coverage from homepage impression request sets, rejects non-zero baseline/candidate request overlap by default, attributes top-slot clicks back to the exposed (request_id, item_id) without requiring click events to repeat the exposure position, and attributes bookings back by request_id. Top-slot CTR uses one clicked/not-clicked outcome per exposed top-slot item request, not raw duplicate click events, so the confidence gate is a valid two-proportion check. Downstream click/booking events do not need to repeat the CatBoost strategy_id, but bookings without a request ID cannot be attributed to a canary exposure and must be fixed at the producer before using CVR evidence.

Rollback Path

If canary guardrails breach:

1. Freeze rollout.
2. Roll back to the previous good registered model version with scripts.rollback_model or .github/workflows/rollback-production-model.yml.
3. Rebuild ANN from previous good run.
4. Validate recovery with canary dashboard guardrails.

Reference:

• docs/runbooks/model_rollback_runbook.md

Post-Run Checklist

1. Record run ID and model version in change log.
2. Record ANN rebuild timestamp and artifact path.
3. Confirm alert baseline is stable for at least 30 minutes.
4. Update retraining cadence decision in weekly review notes.