Production Recommendation Data Preparation & Training Plan

Purpose

This document proposes how to turn hh-lion from a deployable recommendation service into a real production recommendation system with:

• repeatable data preparation
• scheduled model training
• offline evaluation and promotion gates
• ANN refresh tied to the promoted model
• production-safe rollout and rollback
• clear separation between online serving and offline learning

This began as a review plan only. It is now also being used as an implementation/status tracker.

Status Key

• ✅ Done
• 🟡 Partial / implemented but not fully hardened
• ❌ Not done yet

Current Overall Status

Core production learning capability is implemented and operating for real production use:

• ✅ real production data bootstrap into ClickHouse
• ✅ production readiness gate against ClickHouse with production thresholds
• ✅ production feature recomputation before both official training and official evaluation
• ✅ production training runs in-cluster via GitHub Actions + K8s jobs
• ✅ trusted temporal offline evaluation through the official production evaluation workflow
• ✅ gated production promotion based on real production-trained candidates
• ✅ dedicated readiness/bootstrap, training, evaluation, promotion, rollback, and model-state reporting workflows now exist
• ✅ conservative scheduled operating cadence is enabled:
  • daily readiness + feature refresh
  • every-2-days CPU training
  • manual official evaluation
  • manual promotion
• 🟡 ANN/run-lineage and live-serving proof are strongly enforced operationally, but can still be polished further as operator-facing reporting/runbook ergonomics

Remaining Work (short list)

These are now the highest-value cleanup / hardening items rather than missing core architecture:

1. Polish operator-facing release proof

   • keep the combined MLflow alias + ANN lineage + live API consistency checks easy to inspect from workflow summaries and runbooks
   • make the evidence trail clearer for human reviewers

2. Polish readiness/data-quality reporting

   • keep the existing readiness gate, but make its operator-facing report more explicit and reusable

3. Tighten operating docs and ownership

   • document the scheduled cadence, review expectations, and on-call/operator responsibilities more explicitly

4. Advanced follow-ons (optional)

   • richer segmentation slices
   • stronger candidate-review UX
   • promotion automation only if later desired

1. Executive Summary

Proposed training strategy

Primary strategy: offline batch training with scheduled retraining

We should use:

• offline batch training for the core retrieval model (homepage_two_tower)
• event-driven / nearline feature updates for freshness in serving
• manual or gated promotion from Engineering to Production
• no true online learning in the first production-ready version

Why this is the right choice

For Hungry Hub’s current architecture, offline batch training is the most realistic and safe production approach because:

1. the repo already implements this lifecycle:
   • ingest events -> build features -> train -> evaluate -> register -> promote -> serve
2. the current serving system depends on:
   • ClickHouse for interaction history
   • Redis for online/serving state
   • MLflow for model registry/artifacts
   • FAISS ANN index for retrieval
3. true online learning would add much higher risk:
   • model drift without guardrails
   • harder rollback
   • online training instability
   • more difficult reproducibility and auditability

Real-world production architecture target

The production system should behave like a standard recommender stack:

Online loop

• API serves homepage recommendations
• API emits impression events
• client/consumer events are written to Kafka
• Kafka consumer writes validated events to ClickHouse
• Redis / Feast-backed features keep online context fresh

Offline loop

• scheduled data quality validation
• scheduled feature recomputation
• scheduled model training from ClickHouse
• scheduled offline evaluation against temporal holdout
• register candidate model in MLflow Engineering
• build ANN index from same model run
• canary/promotion workflow to Production

This is the right architecture for a real-world production recommender because it preserves:

• low-latency serving
• reproducible training
• promotion safety
• rollback ability
• clear model provenance

2. Current State in This Repository

From the existing repo, hh-lion already contains many of the core pieces:

Existing capabilities

• ClickHouse schema migrations
• Kafka consumer for ingesting interaction events
• feature computation scripts:
  • scripts.compute_member_features
  • scripts.compute_item_features
  • scripts.compute_cohort_features
• training script:
  • scripts.train
• offline evaluation script:
  • scripts.evaluate
• model registration / promotion gating:
  • scripts.register_model
• ANN rebuild script:
  • scripts.backfill_ann
• release/readiness and rollback docs
• Kubernetes training job manifests:
  • k8s/shared/training-job.yaml
  • k8s/shared/training-job-cpu.yaml
• engineering bootstrap workflow:
  • .github/workflows/bootstrap-engineering.yml
• deploy workflow with optional training image build:
  • .github/workflows/deploy.yml

Current gap (status update)

The original gap list has been reduced substantially.

Status against the original gap list:

1. preparing real production training data
   • ✅ Implemented via booking-production bootstrap plus Personalize/Metarank S3 backfills normalized into ClickHouse
2. validating data sufficiency and data quality
   • 🟡 Implemented operationally through readiness gates, production thresholds, bounded runs, count checks, and trusted evaluation; the remaining gap is mostly report polish rather than missing enforcement
3. running scheduled production training jobs
   • ✅ Implemented with a conservative default cadence via scheduled GitHub Actions + Kubernetes jobs
4. evaluating with production-grade holdouts
   • ✅ Implemented with trusted in-cluster temporal holdout evaluation
5. refreshing ANN from the same promoted run
   • ✅ Strongly enforced through model-version lineage checks before alias movement, re-verification on @production, deploy-time alias/ANN validation, and post-rollout live consistency verification
6. promoting only genuinely trained prod-ready models
   • ✅ Implemented in practice; weak candidates were blocked and a real production-trained model was ultimately promoted
7. documenting and automating the whole cadence end-to-end
   • 🟡 Mostly implemented; remaining work is documentation/runbook/operator polish rather than missing workflows

3. Design Principles

We should implement the production training system with these principles:

1. Batch training first, online learning later
   • prioritize correctness and operability over novelty
2. Train/serve consistency
   • the same feature definitions used offline should back serving as much as possible
3. Temporal validation, not random split
   • recommendations are time-sensitive; leakage is dangerous
4. Single run lineage
   • one run ID should tie together training, evaluation, registration, and ANN build
5. Promotion only after evidence
   • no production alias should point to ad hoc or imported artifacts without explicit labeling
6. Environment clarity
   • engineering artifacts stay engineering
   • production artifacts stay production
7. Graceful serving fallback
   • production should degrade safely if training is stale, but model provenance must remain explicit
8. Automate the routine, keep promotion controlled
   • data prep and training can be automated; production promotion should still be reviewed/gated initially

4. Recommended Production Training Architecture

4.1 Training mode

Decision

Use offline batch training for homepage_two_tower.

Not chosen for phase 1

We should not implement true online learning yet, such as:

• continual gradient updates inside serving
• streaming model updates directly from Kafka
• online bandit-only production control loops as the core model lifecycle

Rationale

Offline batch training is the standard first reliable architecture for a production recommender because it gives:

• reproducibility
• explicit model versions
• easy rollback
• easier debugging
• lower operational risk

Recommended cadence

Initial cadence:

• daily data prep / feature recompute
• daily or every-2-days training run depending on data volume
• manual promotion to Production after evaluation and review

Later, after confidence is established:

• move to scheduled Engineering retraining
• optionally promote automatically only if all quality and operational guardrails pass

4.2 System topology

Online serving path

1. user requests homepage
2. API resolves identity/context
3. API retrieves candidates from:
   • model retrieval / ANN
   • cohort popularity
   • session/search intent
4. API reranks / applies serving constraints
5. API returns sections
6. API writes impressions to Kafka
7. downstream events continue feeding ClickHouse

Offline learning path

1. ingest validated interaction events into ClickHouse
2. run data quality checks and sufficiency checks
3. recompute member/item/cohort features
4. export or query temporal training/holdout windows
5. train the retrieval model
6. evaluate offline metrics
7. register candidate in MLflow Engineering
8. build ANN from that same run
9. deploy/canary/promote to Production
10. monitor quality and operational metrics

5. Production-Ready Data Strategy

5.1 Source of truth

Canonical training source

Use ClickHouse as the authoritative offline training source consumed by hh-lion training jobs.

However, for initial production bootstrap we should not assume ClickHouse is already rich enough. We should explicitly leverage upstream production data sources to populate and validate the offline corpus.

Initial production bootstrap sources

A. booking_production database

Use this as the most trustworthy initial label source because it represents real reservation outcomes.

Primary intended use:

• bootstrap positive interactions for training
• validate reservation volume, recency, and restaurant coverage
• backfill booking_confirmed-equivalent events into ClickHouse when needed

Why it matters:

• this is stronger and more trustworthy than synthetic bootstrap events
• it provides real conversion labels for the first real production model

B. production-hh-personalize-s3-bucket

This bucket appears highly relevant to homepage personalization bootstrap. Initial inspection shows:

• interactions-v1/ daily CSV snapshots with user_id,item_id,timestamp,event_type
• production-reservations-*.csv files mixed under interactions-v1/
• items-v1/ restaurant catalog snapshots
• users-v1/ user snapshots
• state/amazon_personalize/ and recommendation-related prefixes

Primary intended use:

• recover historical homepage/user-item interaction data
• bootstrap item and user dimensions
• join reservation labels to broader browsing/view behavior
• inspect whether this bucket already contains a usable historical training corpus

C. production-hh-metarank-s3-bucket

This bucket appears most useful for search/ranking/session behavior. Initial inspection shows:

• interactions-v1-raw/ minute-level JSONL raw events
• interactions-v1/ large daily JSONL(.gz) event exports
• items-v1/ and users-v1/ snapshots
• state/metarank/

Primary intended use:

• extract richer behavioral context such as ranking/search/session interactions
• bootstrap reranker or future feature engineering
• augment user-item affinity signals beyond reservations alone

Recommended source hierarchy

For phase 1, use sources in this order:

1. booking_production for trusted purchase/booking labels
2. personalize S3 for homepage behavior history and reservation-adjacent interaction data
3. metarank S3 for richer engagement/session/search signals
4. ClickHouse as the final normalized store that hh-lion trains from

Event source after normalization

After ingestion/backfill, train from normalized ClickHouse events, especially:

• booking_confirmed
• optionally other engagement signals later if approved for weighting

Phase 1 recommendation

For the first real production-ready model, keep the positive-label definition conservative and aligned with the repo’s current logic:

• primary training label = reservation/booking completion mapped to booking_confirmed
• item type = restaurant
• valid rows require non-null user, item, package ids where available

This is safer than immediately broadening labels with weaker behavioral events.

Important architecture rule

Even if bootstrap data comes from booking_production or S3, we should still normalize everything into ClickHouse before training. That keeps the actual hh-lion training pipeline consistent with the runtime architecture and avoids a one-off training path that diverges from future operations.

5.2 Data onboarding plan for initial production readiness

We should add an explicit bootstrap/offline ingestion step before recurring training starts.

Step 1: Source profiling

Before implementation, profile all three candidate sources:

• booking_production
• production-hh-personalize-s3-bucket
• production-hh-metarank-s3-bucket

For each source, measure:

• date coverage
• row counts by day
• distinct users
• distinct restaurants/items
• event type distribution
• identifier availability (user_id, anonymous_id, session_id, item_id, package_id)
• timestamp format and timezone quality
• duplicate rates
• null rates

Step 2: Identifier mapping

We need a canonical identity contract for bootstrap data.

Proposed mapping priority:

• use user_id when present for member training
• preserve anonymous_id / session_id when available for future entity training or session features
• normalize restaurant/item identifiers to the same item_id contract used by hh-lion

Step 3: Normalize into ClickHouse

Create a repeatable ingestion path that maps source records into the hh-lion interaction schema.

Current implementation direction for phase 1:

• reservation rows in booking_production with arrived = 1 and non-null user_id / restaurant_id are backfilled as canonical booking_confirmed events
• no_show = 1 rows are excluded from the first positive corpus
• active/bookable restaurant checks are enforced during backfill
• browse/view rows in personalize bucket can later be mapped into weaker interaction events such as view
• search/ranking session events in metarank bucket should be retained for future feature/ranker datasets, and only promoted to core retrieval training if justified

Step 4: Build first real production corpus

Construct an initial production training corpus using:

• reservation-confirmed positives from booking_production
• matched restaurant catalog data from S3 item snapshots or existing item sources
• optional auxiliary view/click behavior from personalize bucket for future weighting, negatives, or feature generation

Step 5: Transition to recurring operation

Once bootstrap data has been normalized into ClickHouse and live ingestion is healthy:

• stop relying on ad hoc historical exports for daily operation
• continue retraining from ClickHouse as the canonical store

5.3 Data windows

We should define explicit windows for production training.

Recommended initial windows

• training window: last 90-180 days of valid interactions
• validation/holdout window: most recent 7-14 days
• retraining cadence: daily or every 48h

Why temporal windows matter

Recommendation data is non-stationary:

• restaurant availability changes
• restaurant availability and campaign context change
• user behavior changes with seasonality and weekends

So evaluation must be temporal, not random.

5.4 Data sufficiency gates

Before any training job starts, it should verify:

1. minimum total eligible interactions
2. minimum distinct users
3. minimum distinct restaurants/items
4. minimum recent-window density
5. acceptable null / malformed row rate
6. acceptable skew across countries/cities/traffic sources if those segments matter for serving

Proposed production minima for first version

These are starting gates and may be tuned after data inspection:

• eligible bookings >= 10,000 preferred
• absolute minimum eligible bookings >= 1,000 only for bootstrap/non-prod
• distinct users >= 1,000
• distinct restaurants >= 200
• malformed rate <= 1%
• latest 7-day window must not be near-empty

Important decision

For real production promotion, we should set a higher threshold than the current bootstrap minimum of 1,000 interactions.

5.5 Feature preparation strategy

We should treat feature prep as a first-class production pipeline.

Required feature groups

1. member features
   • history / engagement / booking counts / recency
2. item features
   • restaurant metadata / popularity / cuisine / geography / availability-related fields where appropriate
3. cohort features
   • cold-start fallback and segment-level priors
4. session / nearline features
   • current session context, search signals, freshness

Existing repo alignment

These map directly to:

• scripts.compute_member_features
• scripts.compute_item_features
• scripts.compute_cohort_features
• optional OpenSearch enrichment already supported

Recommendation

Feature jobs should run before training on every scheduled training cycle, not be assumed fresh.

6. Model Strategy

6.1 Retrieval model

Phase 1 production model

Keep the current architecture as the production retrieval backbone:

• Two-Tower retrieval model (homepage_two_tower)
• ANN index derived from the same run

This is the right production choice because the repo is already built around it and the online serving path expects ANN-backed retrieval.

6.2 Ranking strategy

Recommendation

Treat the current retrieval model as the primary production-ready milestone.

If CatBoost reranking is present or planned, we should structure rollout as:

• Phase 1: production-ready retrieval system with stable fallbacks
• Phase 2: production-grade reranker with explicit feature completeness gates

This reduces launch risk. A strong retrieval system with robust fallbacks is already a legitimate real-world recommender foundation.

6.3 Online learning stance

Decision

No true online model updates in phase 1.

What we do support nearline

• event ingestion continuously
• Redis/serving state continuously
• feature freshness improvements continuously
• periodic retraining continuously

That is how many real production recommenders operate before introducing more advanced online adaptation.

7. Production Model Lifecycle

The desired lifecycle should be:

1. Prepare data
2. Validate quality and sufficiency
3. Recompute features
4. Train
5. Evaluate
6. Register to Engineering
7. Backfill ANN from same run
8. Smoke test in Engineering
9. Promote to Production
10. Deploy / canary
11. Observe
12. Rollback if needed

Non-negotiable rule

The model promoted to Production and the ANN index served in Production must come from the same run lineage.

8. Pipeline Workstreams To Implement

Workstream A — Production data readiness

Status: ✅ Done (initial practical gate)

Goal

Ensure production has enough clean event history for real training.

Scope

• inspect ClickHouse production interaction volume
• define production eligibility query and baseline metrics
• implement reusable data sufficiency check script
• implement data quality report artifact

Deliverables

• scripts/check_training_readiness.py or equivalent
• documented production thresholds
• GitHub Actions or K8s job step to fail fast when data is insufficient

Status notes

• ✅ production interaction volume, source coverage, and baseline corpus metrics were inspected repeatedly during real bootstrap/training work
• ✅ production thresholds and promotion expectations were established operationally during the rollout
• ✅ reusable scripts/check_training_readiness.py now checks eligible interaction volume, distinct users/items, recent-window density/freshness, malformed booking rate, and strategy-source breakdown from normalized ClickHouse data
• ✅ .github/workflows/bootstrap-production-training.yml now runs the readiness job as a fail-fast gate before feature recompute, and .github/workflows/train-production-model.yml reuses the same readiness gate before training

Workstream B — Production feature pipeline orchestration

Status: ✅ Done (with some ops hardening still possible)

Goal

Make feature recomputation repeatable before training.

Scope

• orchestrate member/item/cohort feature jobs in the right order
• ensure Feast definitions are applied and consistent
• add explicit job outputs / logs / success criteria

Deliverables

• reusable pipeline entrypoint or workflow for:
  • member features
  • item features
  • cohort features
• environment-aware configuration for production
• observability around feature freshness

Status notes

• ✅ feature recomputation order was operationalized and used successfully in production retraining runs
• ✅ production environment-aware orchestration exists in the bootstrap/training workflow and K8s jobs
• 🟡 feature freshness observability could still be made more explicit/report-driven

Workstream C — Production training orchestration

Status: ✅ Done (combined workflow shape, not yet split exactly as originally proposed)

Goal

Run training as a real production job, not an ad hoc command.

Scope

• use k8s/shared/training-job-cpu.yaml first unless GPU is justified
• parameterize output/model naming cleanly
• capture run ID reliably
• publish run metadata as artifact/output

Deliverables

• workflow/job to launch CPU training in production cluster
• optional GPU path later if needed
• durable run ID capture for downstream steps

Status notes

• ✅ implemented via .github/workflows/train-production-model.yml
• ✅ CPU training path was exercised repeatedly in the production cluster
• ✅ downstream run IDs were captured and used for evaluation/promotion decisions
• ✅ production training now has its own first-class workflow while reusing the same in-cluster readiness guardrail

Recommendation on compute

Start with CPU training unless training duration becomes operationally unacceptable.

Reason:

• simpler infra
• lower cost
• easier scheduling
• enough for early productionization

Workstream D — Evaluation and promotion gates

Status: ✅ Done

Goal

Only register and promote models that pass objective offline checks.

Scope

• standardize temporal holdout generation
• ensure required metrics are always logged
• enforce stronger production promotion thresholds than bootstrap
• store evaluation artifacts for review

Deliverables

• reusable holdout generation step
• training -> evaluation -> registration chain
• stricter production gate config if needed
• documented approval checklist

Status notes

• ✅ trusted temporal holdout evaluation is implemented and was used for real production decisions
• ✅ weak candidates were blocked from promotion
• ✅ gate logic was corrected and shipped so real production-trained candidates can be promoted cleanly
• 🟡 documented approval flow could still be cleaner as a first-class workflow/runbook checklist

Metrics to emphasize

Already aligned with repo docs:

• ndcg_at_10
• recall_at_50
• hitrate_at_10
• coverage_at_50
• diversity_at_10

We should also inspect segment metrics where practical:

• cold-start vs established members
• geo segment slices
• device/source segment slices if relevant

Workstream E — ANN rebuild and synchronization

Status: ✅ Done enough for the current production operating model

Goal

Guarantee production ANN matches the promoted model.

Scope

• automate ANN rebuild from exact promoted run
• store rebuild metadata
• ensure serving sees the right artifact version
• validate ANN freshness post-deploy

Deliverables

• model registration -> ANN backfill coupling
• metadata file / artifact proving source run
• deployment check that ANN and model alias match

Status notes

• ✅ ANN lineage metadata is recorded on registered model versions
• ✅ promotion verifies ANN lineage on the exact candidate version before alias movement
• ✅ promotion re-verifies ANN lineage on @production after alias movement
• ✅ deploy verifies production ANN lineage against the served model alias for production deployments
• ✅ promotion/rollback verify MLflow alias, ANN lineage, and live /v2/model/info agree on the same run/version before completing
• 🟡 remaining work is mostly operator-facing polish (summary/report ergonomics), not missing enforcement

Critical rule

Do not allow a production model alias to move unless the corresponding ANN refresh has succeeded or the serving path can tolerate temporary mismatch safely.

Workstream F — Scheduled orchestration

Status: ✅ Implemented as the current conservative production operating mode

Goal

Make retraining operational rather than manual.

Recommendation

Implement two scheduled tracks:

Track 1: daily data+feature readiness

• validate ClickHouse data volume/quality
• recompute features
• emit report

Track 2: scheduled retraining

• run training on cadence
• evaluate
• register to Engineering
• build ANN
• require review before Production promotion

Orchestration options

Preferred order:

1. GitHub Actions scheduled workflows if sufficient
2. Kubernetes CronJobs if cluster-local execution is better
3. hybrid: GitHub Actions triggers K8s jobs

Recommendation for this repo

Use GitHub Actions as the control plane and Kubernetes jobs as the execution plane.

Why:

• repo already uses Actions heavily
• environment secrets/vars are already centralized there
• Actions gives visibility and approvals
• K8s jobs are still the right place for heavy execution

Status notes

• ✅ this architecture choice was validated in practice during production bootstrap/training/evaluation work
• ✅ daily scheduled readiness + feature refresh is enabled via .github/workflows/bootstrap-production-training.yml
• ✅ every-2-days scheduled production training is enabled via .github/workflows/train-production-model.yml
• ✅ official production evaluation remains manual by design
• ✅ production promotion remains manual by design

9. Proposed Production Workflow Shape

9.1 New workflows to add

A. Production data prep workflow

Suggested file:

• .github/workflows/prepare-production-training-data.yml

Status: ✅ Realized via .github/workflows/bootstrap-production-training.yml

Responsibilities:

• verify prod environment config
• apply/verify migrations and Feast repo if needed
• run data readiness checks
• recompute features
• publish readiness summary

B. Production training workflow

Suggested file:

• .github/workflows/train-production-model.yml

Status: ✅ Realized via .github/workflows/bootstrap-production-training.yml + .github/workflows/train-production-model.yml

Responsibilities:

• build/pull training image if needed
• launch CPU training job
• capture run ID
• generate temporal holdout
• run evaluation
• register to MLflow Engineering
• backfill ANN from same run
• publish summary artifacts

C. Production promotion workflow

Suggested file:

• .github/workflows/promote-production-model.yml

Status: ✅ Implemented (manual, conservative first version)

Responsibilities:

• choose reviewed Engineering run/version
• re-register/promote to Production
• rebuild/verify ANN if required
• deploy or restart consumers/serving if needed
• run smoke checks

10. Real-World Operating Model

This is how the architecture should behave in a normal real-world week.

Daily

• ingest events continuously
• recompute features on schedule
• validate freshness and DQ

Every 1-2 days

• retrain candidate retrieval model
• evaluate on temporal holdout
• register to Engineering if metrics pass
• rebuild ANN from same run

On review / release decision

• inspect metrics and artifacts
• compare with current Production
• promote to Production if justified

On failure

• rollback alias to previous model
• rebuild ANN from previous good run
• keep serving stable via fallback logic

This is a standard, sane production recommender operating model.

11. Architecture Comparison: Batch vs Online Learning

Option A — Offline batch training (recommended)

Pros

• reproducible
• easy to debug
• easier rollback
• easier approvals
• matches current repo design
• good enough for a strong production baseline

Cons

• slower adaptation to same-day behavior shifts
• requires scheduling/orchestration

Option B — True online learning (not recommended now)

Pros

• potentially faster adaptation
• interesting for experimentation

Cons

• much higher complexity
• riskier promotions
• harder observability
• more train/serve skew risk
• much harder incident response

Final decision

Use offline batch training + nearline feature freshness.

That is the right “standard real-world production system” choice for this codebase right now.

12. Implementation Phases

Status snapshot

• Phase 0 — ✅ Done
• Phase 1 — ✅ Done
• Phase 2 — ✅ Done
• Phase 3 — ✅ Done enough for the current production operating model
• Phase 4 — 🟡 Partial (scheduled cadence is enabled; remaining work is ops/reporting/runbook polish)
• Phase 5 — ❌ Not started / intentionally later

Phase 0 — Planning and audit

• confirm production data availability in ClickHouse
• confirm exact event schemas and usable label volume
• confirm desired cadence and owners
• confirm whether CPU is acceptable for initial training runtime

Phase 1 — Data prep pipeline

• ✅ add production readiness checks
• ✅ add feature orchestration workflow
• ✅ add lightweight readiness reporting

Phase 2 — Training pipeline

• ✅ add production training workflow
• capture run IDs and evaluation outputs
• register successful candidates to Engineering

Phase 3 — ANN and promotion hardening

• couple ANN to run lineage
• add promotion workflow
• add smoke tests and explicit rollback path

Phase 4 — Scheduling and ops

• enable scheduled runs
• define alerting and human review expectations
• document runbooks and ownership

Phase 5 — Advanced improvements (later)

• richer labels beyond bookings if justified
• stronger segmentation evaluation
• reranker hardening
• experimentation/bandit layer
• partial automation for promotion

Production CatBoost ranker training uses one group per homepage ranking opportunity and one row per candidate item. The group_id is the non-empty request_id; extraction rejects rows without request IDs so CatBoost query groups, offline CTR/CVR proxy denominators, and live canary attribution all describe the same request-level choice sets. Duplicate (group_id, item_id) rows keep the earliest exposure features and can upgrade only to the strongest post-exposure downstream graded label. Groups with fewer than two items or fewer than two distinct graded labels are dropped only from CatBoost training arrays and counted before training; the extractor also saves evaluation_*.npy arrays over all mature exposure groups for request-level offline rate metrics. CTR proxy metrics read explicit binary click_labels.npy and evaluation_click_labels.npy artifacts so non-click engagement or booking label upgrades do not count as clicks. Training requires dataset_metadata.json to prove a mature attribution window with label_event_end_timestamp >= exposure_end_timestamp + label_attribution_window_hours, preventing offline CTR/CVR proxy gates from passing on labels that had not fully matured. CatBoost pools pass explicit unit group_weight values by default so each request has equal query weight under CatBoost’s GroupWeight semantics; inverse group-size group_weight values are intentionally not used. The extractor has no production default row limit; --limit is only a development sampling knob because truncating the event window can drop candidates or post-exposure label events and bias offline CTR/CVR proxy evidence. The extractor does not train source_type from post-ranking strategy_id; if a pre-rank candidate source is unavailable, training uses the serving-compatible default to avoid treatment leakage and train/serve skew. Offline evaluation logs model and baseline NDCG@10, MAP@10, MRR@10, recall@10, and top-slot business proxy metrics; offline promotion must beat a logged-order baseline on NDCG@10 plus request-level top-1 click and booking rate proxies over all holdout groups, while positive-group hit-rate remains diagnostic. The extractor uses real retrieval scores as the baseline only for request groups where every candidate has a finite retrieval score; otherwise it stores inverse logged position for the whole request in baseline_scores.npy and evaluation_baseline_scores.npy instead of mixing incompatible score scales or comparing against an all-zero baseline. Extraction and training reject duplicate logged positions within a request; serving and offline model metrics break tied CatBoost scores with the logged baseline score, and labels never break ties or inflate evaluation. By default, training fails before saving/registering unless CTR/CVR proxy evidence includes at least 1,000 click-positive and 100 booking-positive holdout groups, at least 10 top-slot successes and 10 top-slot failures for model and baseline click/booking proxy rates, internally consistent top-slot success/failure counts, and at least 0.001 absolute lift on both request-level top-1 proxy metrics. Saved metadata.json and registered MLflow runs record offline_promotion_gate status, registration eligibility, and the exact thresholds used; skipped or unrecorded gates are not sufficient evidence for production promotion.

13. What “Production-Ready” Should Mean Here

Current status against this definition

1. it is trained from production data, not imported from engineering
   • ✅ Achieved
2. it passes temporal offline evaluation
   • ✅ Achieved
3. it is registered in MLflow with clean provenance
   • ✅ Achieved for the promoted production-trained run
4. its ANN index is rebuilt from the same run
   • ✅ Achieved operationally with registry lineage checks, alias re-verification, and release-consistency validation
5. engineering validation succeeds
   • 🟡 Candidate review still happens primarily through the official production evaluation path rather than a separate engineering-only workflow gate
6. production rollout/smoke tests succeed
   • ✅ Achieved operationally for the deployed production system
7. rollback path is proven
   • ✅ Achieved operationally through the dedicated rollback workflow plus live consistency verification, with some runbook polish still possible

A model is only production-ready when all of the following are true:

1. it is trained from production data, not imported from engineering
2. it passes temporal offline evaluation
3. it is registered in MLflow with clean provenance
4. its ANN index is rebuilt from the same run
5. engineering validation succeeds
6. production rollout/smoke tests succeed
7. rollback path is proven

Anything less than that is bootstrap, not true production readiness.

14. Current Operating Decision

The repo now implements the intended phase-1 operating model:

• offline batch training from production data
• daily scheduled readiness + feature refresh
• every-2-days scheduled CPU candidate training
• manual official production evaluation
• manual production promotion
• explicit rollback workflow
• registry/ANN/live-serving consistency checks before release completion

This remains the recommended operating posture for now because it is conservative, observable, and reversible.

15. Remaining Recommended Improvements

The most useful follow-on improvements are now smaller operational refinements:

1. make readiness/dq reporting more reviewer-friendly
2. improve workflow summaries/runbook ergonomics for release proof
3. add richer candidate-review slices only if they materially change decisions
4. keep promotion manual unless there is a later decision to automate it

16. Final Recommendation

Treat the core production recommendation training plan as implemented.

From here, prioritize:

• operational clarity
• reviewer ergonomics
• release-proof visibility
• low-risk incremental polish

Do not broaden scope into online learning or automatic production promotion until the current operating model has been comfortably stable for a longer period.