Production Training Bootstrap Runbook

This runbook bootstraps real production recommendation training data for hh-lion by backfilling canonical booking_confirmed events from booking_production into liondb_production.interaction_events, then recomputing features before handing off to the dedicated production training workflow.

Use this when:

production serving is deployed
booking_production contains the real reservation history we want to learn from
hh-lion ClickHouse tables are present but do not yet contain enough trustworthy production training events

Current phase-1 mapping

The initial bootstrap uses a conservative rule:

booking_production.reservations.arrived = 1 -> interaction_events.event_type = 'booking_confirmed'

Additional filters:

user_id IS NOT NULL
restaurant_id IS NOT NULL
no_show = 0
restaurant is active and bookable

This intentionally favors label quality over recall for the first production model.

Workflow

GitHub Actions workflow:

.github/workflows/bootstrap-production-training.yml

Inputs:

since_days
run_backfill
recompute_features

Related training workflow:

.github/workflows/train-production-model.yml

Related production evaluation workflow:

.github/workflows/evaluate-production-model.yml

Related promotion workflow:

.github/workflows/promote-production-model.yml

Scheduled cadence

The production workflows now support a conservative scheduled operating cadence:

Bootstrap Production Training Data
- schedule: 0 1 * * * (daily, UTC)
- scheduled behavior: readiness + feature recompute only
- scheduled runs do not execute booking or S3 backfills by default
Train Production Model
- schedule: 0 3 */2 * * (every 2 days, UTC)
- scheduled behavior: run the existing readiness gate first, then CPU training
Evaluate Production Candidate
- remains manual
- this is now the only blessed production evaluation path
- disposable/manual debug pods (for example hh-lion-eval-manual) are emergency-only and should not be used for normal candidate review
Promote Model to Production
- remains manual

This is now the standard conservative production operating model:

scheduled data/feature health refresh
scheduled candidate generation
manual official evaluation
manual promotion
explicit rollback path

Default operator responsibilities

Daily scheduled bootstrap/readiness run
- confirms training readiness and refreshes member/item/cohort features
- does not perform booking/S3 backfills unless manually dispatched with those inputs enabled
Every-2-days scheduled training run
- produces the next candidate run id through the official production training path
- does not auto-evaluate or auto-promote
Manual evaluation
- use .github/workflows/evaluate-production-model.yml
- this is the required review point before any promotion decision
Manual promotion
- use .github/workflows/promote-production-model.yml
- only after the official evaluation supports replacement of the current production model
Manual rollback (if needed)
- use .github/workflows/rollback-production-model.yml
- treat the workflow summary as the proof artifact for restored registry/ANN/live-serving consistency

Required GitHub environment settings (`production`)

Variables:

AWS_ROLE_ARN
AWS_REGION
EKS_CLUSTER_NAME
SOURCE_CLICKHOUSE_HOST
SOURCE_CLICKHOUSE_HTTP_PORT
SOURCE_CLICKHOUSE_DATABASE

Secrets:

SOURCE_CLICKHOUSE_USER
SOURCE_CLICKHOUSE_PASSWORD

The target hh-lion ClickHouse connection continues using the existing production lion-config / lion-secrets values.

Kubernetes job used for backfill

Manifest:

k8s/shared/bootstrap-booking-production-backfill-job.yaml

Script executed:

python -m scripts.backfill_booking_production_events --since-days <N>

Expected sequence

Patch source ClickHouse bootstrap credentials into lion-config / lion-secrets
Run booking-production backfill job
Run the production training readiness gate when feature recompute is requested
Run full feature recomputes:
- member features
- item features
- cohort features
Validate cohort historical rows and Redis online candidate sets before serving selected-items traffic
Trigger the dedicated production training workflow when the refreshed corpus is ready
Trigger the dedicated production evaluation workflow for the resulting MLflow run_id
Trigger the dedicated production promotion workflow only after the evaluation looks acceptable
Treat the release as complete only after the promotion workflow proves MLflow alias, ANN lineage, and live /v2/model/info all match

Notes

The backfill currently creates canonical hh-lion booking_confirmed rows directly in interaction_events.
It assigns strategy_id=booking_production_bootstrap_v1 for provenance.
It derives package_id from reservation_packages.restaurant_package_id when available.
It uses reservation/user/session information to create stable event_id, anonymous_id, and request_id values.

Next validation steps after first backfill

Count backfilled booking_confirmed events in interaction_events
Verify member/item feature table freshness

Verify cohort candidate online-store publication with:

python -m scripts.compute_cohort_features --mode full --publish-online
python -m scripts.validate_feature_store_v2 \
  --feature-set cohort_popularity_features_v2 \
  --validate-candidate-online-store \
  --candidate-online-probe-key <known-cohort-key> \
  --output text
python -m scripts.benchmark_serving_api_v2 \
  --base-url <hh-lion-base-url> \
  --require-selected-items-replace \
  --fail-on-regression \
  --output text

Verify training-eligible row counts
Run the first production training workflow (.github/workflows/train-production-model.yml)
Run the production evaluation workflow (.github/workflows/evaluate-production-model.yml) against the resulting run_id
Promote only after the trusted production evaluation is acceptable (.github/workflows/promote-production-model.yml)
Confirm the promotion workflow summary shows release-consistency verification across MLflow alias, ANN lineage, and live serving

Keyboard shortcuts

HungryHub Knowledge Base