Production-Mode Bootstrap Guide (From Empty Data)

This guide explains how hh-lion works end-to-end and how to bring it up in real (non-mock) mode when both ClickHouse and MLflow are empty.

Use this when:

engineering/production cannot use mock mode
you do not yet have enough events in ClickHouse
MLFLOW_MODEL_NAME / MLFLOW_MODEL_ALIAS are not set because no promoted model alias exists yet

Related docs:

1. Mental Model: How This System Works

hh-lion has two loops that must both be healthy.

Offline/model loop:

Events are ingested into ClickHouse (interaction_events).
Feature jobs compute member/item/cohort features.
Training reads ClickHouse booking events and logs model artifacts to MLflow.
Serving loads one registered model alias via MLFLOW_MODEL_NAME + MLFLOW_MODEL_ALIAS.

Online/serving loop:

API receives homepage requests.
API uses model + FAISS + feature services to build recommendations.
API writes impression events to Kafka.
Kafka consumer inserts events into ClickHouse, feeding the next training cycle.

If either loop is empty, personalization quality or readiness will degrade.

2. Non-Negotiable Requirements for Non-Mock Mode

For ENVIRONMENT=engineering and ENVIRONMENT=production:

MLFLOW_MODEL_NAME and MLFLOW_MODEL_ALIAS must be set.
FORCE_MOCK_MODEL=false.
MLflow and artifact storage must be reachable from serving pods.
ClickHouse must contain enough booking_confirmed events for training.

Training minimum from code:

at least 1000 valid booking interactions in interaction_events.

Valid training rows must satisfy:

event_type='booking_confirmed'
item_type='restaurant'
package_id IS NOT NULL
user_id IS NOT NULL
item_id IS NOT NULL

3. Bootstrap Plan (Zero -> Production Mode)

Do these phases in order.

Phase A: Infrastructure and Schema

Ensure services are reachable:

ClickHouse
Redis
Kafka
Schema Registry
MLflow
S3/MinIO artifact storage

Apply ClickHouse migrations:

uv run python -m scripts.migrations.apply_migrations

Apply Feast definitions:

uv run python -m scripts.apply_feast_repo

Phase B: Fill ClickHouse with Events

Run a consumer continuously:

uv run python -m scripts.run_kafka_consumer

Produce bootstrap events (if real traffic is not enough yet):

uv run python -m scripts.produce_test_events --event-type booking_confirmed --count 50000

Check event count in ClickHouse:

curl -sS -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" \
  "https://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?database=${CLICKHOUSE_DATABASE}&query=SELECT%20count()%20FROM%20interaction_events"

Check training-eligible row count:

curl -sS -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" \
  "https://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?database=${CLICKHOUSE_DATABASE}&query=SELECT%20count()%20FROM%20interaction_events%20WHERE%20event_type%3D%27booking_confirmed%27%20AND%20item_type%3D%27restaurant%27%20AND%20package_id%20IS%20NOT%20NULL%20AND%20user_id%20IS%20NOT%20NULL%20AND%20item_id%20IS%20NOT%20NULL"

Phase C: Build Feast V2 Features

Compute and publish features from ClickHouse:

uv run python -m scripts.compute_member_features --mode full --publish-online
uv run python -m scripts.compute_item_features --mode full --publish-online
uv run python -m scripts.compute_cohort_features --mode full --publish-online

Validate the cohort historical rows and Redis online candidate sets before serving selected-items traffic:

uv run python -m scripts.validate_feature_store_v2 \
  --feature-set cohort_popularity_features_v2 \
  --validate-candidate-online-store \
  --candidate-online-probe-key <known-cohort-key> \
  --output text

Run the serving benchmark with the selected-items replacement gate:

uv run python -m scripts.benchmark_serving_api_v2 \
  --base-url <hh-lion-base-url> \
  --require-selected-items-replace \
  --fail-on-regression \
  --output text

Optional metadata enrichment from OpenSearch:

uv run python -m scripts.compute_item_features --mode full --publish-online --with-opensearch-metadata

Phase D: Train First Real Model and Generate Run ID

Set MLflow/artifact env:

export MLFLOW_TRACKING_URI=<mlflow-url>
export MLFLOW_S3_ENDPOINT_URL=<s3-or-minio-endpoint>
export AWS_ACCESS_KEY_ID=<artifact-access-key>
export AWS_SECRET_ACCESS_KEY=<artifact-secret-key>

Train:

TRAINING_DATA_SOURCE=clickhouse uv run python -m scripts.train --config homepage_personalization

Capture printed line:

Run ID: <RUN_ID>

Export a temporal holdout, then evaluate, register, backfill ANN:

uv run python - <<'PY'
from recsys.shared.polars import ClickHousePolarsClient

client = ClickHousePolarsClient.from_env()
df = client.query(
    """
    SELECT
        toString(user_id) AS user_id,
        toString(item_id) AS item_id,
        event_timestamp AS timestamp
    FROM interaction_events
    WHERE event_type = 'booking_confirmed'
      AND item_type = 'restaurant'
      AND package_id IS NOT NULL
      AND user_id IS NOT NULL
      AND item_id IS NOT NULL
    ORDER BY event_timestamp
    """
)
df.write_csv("data/temporal_holdout.csv")
print("Wrote data/temporal_holdout.csv")
PY

uv run python -m scripts.evaluate --run_id <RUN_ID> --holdout_path data/temporal_holdout.csv
uv run python -m scripts.register_model --run_id <RUN_ID> --name homepage_two_tower --stage Engineering
uv run python -m scripts.backfill_ann --run_id <RUN_ID>

Phase E: Wire Model Alias Into Deployment

Set environment variable used by deploy workflow:

gh variable set MLFLOW_MODEL_NAME --env engineering --body "homepage_two_tower"
gh variable set MLFLOW_MODEL_ALIAS --env engineering --body "engineering"

For production rollout later:

gh variable set MLFLOW_MODEL_NAME --env production --body "homepage_two_tower"
gh variable set MLFLOW_MODEL_ALIAS --env production --body "production"

Deploy with your normal workflow (.github/workflows/deploy.yml).

4. Runtime Processes You Must Keep Running

For stable production-mode behavior, these must be active:

Serving API deployment (FastAPI)
Kafka consumer(s) for event ingestion (scripts.run_kafka_consumer equivalent)
Personalization RQ worker(s) (rq worker hh_lion_personalization)
Periodic feature recomputation jobs
Retrain + reindex operational cadence

If worker/consumer are missing, readiness, queue lag, and model freshness will degrade.

5. Verification Checklist After Deploy

Startup probe:

GET /health/startup returns 200.

Model source:

GET /v2/model/info shows:
- "model_loaded": true
- "model_source": "mlflow"
- "mlflow_model_name": "homepage_two_tower"
- "mlflow_model_alias": "engineering" or "production"