Quickstart

Train a model and serve recommendations locally with the current ClickHouse-backed training flow.

Prerequisite: Complete Installation first.

For complete non-mock operation (Kafka consumer, Feast, feature computation, RQ worker), use Full Localhost Runbook.

For production rollout response, use Canary Rollout and Rollback Runbook.

For incident response playbooks, use Operations Runbook Index.

1. Sync Environment

uv sync --all-extras

2. Verify Infrastructure

docker compose ps
curl -s http://localhost:15000/health && echo " ✅ MLflow"

Expected: core containers running (redis, clickhouse, kafka, schema-registry, minio, mlflow, tei, opensearch)

3. Apply Schema and Start Ingestion

Training reads real booking events from ClickHouse, so seed Redis alone is not enough.

Apply the ClickHouse schema:

uv run python -m scripts.migrations.apply_migrations

Start the Kafka consumer in another terminal:

uv run python -m scripts.run_kafka_consumer

Produce enough booking events for training:

uv run python -m scripts.produce_test_events --event-type booking_confirmed --count 50000

4. Publish Feast V2 Features

Populate the offline and online Feast v2 stores from ClickHouse:

uv run python -m scripts.compute_member_features --mode full --publish-online
uv run python -m scripts.compute_item_features --mode full --publish-online
uv run python -m scripts.compute_cohort_features --mode full --publish-online

5. Train a Model

Model lifecycle step 1: train a model and log the run to MLflow.

export MLFLOW_TRACKING_URI=http://localhost:15000
export MLFLOW_S3_ENDPOINT_URL=http://localhost:19100
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin

TRAINING_DATA_SOURCE=clickhouse uv run python -m scripts.train --config homepage_personalization

Output:

Training complete!
Run ID: abc123def456

📝 Save the Run ID for next steps.

6. Export a Temporal Holdout

scripts/evaluate.py requires an explicit .csv or .parquet holdout file. Export one from ClickHouse:

uv run python - <<'PY'
from recsys.shared.polars import ClickHousePolarsClient

client = ClickHousePolarsClient.from_env()
df = client.query(
    """
    SELECT
        toString(user_id) AS user_id,
        toString(item_id) AS item_id,
        event_timestamp AS timestamp
    FROM interaction_events
    WHERE event_type = 'booking_confirmed'
      AND item_type = 'restaurant'
      AND package_id IS NOT NULL
      AND user_id IS NOT NULL
      AND item_id IS NOT NULL
    ORDER BY event_timestamp
    """
)
df.write_csv("data/temporal_holdout.csv")
print("Wrote data/temporal_holdout.csv")
PY

7. Evaluate the Model

(Model Lifecycle Step 2)

uv run python -m scripts.evaluate --run_id <your_run_id> --holdout_path data/temporal_holdout.csv

Output:

Recall@10: 0.85
NDCG@10: 0.72

8. Register the Model

(Model Lifecycle Step 3)

uv run python -m scripts.register_model --run_id <your_run_id> --name homepage_two_tower --stage Engineering

Promotes to MLflow Engineering.

9. Backfill ANN Index

(Model Lifecycle Step 4)

Generate the Faiss index for fast similarity search:

uv run python -m scripts.backfill_ann --run_id <your_run_id>

Output:

✓ Saved Faiss index to data/ann/index.faiss

10. Serve the API

With trained model (recommended):

export MLFLOW_MODEL_NAME=homepage_two_tower
export MLFLOW_MODEL_ALIAS=engineering
export MLFLOW_TRACKING_URI=http://localhost:15000
export MLFLOW_S3_ENDPOINT_URL=http://localhost:19100
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export REDIS_HOST=localhost
export REDIS_PORT=16379
export GROWTHBOOK_API_HOST=http://localhost:3100
export GROWTHBOOK_CLIENT_KEY=sdk-12345

uv run uvicorn recsys.serving.app:app --host 0.0.0.0 --port 8000

Development mode (mock data):

uv run uvicorn recsys.serving.app:app --reload

11. Test the API

Health check:

curl http://localhost:8000/health

Homepage recommendations (guest user):

# Example Nano ID values for anonymous_id and session_id
ANON_ID="V1StGXR8_Z5jdHi6B-myT"
SESSION_ID="IRFa-VaY2b8A9xQ7mL0Pn"

curl -X POST "http://localhost:8000/v2/homepage" \
  -H "Content-Type: application/json" \
  -d '{
    "anonymous_id": "'"$ANON_ID"'",
    "session_id": "'"$SESSION_ID"'",
    "context": {"geo": "TH-1", "language": "en", "device_type": "mobile"}
  }'

Expected response:

{
  "request_id": "...",
  "cache_ttl_seconds": 60,
  "sections": [
    {"section_id": "popular_restaurants", "section_type": "restaurant_list", "item_type": "restaurant", "strategy": "cohort_popularity", "...": "..."},
    {"section_id": "trending_now", "section_type": "restaurant_list", "item_type": "restaurant", "...": "..."}
  ],
  "personalization_pending": false
}

⚠️ Note: anonymous_id and session_id must be valid Nano ID values using the default 21-character URL-safe alphabet.

The API also persists anonymous_id in a cookie (HttpOnly by default; Secure in production). Set ANONYMOUS_ID_COOKIE_HTTPONLY=false only when frontend JavaScript access is required.

12. Open Dashboards

✅ Sanity Check Complete

• Docker services running
• Redis seeded with sample data
• Model trained
• Model evaluated
• Model registered
• API serving
• Homepage returns sections

You’re ready to develop!

Quick Reference

CatBoost ranker extraction requires non-empty request_id values so training groups, offline CTR/CVR proxy denominators, and live canary attribution all describe the same request-level choice sets. Do not pass --limit for production extraction; it is only a development sampling knob because truncating the event window can drop candidates or post-exposure label events and bias offline CTR/CVR proxy evidence. The extractor writes graded relevance labels.npy for CatBoost/NDCG plus binary click_labels.npy for CTR proxy metrics, so non-click label upgrades are not counted as clicks. Training also requires dataset_metadata.json to prove that every saved exposure had a full label attribution window before offline CTR/CVR proxy gates run. The offline baseline uses retrieval scores only when every candidate in a request has a finite retrieval score; otherwise the whole request falls back to inverse logged position so baseline deltas do not mix incompatible score scales. CatBoost ranker training passes explicit unit CatBoost group_weight values and sampling_unit=Group by default so every request-level choice set is weighted and sampled as the ranking unit. Do not use inverse group-size group_weight values; request-level equalization belongs in offline metrics, while CatBoost requires all GroupWeight values inside a group to be equal. Saved ranker metadata and MLflow runs record whether the offline promotion gate passed, was skipped, or was not recorded; strict serving requires the compatible metadata.json sidecar, and only passed gate metadata is promotion evidence.

Identity Linking

The unified identity system links guest browsing history to member accounts.

Link identity on signup/login:

curl -X POST "http://localhost:8000/v2/identity/link" \
  -H "Content-Type: application/json" \
  -d '{
    "anonymous_id": "'"$ANON_ID"'",
    "user_id": "user_12345"
  }'

Resolve identity (check if guest is linked to member):

curl -X POST "http://localhost:8000/v2/identity/resolve" \
  -H "Content-Type: application/json" \
  -d '{
    "anonymous_id": "'"$ANON_ID"'"
  }'

Unlink identity (privacy compliance):

curl -X DELETE "http://localhost:8000/v2/identity/link/$ANON_ID"

Prometheus Metrics

Prometheus metrics are available at /metrics:

curl http://localhost:8000/metrics

Key metrics:

• hh_lion_request_latency_seconds - Total request latency
• hh_lion_ranking_latency_seconds - Ranker inference latency
• hh_lion_identity_link_operations_total - Identity link/unlink operations
• hh_lion_ranker_fallback_total - Ranker fallback events

Support

Slack: #team-end-user