Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Full Localhost Runbook (No Mock / No Bypass)

This guide runs the complete hh-lion stack on localhost with real infrastructure and a real MLflow model.

Related incident docs:

Scope

  • Uses Docker services from docker-compose.yml:
    • Redis, ClickHouse, Kafka, Schema Registry, MinIO, MLflow, TEI, OpenSearch
  • Uses real feature computation and Feast online serving
  • Uses real model loading (MLFLOW_MODEL_NAME + MLFLOW_MODEL_ALIAS required)
  • Runs background personalization worker (RQ)

Prerequisites

  • Docker Desktop running
  • uv installed
  • Python 3.12 (managed by uv)
  • At least 8 GB RAM free (more is better if all services run together)

1. Setup Project Environment

cp .env.example .env
uv sync --all-extras

Load environment variables in each shell before running commands:

set -a
source .env
set +a

Important no-mock settings:

  • FORCE_MOCK_MODEL=false
  • MLFLOW_MODEL_NAME and MLFLOW_MODEL_ALIAS must be set before starting the API

2. Start Infrastructure

docker compose up -d
docker compose ps

Expected:

  • Running: redis, clickhouse, kafka, schema-registry, minio, mlflow, tei, opensearch
  • Completed once: minio-init

Useful endpoints:

  • MLflow: http://localhost:15000
  • MinIO API: http://localhost:19100
  • MinIO Console: http://localhost:19001
  • ClickHouse HTTP: http://localhost:18123/ping
  • Schema Registry: http://localhost:18081/_health
  • TEI: http://localhost:18080/health
  • OpenSearch: http://localhost:19200

3. Apply ClickHouse Schema Migrations

uv run python -m scripts.migrations.apply_migrations

Sanity check:

uv run python -m scripts.migrations.apply_migrations --dry-run

4. Apply Feast Definitions

uv run python -m scripts.apply_feast_repo

4A. Validate Feast v2 Against Local Production-Like Stores

Before running feature backfills or training, verify that the v2 Feast contract works against the same local ClickHouse and Redis images declared in docker-compose.yml. This gate uses isolated test databases and fixture rows; it does not require customer or production data.

docker compose up -d redis clickhouse
docker compose ps redis clickhouse

Run the v2 integration gate against the compose services:

USE_EXISTING_CLICKHOUSE=1 \
USE_EXISTING_REDIS=1 \
CLICKHOUSE_HOST=localhost \
CLICKHOUSE_PORT=19000 \
CLICKHOUSE_HTTP_PORT=18123 \
CLICKHOUSE_USER="${CLICKHOUSE_USER:-default}" \
CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}" \
CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-liondb_production}" \
REDIS_HOST=localhost \
REDIS_PORT=16379 \
uv run pytest -m "integration and feature_store_v2" -rs -q

Expected result:

  • Feast offline retrieval returns point-in-time historical features from ClickHouse.
  • Feast online retrieval returns the latest published feature values from Redis.
  • Failed online publication raises and writes failed compute/publication audit rows.
  • No test is skipped because ClickHouse or Redis is unavailable.

For a faster developer-only loop, omit USE_EXISTING_CLICKHOUSE and USE_EXISTING_REDIS; pytest will create Docker testcontainers. The compose-backed gate above is the release-safety gate because it exercises the project service versions, ports, and environment wiring.

5. Start Streaming Ingestion Consumer (Terminal A)

Wait for Schema Registry to be healthy before starting the consumer:

# Verify Schema Registry is ready (should return 200)
curl -s -o /dev/null -w '%{http_code}' http://localhost:18081/_health

Then start the consumer:

set -a; source .env; set +a
uv run python -m scripts.run_kafka_consumer

Keep this process running.

6. Produce Test Events Into Kafka (Terminal B)

set -a; source .env; set +a
uv run python -m scripts.produce_test_events --count 50000

If training later says there is insufficient data, run this again with a bigger count.

7. Compute Feast V2 Features

Compute and publish features from ClickHouse to Feast v2:

set -a; source .env; set +a
uv run python -m scripts.compute_member_features --mode full --publish-online
uv run python -m scripts.compute_item_features --mode full --publish-online
uv run python -m scripts.compute_cohort_features --mode full --publish-online

Optional OpenSearch metadata merge into Feast v2 item features:

uv run python -m scripts.compute_item_features --mode full --publish-online --with-opensearch-metadata

8. Train and Register a Real Model

Train from ClickHouse data (not synthetic):

set -a; source .env; set +a
TRAINING_DATA_SOURCE=clickhouse uv run python -m scripts.train --config homepage_personalization

Capture the printed Run ID, export a temporal holdout, then evaluate, register, and backfill the ANN index. These scripts also need the MLflow/MinIO environment:

set -a; source .env; set +a
export MLFLOW_TRACKING_URI=http://localhost:15000
export MLFLOW_S3_ENDPOINT_URL=http://localhost:19100
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin

uv run python - <<'PY'
from recsys.shared.polars import ClickHousePolarsClient

client = ClickHousePolarsClient.from_env()
df = client.query(
    """
    SELECT
        toString(user_id) AS user_id,
        toString(item_id) AS item_id,
        event_timestamp AS timestamp
    FROM interaction_events
    WHERE event_type = 'booking_confirmed'
      AND item_type = 'restaurant'
      AND package_id IS NOT NULL
      AND user_id IS NOT NULL
      AND item_id IS NOT NULL
    ORDER BY event_timestamp
    """
)
df.write_csv("data/temporal_holdout.csv")
print("Wrote data/temporal_holdout.csv")
PY

uv run python -m scripts.evaluate --run_id <RUN_ID> --holdout_path data/temporal_holdout.csv
uv run python -m scripts.register_model --run_id <RUN_ID> --name homepage_two_tower --stage Engineering
uv run python -m scripts.backfill_ann --run_id <RUN_ID>

9. Start Background Personalization Worker (Terminal C)

set -a; source .env; set +a
uv run rq worker hh_lion_personalization --url redis://localhost:16379/0

Keep this process running so /v2/homepage/refresh can complete personalization.

10. Start API in Real Model Mode (Terminal D)

set -a; source .env; set +a
export MLFLOW_MODEL_NAME=homepage_two_tower
export MLFLOW_MODEL_ALIAS=engineering
export FORCE_MOCK_MODEL=false
export MLFLOW_TRACKING_URI=http://localhost:15000
export MLFLOW_S3_ENDPOINT_URL=http://localhost:19100
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export REDIS_HOST=localhost
export REDIS_PORT=16379
export KAFKA_BOOTSTRAP_SERVERS=localhost:19092

uv run uvicorn recsys.serving.app:app --host 0.0.0.0 --port 8000

11. Verify It Is Truly Non-Mock

curl -s http://localhost:8000/v2/model/info | jq
curl -s http://localhost:8000/ | jq

Expected:

  • "model_loaded": true
  • "model_source": "mlflow"
  • "mlflow_model_name": "homepage_two_tower"
  • "mlflow_model_alias": "engineering"
  • Root endpoint shows "model_mode": "production"

If model_source is mock, you are not running in full mode.

Check full readiness (all subsystems including new-items candidates):

curl -s http://localhost:8000/health/ready | jq

Expected: "status": "ready" at the top level and for every check. If new_items_candidates shows an error, ensure ClickHouse migrations were applied (Step 3).

12. API Smoke Tests

Guest request:

ANON_ID="V1StGXR8_Z5jdHi6B-myT"
SESSION_ID_1="IRFa-VaY2b8A9xQ7mL0Pn"
SESSION_ID_2="Qk9LmN0PqRsTuVwXyZ_12"
curl -s -X POST "http://localhost:8000/v2/homepage" \
  -H "Content-Type: application/json" \
  -d '{
    "anonymous_id": "'"$ANON_ID"'",
    "session_id": "'"$SESSION_ID_1"'",
    "context": {"geo": "TH-1", "language": "en", "device_type": "mobile"}
  }' | jq

Member request:

curl -s -X POST "http://localhost:8000/v2/homepage" \
  -H "Content-Type: application/json" \
  -d '{
    "anonymous_id": "'"$ANON_ID"'",
    "user_id": "1",
    "session_id": "'"$SESSION_ID_2"'",
    "context": {"geo": "TH-1", "language": "en", "device_type": "mobile"}
  }' | jq

If response has "personalization_pending": true, poll refresh:

curl -s "http://localhost:8000/v2/homepage/refresh?request_id=<REQUEST_ID>&anonymous_id=<ANON_ID>&session_id=<SESSION_ID_2>" | jq

Alternatively, use the VS Code REST Client with tools/rest.http which has pre-built examples for all API endpoints.

13. Common Failure Modes

  • API starts in mock mode:
    • MLFLOW_MODEL_NAME or MLFLOW_MODEL_ALIAS is missing or invalid
    • FORCE_MOCK_MODEL=true
    • MLflow/MinIO env vars are wrong
  • MLflow artifact download fails:
    • Use MinIO API endpoint http://localhost:19100 (not 19000)
  • Refresh stays pending forever:
    • RQ worker is not running
  • No member/cohort features:
    • Kafka consumer not running when events were produced
    • Feature compute steps were skipped
    • Feast definitions not applied
  • Training fails with insufficient rows:
    • Produce more events and re-run feature jobs
  • /health/ready shows new_items_candidates error:
    • ClickHouse items view is missing — re-run migrations (Step 3)
    • Migration 024_recreate_items_view_from_feature_store_v2.sql recreates the view from Feast v2 latest item tables

14. Stop Everything

docker compose down

Remove volumes too (full reset):

docker compose down -v