Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Engineering Bootstrap Command Sheet (Non-Mock, Prefilled)

This is a copy-paste runbook to bootstrap engineering into real model mode when MLflow/ClickHouse are empty.

Use this with:

0. One-Time: Ensure Engineering Kubeconfig

doctl auth init
doctl kubernetes cluster kubeconfig save 8cb14981-6264-45d5-a29b-31be39688e43
kubectl config current-context

1. Load Current Engineering Vars (Prefilled from GitHub Environment)

The following command loads all current engineering GitHub environment variables into your shell:

eval "$(gh variable list --env engineering --json name,value --jq '.[] | \"export \\(.name)=\\(.value|@sh)\"')"

Current values in GitHub engineering (snapshot from February 27, 2026):

export ENVIRONMENT='engineering'
export FORCE_MOCK_MODEL='false'
export CLICKHOUSE_HOST='clickhouse-1c062de2-hh-production.b.aivencloud.com'
export CLICKHOUSE_PORT='25940'
export CLICKHOUSE_HTTP_PORT='25941'
export CLICKHOUSE_SECURE='true'
export CLICKHOUSE_USER='hh_lion'
export CLICKHOUSE_DATABASE='liondb_engineering'
export REDIS_HOST='valkey-hh-search-primary.valkey.svc.cluster.local'
export REDIS_PORT='6379'
export MLFLOW_TRACKING_URI='http://mlflow:5000'
export MLFLOW_S3_ENDPOINT_URL='https://6bacbc90386bf7634898c165b7dbc611.r2.cloudflarestorage.com/hh-lion-bucket-engineering'
export MLFLOW_MODEL_NAME='homepage_two_tower'
export MLFLOW_MODEL_ALIAS='engineering'

Storage naming in engineering should now be interpreted as:

  • ClickHouse analytics DB: liondb_engineering
  • MLflow metadata DB inside MLFLOW_BACKEND_STORE_URI: liondb_engineering
  • Feast registry DB inside FEAST_REGISTRY_PATH: liondb_engineering

Add missing local-only vars/secrets for bootstrap commands:

read -rsp "CLICKHOUSE_PASSWORD: " CLICKHOUSE_PASSWORD; echo
read -rsp "AWS_ACCESS_KEY_ID: " AWS_ACCESS_KEY_ID; echo
read -rsp "AWS_SECRET_ACCESS_KEY: " AWS_SECRET_ACCESS_KEY; echo
export CLICKHOUSE_VERIFY_CERT="${CLICKHOUSE_VERIFY_CERT:-true}"
export NAMESPACE="${NAMESPACE:-engineering}"
export KAFKA_EVENTS_TOPIC="${KAFKA_EVENTS_TOPIC:-hh.lion.interaction.events}"
export KAFKA_CONSUMER_GROUP="${KAFKA_CONSUMER_GROUP:-feature-store-consumer-engineering-bootstrap}"
export KAFKA_DLQ_TOPIC="${KAFKA_DLQ_TOPIC:-hh.lion.interaction.events.dlq}"
export KAFKA_IMPRESSION_TOPIC="${KAFKA_IMPRESSION_TOPIC:-hh.lion.recsys.impressions}"
export KAFKA_IMPRESSION_SCHEMA_PATH="${KAFKA_IMPRESSION_SCHEMA_PATH:-$(pwd)/specs/018-recsys-audit-remediation/contracts/impression-event.avsc}"

If you want engineering bootstrap to enrich Feast v2 item metadata from engineering OpenSearch, set these before running the feature jobs locally:

export OPENSEARCH_NODE='https://os-hh-search-engineering.hh-engineering.my.id:443'
export OPENSEARCH_VERIFY_CERTS='true'
export OPENSEARCH_INDEX='engineering_restaurants'
read -rsp "OPENSEARCH_USER: " OPENSEARCH_USER; echo
read -rsp "OPENSEARCH_PASSWORD: " OPENSEARCH_PASSWORD; echo

With those OPENSEARCH_* variables present, scripts.generate_engineering_kafka_data now also uses the active documents from the engineering alias as its restaurant ID pool. If you explicitly want the old synthetic IDs for a local run, add --restaurant-id-source synthetic.

Use the actual engineering alias name if it differs, but keep the engineering_ prefix so the bootstrap job reads from the engineering alias instead of the default production_restaurants alias.

Set Kafka endpoints explicitly (required for event flow):

export KAFKA_BOOTSTRAP_SERVERS='<your-engineering-kafka-bootstrap>'
export KAFKA_SCHEMA_REGISTRY_URL='<your-engineering-schema-registry-url>'

Set the shared Feast registry DSN before any feature refresh or training step. This should match the engineering GitHub environment secrets used by the workflows:

read -rsp "FEAST_REGISTRY_PATH: " FEAST_REGISTRY_PATH; echo
export FEAST_REGISTRY_TYPE='sql'
export FEAST_REGISTRY_SQLALCHEMY_POOL_PRE_PING='true'
# Optional read replica:
# read -rsp "FEAST_REGISTRY_READ_PATH: " FEAST_REGISTRY_READ_PATH; echo

For engineering, use a registry DSN that targets liondb_engineering, for example:

export FEAST_REGISTRY_PATH='postgresql://user:pass@host:5432/liondb_engineering?sslmode=require'

With the defaults above, the effective engineering topic names become:

  • engineering.hh.lion.interaction.events
  • engineering.hh.lion.interaction.events.dlq
  • engineering.hh.lion.recsys.impressions

If training from your laptop, port-forward MLflow first (because MLFLOW_TRACKING_URI=http://mlflow:5000 is in-cluster):

kubectl -n hh-lion port-forward svc/mlflow 15000:5000

In a second shell:

export MLFLOW_TRACKING_URI='http://localhost:15000'

2. Sanity Check Connections

ClickHouse native + HTTP:

uv run python - <<'PY'
import os
from clickhouse_driver import Client
import clickhouse_connect

host = os.environ["CLICKHOUSE_HOST"]
port = int(os.environ["CLICKHOUSE_PORT"])
http_port = int(os.environ["CLICKHOUSE_HTTP_PORT"])
user = os.environ["CLICKHOUSE_USER"]
password = os.environ["CLICKHOUSE_PASSWORD"]
database = os.environ["CLICKHOUSE_DATABASE"]
secure = os.environ.get("CLICKHOUSE_SECURE", "false").lower() == "true"
verify = os.environ.get("CLICKHOUSE_VERIFY_CERT", "true").lower() == "true"

native = Client(host=host, port=port, user=user, password=password, database=database, secure=secure, verify=verify)
print("native:", native.execute("SELECT 1"))

http = clickhouse_connect.get_client(host=host, port=http_port, username=user, password=password, database=database, secure=secure, verify=verify)
print("http:", http.query("SELECT 1").result_rows)
PY

Kafka + schema registry:

uv run python - <<'PY'
import os
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from recsys.shared.kafka_config import build_kafka_producer_config, build_schema_registry_config

Producer(build_kafka_producer_config(os.environ["KAFKA_BOOTSTRAP_SERVERS"]))
SchemaRegistryClient(build_schema_registry_config(os.environ["KAFKA_SCHEMA_REGISTRY_URL"]))
print("kafka+schema-registry: ok")
PY

3. Apply Schema + Feature Definitions

uv run python -m scripts.migrations.apply_migrations
uv run python -m scripts.apply_feast_repo

4. Start Event Ingestion and Seed ClickHouse

Terminal A (keep running):

uv run python -m scripts.run_kafka_consumer --json-logs

Terminal B:

uv run python -m scripts.produce_test_events --event-type booking_confirmed --count 50000

Check counts:

curl -sS -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" \
  "https://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?database=${CLICKHOUSE_DATABASE}&query=SELECT%20count()%20FROM%20interaction_events"

curl -sS -u "${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}" \
  "https://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?database=${CLICKHOUSE_DATABASE}&query=SELECT%20count()%20FROM%20interaction_events%20WHERE%20event_type%3D%27booking_confirmed%27%20AND%20item_type%3D%27restaurant%27%20AND%20package_id%20IS%20NOT%20NULL%20AND%20user_id%20IS%20NOT%20NULL%20AND%20item_id%20IS%20NOT%20NULL"

5. Rebuild Derived Recommendation Data With Raw IDs

Restaurant IDs are raw source-system IDs. Before validating Tiger-facing flows, reset and refill derived recommendation feature outputs so staging does not mix old and new ID formats.

uv run python - <<'PY'
import os

import clickhouse_connect

client = clickhouse_connect.get_client(
  host=os.environ["CLICKHOUSE_HOST"],
  port=int(os.environ["CLICKHOUSE_HTTP_PORT"]),
  username=os.environ["CLICKHOUSE_USER"],
  password=os.environ["CLICKHOUSE_PASSWORD"],
  database=os.environ["CLICKHOUSE_DATABASE"],
  secure=os.getenv("CLICKHOUSE_SECURE", "true").lower() == "true",
)

client.command("SET mutations_sync = 2")

checks = {
  "interaction_events": """
    SELECT count()
    FROM interaction_events
    WHERE item_type = 'restaurant'
      AND item_id IS NOT NULL
      AND NOT match(toString(item_id), '^[0-9]+$')
  """,
  "cohort_candidate_sets_v2": """
    SELECT count()
    FROM cohort_candidate_sets_v2
    ARRAY JOIN item_ids AS item_id
    WHERE item_id IS NOT NULL
      AND NOT match(toString(item_id), '^[0-9]+$')
  """,
  "item_popularity_feature_latest_v2": """
    SELECT count()
    FROM item_popularity_feature_latest_v2
    WHERE NOT match(toString(item_id), '^[0-9]+$')
  """,
}

for name, query in checks.items():
  remaining = client.query(query).first_row[0]
  print(f"{name}: remaining non-raw restaurant ID rows = {remaining}")
  if remaining != 0:
    raise SystemExit(f"Raw-ID validation failed for {name}")
PY

Optional validation for the exact training slice:

uv run python - <<'PY'
import os

import clickhouse_connect

client = clickhouse_connect.get_client(
  host=os.environ["CLICKHOUSE_HOST"],
  port=int(os.environ["CLICKHOUSE_HTTP_PORT"]),
  username=os.environ["CLICKHOUSE_USER"],
  password=os.environ["CLICKHOUSE_PASSWORD"],
  database=os.environ["CLICKHOUSE_DATABASE"],
  secure=os.getenv("CLICKHOUSE_SECURE", "true").lower() == "true",
)

result = client.query(
  """
  SELECT
    count() AS total,
    countIf(NOT match(toString(item_id), '^[0-9]+$')) AS non_raw_item_id,
    countIf(match(toString(item_id), '^[0-9]+$')) AS numeric_like,
    countIf(NOT match(toString(user_id), '^[0-9]+$')) AS non_raw_user_id,
    uniqExact(item_id) AS uniq_items
  FROM interaction_events
  WHERE event_type = 'booking_confirmed'
    AND item_type = 'restaurant'
    AND package_id IS NOT NULL
    AND user_id IS NOT NULL
    AND item_id IS NOT NULL
    AND event_timestamp >= now() - INTERVAL 90 DAY
  """
)
print(result.result_rows[0])
PY

Expected result: rest_like = 0; numeric_like should contain the raw restaurant training corpus.

6. Build Features

uv run python -m scripts.compute_member_features --mode full --publish-online
uv run python -m scripts.compute_item_features --mode full --publish-online --with-opensearch-metadata
uv run python -m scripts.compute_cohort_features --mode full --publish-online

7. Train Real Model and Capture Run ID

TRAINING_DATA_SOURCE=clickhouse uv run python -m scripts.train --config homepage_personalization | tee /tmp/hh-lion-train.log
RUN_ID="$(grep -Eo 'Run ID: [0-9a-f]+' /tmp/hh-lion-train.log | awk '{print $3}' | tail -1)"
echo "RUN_ID=${RUN_ID}"
test -n "${RUN_ID}"

8. Evaluate, Register, Backfill ANN

uv run python - <<'PY'
from recsys.shared.polars import ClickHousePolarsClient

client = ClickHousePolarsClient.from_env()
df = client.query(
    """
    SELECT
        toString(user_id) AS user_id,
        toString(item_id) AS item_id,
        event_timestamp AS timestamp
    FROM interaction_events
    WHERE event_type = 'booking_confirmed'
      AND item_type = 'restaurant'
      AND package_id IS NOT NULL
      AND user_id IS NOT NULL
      AND item_id IS NOT NULL
    ORDER BY event_timestamp
    """
)
df.write_csv("data/temporal_holdout.csv")
print("Wrote data/temporal_holdout.csv")
PY

uv run python -m scripts.evaluate --run_id "${RUN_ID}" --holdout_path data/temporal_holdout.csv
uv run python -m scripts.register_model --run_id "${RUN_ID}" --name homepage_two_tower --stage Engineering
uv run python -m scripts.backfill_ann --run_id "${RUN_ID}"

8. Promote Model Alias to Engineering and Deploy

gh variable set MLFLOW_MODEL_NAME --env engineering --body "homepage_two_tower"
gh variable set MLFLOW_MODEL_ALIAS --env engineering --body "engineering"

Recommended: also set missing engineering vars used by runtime/deploy pipeline:

gh variable set CLICKHOUSE_HTTP_PORT --env engineering --body "25941"
gh variable set CLICKHOUSE_SECURE --env engineering --body "true"
gh variable set CLICKHOUSE_VERIFY_CERT --env engineering --body "true"
gh variable set OPENSEARCH_NODE --env engineering --body "https://os-hh-search-engineering.hh-engineering.my.id:443"
gh variable set OPENSEARCH_VERIFY_CERTS --env engineering --body "true"
gh variable set OPENSEARCH_INDEX --env engineering --body "engineering_restaurants"
gh variable set KAFKA_BOOTSTRAP_SERVERS --env engineering --body "${KAFKA_BOOTSTRAP_SERVERS}"
gh variable set KAFKA_SCHEMA_REGISTRY_URL --env engineering --body "${KAFKA_SCHEMA_REGISTRY_URL}"
gh variable set KAFKA_EVENTS_TOPIC --env engineering --body "${KAFKA_EVENTS_TOPIC}"
gh variable set KAFKA_CONSUMER_GROUP --env engineering --body "${KAFKA_CONSUMER_GROUP}"
gh variable set KAFKA_DLQ_TOPIC --env engineering --body "${KAFKA_DLQ_TOPIC}"
gh variable set KAFKA_IMPRESSION_TOPIC --env engineering --body "${KAFKA_IMPRESSION_TOPIC}"

Store the engineering OpenSearch credentials as environment secrets so lion-secrets can inject them into the bootstrap job and serving pods:

gh secret set OPENSEARCH_USER --env engineering
gh secret set OPENSEARCH_PASSWORD --env engineering

Keep CLICKHOUSE_VERIFY_CERT=true in engineering. The app defaults to certificate verification, and an empty value can disable TLS verification in some ClickHouse code paths.

This only changes bootstrap item metadata enrichment. The emergency homepage fallback still uses data/fallback_restaurants.json unless you separately replace the fallback loader path.

For engineering Redpanda with SASL, leave these GitHub environment variables unset unless you switch to mounted CA/client certificate files:

  • KAFKA_SSL_CA_LOCATION
  • KAFKA_SSL_CERT_LOCATION
  • KAFKA_SSL_KEY_LOCATION
  • KAFKA_SCHEMA_REGISTRY_SSL_CA_LOCATION
  • KAFKA_SCHEMA_REGISTRY_SSL_CERT_LOCATION
  • KAFKA_SCHEMA_REGISTRY_SSL_KEY_LOCATION

Only set KAFKA_IMPRESSION_SCHEMA_PATH in GitHub environment if that schema file is mounted in the runtime container; otherwise leave it unset and JSON fallback will be used.

Deploy engineering:

gh workflow run deploy.yml -f environment=engineering -f run_training_cpu=false -f run_training_gpu=false

9. Verify Engineering Is Truly Non-Mock

kubectl -n hh-lion rollout status deployment/hh-lion-api --timeout=300s
kubectl -n hh-lion logs deploy/hh-lion-api --tail=200 | rg -n "MLFLOW_MODEL_|mock|faiss|ready|startup|Unexpected packet"

Health/model checks:

kubectl -n hh-lion port-forward svc/hh-lion 18000:80
curl -sS http://localhost:18000/health/startup
curl -sS http://localhost:18000/health/ready
curl -sS http://localhost:18000/v2/model/info

Expected:

  • /health/startup = 200
  • /health/ready = 200
  • /v2/model/info has "model_source":"mlflow", "mlflow_model_name":"homepage_two_tower", and "mlflow_model_alias":"engineering"