HH-Lion Deployment Setup Guide

Step-by-step instructions to deploy hh-lion to engineering (DigitalOcean DOKS) and production (AWS EKS).

Table of Contents

1. Architecture Overview
2. Prerequisites
3. Kubernetes Cluster Setup
4. GitHub Environments Setup
5. GitHub Repository Secrets
6. GitHub Environment Variables (per env)
7. GitHub Environment Secrets (per env)
8. AWS OIDC Setup (Production)
9. Create Kubernetes Namespace
10. Trigger a Deployment
11. Verify Deployment
12. Troubleshooting

1. Architecture Overview

┌─────────────┐    push develop     ┌──────────────────┐
│  Developer   │ ──────────────────► │  GitHub Actions   │
│  / GitHub    │    or manual        │  deploy.yml       │
└─────────────┘    dispatch          └────────┬─────────┘
                                              │
                    ┌─────────────────────────┼──────────────────────────┐
                    │ Engineering                  │ Production               │
                    ▼                          ▼                          │
           ┌──────────────┐          ┌──────────────┐                    │
           │ DOCR Registry│          │ GHCR Registry│                    │
           └──────┬───────┘          └──────┬───────┘                    │
                  │                          │                            │
           ┌──────▼───────┐          ┌──────▼───────┐                    │
           │ DigitalOcean │          │   AWS EKS    │                    │
           │   DOKS       │          │   (OIDC)     │                    │
           └──────────────┘          └──────────────┘                    │
                                                                         │

• Engineering: Auto-deploys on push to develop branch. Images pushed to DigitalOcean Container Registry (DOCR).
• Production: Manual workflow_dispatch only. Images pushed to GitHub Container Registry (GHCR). Auth via OIDC.

2. Prerequisites

Before starting, ensure you have:

• A DigitalOcean account with a managed Kubernetes cluster (DOKS) for engineering
• An AWS account with an EKS cluster for production
• GitHub repository hungryhub-team/hh-lion with admin access
• DigitalOcean CLI (doctl) installed locally (optional, for debugging)
• AWS CLI (aws) installed locally (optional, for debugging)
• kubectl installed locally

3. Kubernetes Cluster Setup

Engineering (DigitalOcean DOKS)

1. Create a DOKS cluster via the DigitalOcean console.
2. Note the Cluster ID (visible in the cluster URL or via doctl kubernetes cluster list).
3. Create a DigitalOcean Container Registry (DOCR) if you don’t have one.
4. Note the Registry URL (e.g., registry.digitalocean.com/hungryhub).

Production (AWS EKS)

1. Create an EKS cluster via AWS Console or Terraform.
2. Note the cluster name and region.
3. Set up OIDC federation (see Section 8).

3a. Optional: Deploy MLflow in Kubernetes (Engineering + Production)

Serving mode in engineering and production requires a real model reference (FORCE_MOCK_MODEL=false and valid MLFLOW_MODEL_NAME + MLFLOW_MODEL_ALIAS). The app needs a reachable MLflow tracking server. This repo now includes:

For copy-paste engineering bootstrap commands (from empty data to non-mock deploy), see Engineering Bootstrap Command Sheet.

For production training-data bootstrap from booking_production, see Production Training Bootstrap Runbook.

• k8s/engineering/mlflow.yaml
• k8s/production/mlflow.yaml

Each env-specific manifest defines the Deployment/Service/Ingress only and expects:

• mlflow-config ConfigMap
• mlflow-secrets Secret

There is no shared root k8s/mlflow.yaml template anymore; k8s/engineering/mlflow.yaml and k8s/production/mlflow.yaml are the canonical manifests for cluster deployments.

You can create those manually (see k8s/README.md) or let GitHub Actions create/update them from environment vars and secrets.

Apply per environment after editing placeholders:

# Engineering cluster
kubectl apply -f k8s/engineering/mlflow.yaml

# Production cluster
kubectl apply -f k8s/production/mlflow.yaml

Then set hh-lion environment values:

• MLFLOW_TRACKING_URI -> your MLflow URL (Ingress host or internal service URL)
• MLFLOW_MODEL_NAME + MLFLOW_MODEL_ALIAS -> preferred registered-model reference for serving (for example homepage_two_tower + production)

The hh-lion serving app no longer needs direct object-store credentials to download model artifacts when the MLflow server runs in artifact-proxy mode. The MLflow server still needs its own R2/S3 credentials.

Aliases used for serving should resolve to model versions logged through the proxied MLflow server. If an alias still points at an older run whose artifact URI is a direct object-store path such as s3://..., the deploy workflow now fails early and you should re-log or re-register that model through the current MLflow deployment before rolling out serving.

To let GitHub Actions deploy MLflow automatically, also set:

• Env var MLFLOW_DEPLOY_ENABLED=true
• Env var MLFLOW_ARTIFACT_ROOT (e.g. s3://hh-lion-bucket-engineering/mlflow/artifacts)
• Optional env var MLFLOW_AWS_REGION (defaults to auto for R2)
• Optional env var MLFLOW_WORKERS (defaults to 1 engineering / 2 production)
• Optional env var MLFLOW_HOSTNAME (overrides ingress host in the manifest)
• Optional env var MLFLOW_SERVER_ALLOWED_HOSTS (for MLflow host header protection; e.g. mlflow-engineering.hh-engineering.my.id)
• Optional env var MLFLOW_SERVER_CORS_ALLOWED_ORIGINS (for browser access; e.g. https://mlflow-engineering.hh-engineering.my.id)
• Env secret MLFLOW_BACKEND_STORE_URI (Postgres connection URI)

Serving access is configured per environment in the serving manifest:

• k8s/engineering/serving.yaml is internal-only behind a ClusterIP service
• k8s/production/serving.yaml is internal-only behind a ClusterIP service

4. GitHub Environments Setup

Go to GitHub → Repository → Settings → Environments and create two environments:

Important: The environment names must be exactly engineering and production (lowercase).

5. GitHub Repository Settings (repo-level)

These are repo-level settings (not environment-specific).

Repository Variables

GitHub → Settings → Secrets and variables → Actions → Variables (Repository variables)

Repository Secrets

GitHub → Settings → Secrets and variables → Actions → Repository secrets

Note: PAT is used for GHCR login (production image push) and for creating the imagePullSecret in EKS. Set repository variable GHCR_USERNAME to the exact GitHub username that owns this PAT; the workflow now fails production deploys if it is missing.

6. GitHub Environment Variables (per env)

Set these under each GitHub Environment (engineering and production separately):

GitHub → Settings → Environments → [engineering/production] → Environment variables

These become a Kubernetes ConfigMap (lion-config) at deploy time.

Keep the storage roles separate when naming databases:

• CLICKHOUSE_DATABASE is the analytics / feature-source database and can remain liondb_engineering / liondb_production.
• MLFLOW_BACKEND_STORE_URI should point to dedicated MLflow metadata databases such as liondb_engineering / liondb_production.
• FEAST_REGISTRY_PATH should point to dedicated Feast registry databases such as liondb_engineering / liondb_production.

For the current Feast architecture, set FEAST_REGISTRY_TYPE=sql and provide a shared registry DSN via the environment secrets below. The deploy workflow now applies the tracked Feast repo explicitly before rolling out serving, and the engineering bootstrap workflow reuses the same registry when refreshing features or training.

The current Feast retrieval contract is:

• Feast online store (Redis) is the required serving path for member, item, and cohort model features.
• Feast offline store (ClickHouse) is the required training and export path through v2 FeatureService contracts.
• Training features must be point-in-time correct. Features that cannot be retrieved through v2 historical tables with feature_timestamp semantics must be excluded from the model contract.
• Direct ClickHouse reads are limited to diagnostics, validation, candidate-source support, and session features that are not part of the Feast model-feature contract. They are not an acceptable substitute for Feast online serving.

The v2 ClickHouse feature tables retain history by entity and feature_timestamp. Feast source definitions must point only at those v2 historical tables, and online publication must occur only after historical coverage, freshness, quality, and recommendation-quality gates pass.

For the current deployment policy, set GROWTHBOOK_DEFAULT_HOMEPAGE_MODEL=member_two_tower in both engineering and production so the service keeps the same fallback behavior across environments when GrowthBook is unavailable.

Kafka Topic Naming Standard

Use the following rule for all Kafka topics managed by hh-lion:

• Effective topic format: <NAMESPACE>.<base-topic>
• Namespace format: <environment>
• Base topic format: lowercase, dot-separated, domain-first

Naming rules:

• Prefer domain-first names such as hh.lion.recsys.impressions
• Use *.events for canonical event streams such as hh.lion.interaction.events
• Use .dlq for dead-letter topics such as hh.lion.interaction.events.dlq
• Do not create new Kafka topics using _
• Do not use camelCase in topic names

Canonical examples for this repo:

• engineering.hh.lion.interaction.events
• engineering.hh.lion.interaction.events.dlq
• engineering.hh.lion.recsys.impressions
• production.hh.lion.interaction.events
• production.hh.lion.interaction.events.dlq
• production.hh.lion.recsys.impressions

7. GitHub Environment Secrets (per env)

Set these under each GitHub Environment (engineering and production separately):

GitHub → Settings → Environments → [engineering/production] → Environment secrets

The deploy workflow mounts optional Kafka certificate material from the lion-kafka-certs secret into /etc/hh-lion/kafka for both serving and consumer pods. Set the *_LOCATION variables above to those in-container paths and provide the corresponding PEM contents through GitHub environment secrets when SSL or mTLS is required.

In addition, set these infrastructure variables per environment:

Engineering environment variables

Production environment variables

8. AWS OIDC Setup (Production)

Production uses OIDC federation instead of static AWS access keys. This is more secure — no long-lived credentials stored in GitHub.

Step 1: Create OIDC Identity Provider in AWS IAM

1. Go to AWS IAM → Identity providers → Add provider
2. Set:
   • Provider type: OpenID Connect
   • Provider URL: https://token.actions.githubusercontent.com
   • Audience: sts.amazonaws.com
3. Click Add provider

Step 2: Create IAM Role

1. Go to IAM → Roles → Create role
2. Choose Web identity
3. Select the OIDC provider you just created
4. Set the trust policy (replace YOUR_GITHUB_ORG and YOUR_REPO):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:hungryhub-team/hh-lion:environment:production"
        }
      }
    }
  ]
}

Step 3: Attach Policies to the Role

Attach these AWS managed policies (or more restrictive custom ones):

• AmazonEKSClusterPolicy — or a custom policy scoped to your cluster
• A custom policy for EKS access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "eks:DescribeCluster",
        "eks:ListClusters"
      ],
      "Resource": "arn:aws:eks:ap-southeast-1:YOUR_ACCOUNT_ID:cluster/YOUR_CLUSTER_NAME"
    }
  ]
}

Step 4: Configure EKS RBAC

The IAM role also needs Kubernetes RBAC permissions. Edit the aws-auth ConfigMap or use EKS access entries:

# Option A: aws-auth ConfigMap
kubectl edit configmap aws-auth -n kube-system

Add under mapRoles:

- rolearn: arn:aws:iam::YOUR_ACCOUNT_ID:role/hh-lion-deploy
  username: hh-lion-deploy
  groups:
    - hh-lion-deployers  # avoid system:masters; bind this group to least-privilege RBAC

Step 5: Save the Role ARN

Copy the role ARN and set it as AWS_ROLE_ARN in the production environment variables (see Section 7).

9. Create Kubernetes Namespace

On both clusters, create the namespace before the first deploy:

# Engineering (connect to DOKS first)
doctl kubernetes cluster kubeconfig save YOUR_CLUSTER_ID
kubectl create namespace hh-lion

# Production (connect to EKS first)
aws eks update-kubeconfig --name YOUR_CLUSTER --region ap-southeast-1
kubectl create namespace hh-lion

The deploy workflow creates the lion-config ConfigMap and lion-secrets Secret automatically on each deploy. You only need to create the namespace manually once.

10. Trigger a Deployment

Engineering (automatic)

Push or merge to the develop branch:

git checkout develop
git push origin develop

The workflow triggers automatically and deploys to engineering.

Production (manual)

1. Go to GitHub → Actions → “Build & Deploy to Kubernetes”
2. Click “Run workflow”
3. Select environment: production
4. Click “Run workflow”

The workflow will:

1. Build the Docker image from recsys/Dockerfile
2. Push to the appropriate registry (DOCR or GHCR)
3. Patch the K8s manifest with the new image tag
4. Create/update the ConfigMap (lion-config) and Secret (lion-secrets)
5. Apply the K8s manifests
6. Wait for rollout to complete

11. Verify Deployment

After the workflow completes:

# Check pods are running
kubectl get pods -n hh-lion

# Check the deployed image
kubectl get deployment hh-lion-api -n hh-lion -o jsonpath='{.spec.template.spec.containers[0].image}'

# Check health endpoint (port-forward for testing)
kubectl port-forward svc/hh-lion 8000:80 -n hh-lion
curl http://localhost:8000/health

12. Troubleshooting

Pod in CrashLoopBackOff

kubectl logs deployment/hh-lion-api -n hh-lion --previous

Common causes:

• Missing or incorrect env vars in ConfigMap/Secret
• ClickHouse or Redis not reachable from the cluster
• Invalid MLFLOW_MODEL_NAME or MLFLOW_MODEL_ALIAS — model version or artifacts not found

ImagePullBackOff

kubectl describe pod -l app=hh-lion -n hh-lion | grep -A5 Events

Common causes:

• Engineering: DOCR token expired or registry URL wrong
• Production: hh-lion-ghcr-secret not created or PAT expired

Workflow Fails at “Configure AWS credentials”

• Verify the OIDC provider is set up in AWS IAM
• Check the trust policy matches repo:hungryhub-team/hh-lion:environment:production
• Ensure AWS_ROLE_ARN environment variable is correct

ConfigMap/Secret Not Updating

The deploy workflow uses --dry-run=client -o yaml | kubectl apply -f - for idempotent updates. If values seem stale:

# Check current configmap values
kubectl get configmap lion-config -n hh-lion -o yaml

# Check secrets exist (values are base64-encoded)
kubectl get secret lion-secrets -n hh-lion -o yaml

Quick Reference: All GitHub Settings

Repository Variables (repo-level)

Repository Secrets (repo-level)

Engineering Environment — Variables

Engineering Environment — Secrets

Production Environment — Variables

Production Environment — Secrets