LiteLLM + Langfuse + Grafana — LLM Gateway Stack on EKS Dev

Status (2026-06-23): Fully deployed and live on eks-dev-262 (sandbox). All three services accessible via Cloudflare Tunnel + Zero Trust Access. Langfuse blob storage moved from in-cluster MinIO to real S3 (2026-06-23) — see §8 Blob storage: real S3, not in-cluster MinIO.

Live URLs

All URLs gated by Cloudflare Zero Trust Access — @hungryhub.com (Google SSO or email OTP).

Architecture

Developers (30×)
    │  Virtual API Keys (per-dev)
    ▼
┌─────────────────────────────────────┐
│  LiteLLM Proxy                      │  litellm.hhstaging.dev
│  Namespace: litellm                 │  (OpenAI-compatible, port 4000)
│  - 2 replicas, HPA 2→4             │
│  - Per-dev budget & rate limiting   │
│  - Prometheus /metrics              │
│  - Callbacks: langfuse, prometheus  │
└────────────────┬────────────────────┘
                 │
    ┌────────────┼────────────────┐
    │            │                │
    ▼            ▼                ▼
 MiniMax     Langfuse          Prometheus
 api.minimax  (traces)         (metrics)
 .io/v1                           │
                                  ▼
┌──────────────────────┐   ┌──────────────┐
│  Langfuse v3         │   │  Grafana     │
│  NS: langfuse        │   │  NS: kube-   │
│  langfuse.hhstaging  │   │  prometheus  │
│  .dev                │   │  -stack      │
│                      │   │  grafana.    │
│  - Web (Next.js)     │   │  hhstaging   │
│  - Worker            │   │  .dev        │
│  - PostgreSQL        │   └──────────────┘
│  - Redis             │
│  ───────────────     │
│  Blob storage:       │
│  S3 (sandbox)        │
│  hungryhub-dev-      │
│  langfuse-blobs      │
│  (7-day lifecycle)   │
│  ───────────────     │
│  ClickHouse: SHARED  │
│  (clickhouse NS)     │
└──────────────────────┘

Components

Terraform & PRs

File Layout

hungryhub-terraform/
├── s3/
│   └── s3_langfuse_blobs.tf          # Bucket + lifecycle + bucket policy
├── helm-releases/
│   ├── langfuse.tf                   # helm_release + data "aws_iam_user" lookup
│   │                                # + aws_iam_access_key (dev-only)
│   ├── litellm.tf                    # helm_release + namespace + K8s secret (dev-only)
│   ├── grafana.tf                    # ServiceMonitor for LiteLLM (dev-only)
│   ├── charts/litellm-helm/          # Vendored OCI chart (v1.88.1)
│   ├── values/langfuse.yaml          # Langfuse values (S3 config, no chart-bundled MinIO)
│   ├── values/litellm.yaml           # LiteLLM values (MiniMax models)
│   └── values/kube-prometheus-stack-grafana.yaml

hungryhub-iam/
└── stacks/
    └── langfuse-s3-blobs/            # aws_iam_user + inline S3 access policy
                                     # ARN-scoped to the langfuse blobs bucket

hungryhub-terraform/hh-cloudflare/environments/staging/hhstaging-dev/
└── tunnel.tf    # Tunnel ingress + DNS CNAMEs + Zero Trust Access apps/policies

All helm-releases resources are gated: count = var.environment == "dev" ? 1 : 0

Key Decisions & Lessons Learned

1. Shared ClickHouse (not dedicated)

Langfuse uses the existing ClickStack ClickHouse in the clickhouse namespace. This saved 3 ClickHouse + 3 ZooKeeper pods and avoided hitting the cluster node group max size.

• Database: langfuse (12 tables, auto-migrated by Langfuse on first boot)
• User: default (no_password auth, has GRANT ALL ON *.*)
• NetworkPolicy fix required: the clickhouse-restricted-ingress policy must allow the langfuse namespace. Without this, Langfuse web hangs on health checks.

2. Langfuse web requires NODE_OPTIONS (critical)

Langfuse v3.179.1 Next.js hits V8 heap OOM during MCP feature registration (~510MB usage). The default V8 heap limit (~512MB) is insufficient.

Correct fix (chart path matters!):

langfuse:
  web:
    pod:                          # ← must be under pod, not directly under web
      additionalEnv:
        - name: NODE_OPTIONS
          value: "--max-old-space-size=768"

Wrong: langfuse.web.additionalEnv (silently ignored) Right: langfuse.web.pod.additionalEnv

Container memory limit must be ≥1Gi to accommodate the 768MB heap + overhead.

3. LiteLLM image requires 2Gi memory

The ghcr.io/berriai/litellm-database image runs Prisma + Python and OOMs at <1Gi during startup.

4. Bitnami images are now paid

LiteLLM’s bundled PostgreSQL/Redis subcharts reference docker.io/bitnami/* which returns “not found”. Override to free bitnamilegacy/*:

postgresql:
  image:
    repository: bitnamilegacy/postgresql
    tag: 17.3.0-debian-12-r1
redis:
  image:
    repository: bitnamilegacy/redis
    tag: 7.4.3-debian-12-r0

5. Separate PostgreSQL instances (keep them)

LiteLLM and Langfuse each have their own PostgreSQL. Don’t merge — different Prisma schemas, conflicting migrations. For production, point both at RDS Aurora with separate databases.

6. Cloudflare deployment prerequisites

• CLOUDFLARE_ACCOUNT_ID must exist in /hungryhub/cloudflare secret (AWS Secrets Manager, sandbox account). Was missing initially — added value 6bacbc90386bf7634898c165b7dbc611.
• API token needs Zero Trust: Edit + Cloudflare Tunnel: Edit account-level permissions (not just DNS).
• DNS records that exist in Cloudflare but not in TF state need import {} blocks.
• Workflow: gh workflow run terraform.yml --ref main --field environment=dev --field services=hh-cloudflare --field cloudflare_deployment_target=hhstaging-dev --field cloudflare_auto_approve=true --field action=apply

7. Langfuse access control

• Sign-up: disabled (signUpDisabled: true)
• Org creation: restricted to saiqulhaq@hungryhub.com
• New users: invited via Langfuse UI → Settings → Members

8. Blob storage: real S3, not in-cluster MinIO

The Langfuse chart ships a bundled langfuse-s3 MinIO StatefulSet backed by a 10Gi gp3 PVC. This filled up in ~9 days of trace accumulation and silently dropped every trace/observation upload with XMinioStorageFull (HTTP 507). The Langfuse API still accepted the trace — LiteLLM’s Langfuse Layer Logging - logging success callback masked the failure — but the underlying S3 PUT to MinIO failed because /dev/nvme1n1 was 100% full. Traces persisted in ClickHouse with broken blob refs and the UI showed nothing.

Fix (2026-06-23, PRs #519 / hungryhub-iam#54 / #55 / #521):

• Langfuse chart is deployed with s3.deploy: false — no in-cluster MinIO subchart, no langfuse-s3 pod, no PVC.
• Blobs land in a real S3 bucket in the sandbox account: hungryhub-dev-langfuse-blobs (region ap-southeast-1).
• 7-day bucket lifecycle expires objects (1-day multipart abort cleanup for incomplete uploads).
• Bucket policy allows only the langfuse-s3-blobs-dev IAM user. ARN-scoped to the bucket + /* (no Resource = "*").
• IAM user lives in hungryhub-iam stack langfuse-s3-blobs/ (proper home for IAM, not in the infra repo). The static access key lives in hungryhub-terraform/helm-releases/langfuse.tf, which looks the user up by name via data "aws_iam_user". Split ownership — see hungryhub-iam langfuse-s3-blobs README §Why this lives here, not in hungryhub-terraform.

Key chart env vars (in helm-releases/values/langfuse.yaml.tft):

langfuse:
  blobStorage:
    s3:
      deploy: false                # skip the chart's bundled MinIO subchart
      enabled: true
      bucket: hungryhub-dev-langfuse-blobs
      region: ap-southeast-1
      endpoint: https://s3.ap-southeast-1.amazonaws.com
      forcePathStyle: false
      accessKeyId: <from aws_iam_access_key.langfuse_s3.id>
      secretAccessKey: <from aws_iam_access_key.langfuse_s3.secret>
      # Per-prefix credentials (chart sets LANGFUSE_S3_EVENT_UPLOAD_*,
      # LANGFUSE_S3_BATCH_EXPORT_*, LANGFUSE_S3_MEDIA_UPLOAD_*):
      eventUpload:
        prefix: events/
      batchExport:
        enabled: true
        prefix: exports/
      mediaUpload:
        enabled: true
        prefix: media/

Apply order (new env): hungryhub-iam langfuse-s3-blobs → hungryhub-terraform s3 → hungryhub-terraform helm-releases. The bucket policy references the IAM user by ARN, so the user must exist in AWS before the s3 apply can succeed (bucket policy apply would fail with MalformedPolicy: Invalid principal otherwise).

Retention caveat — 7 days. This is the only retention mechanism. The small langfuse ClickHouse metadata tables (observations, traces, scores, sessions) have no TTL and will grow unbounded. Add a follow-up migration with ALTER TABLE ... MODIFY TTL toDateTime(created_at) + INTERVAL 7 DAY on the langfuse DB tables.

Credentials

TODO (prod): Move all secrets to AWS Secrets Manager /hungryhub/{env} and read via local._secrets pattern.

Models Configured

- minimax-m2.1           # MiniMax-M2.1 (flagship)
- minimax-m2.1-lightning # MiniMax-M2.1-lightning (fast)
- minimax-m2             # MiniMax-M2

All via https://api.minimax.io/v1 (OpenAI-compatible). API key stored in K8s secret litellm-env-secret.

Operations

Create virtual key for a developer

curl -X POST https://litellm.hhstaging.dev/key/generate \
  -H "Authorization: Bearer sk-hh-litellm-dev-2026" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "dev@hungryhub.com", "max_budget": 50, "budget_duration": "30d"}'

Update MiniMax API key

kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  -n litellm patch secret litellm-env-secret --type='json' \
  -p='[{"op":"replace","path":"/data/MINIMAX_API_KEY","value":"'$(echo -n "YOUR_KEY" | base64)'"}]'
kubectl -n litellm rollout restart deployment litellm

Connect Langfuse to LiteLLM

1. In Langfuse UI: create a project → Settings → API Keys → Create
2. Update LiteLLM secret:

kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  -n litellm patch secret litellm-env-secret --type='json' \
  -p='[{"op":"replace","path":"/data/LANGFUSE_PUBLIC_KEY","value":"'$(echo -n "pk-lf-xxx" | base64)'"},{"op":"replace","path":"/data/LANGFUSE_SECRET_KEY","value":"'$(echo -n "sk-lf-xxx" | base64)'"}]'
kubectl -n litellm rollout restart deployment litellm

Invite a teammate to Langfuse

Sign-up is disabled. Go to Langfuse UI → Settings → Members → Invite by email.

Helm upgrade (after values change)

# Langfuse
helm --kube-context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  upgrade langfuse langfuse/langfuse --version 1.5.34 -n langfuse \
  -f helm-releases/values/langfuse.yaml

# LiteLLM (from repo root, uses vendored chart)
helm --kube-context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  upgrade litellm helm-releases/charts/litellm-helm -n litellm \
  -f helm-releases/values/litellm.yaml

# Grafana
helm --kube-context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --version 86.2.2 -n kube-prometheus-stack \
  -f helm-releases/values/kube-prometheus-stack-grafana.yaml

Troubleshooting

Related Docs

• ClickStack EKS Dev Runbook — shared ClickHouse cluster details
• ClickHouse Access Plan — NetworkPolicy and access architecture
• HungryHub Terraform Architecture — helm-releases module