ClickStack on EKS Dev — Runbook

Status (2026-06-05): Sandbox stack is fully deployed end-to-end on eks-dev-262. ClickHouse + HyperDX accessible via Cloudflare Tunnel + Zero Trust Access. HyperDX is wired to MongoDB Atlas (not in-cluster MongoDB).

What is deployed

Login to HyperDX

There is no direct HyperDX username/password. Auth is delegated to Cloudflare Access:

1. Open https://hyperdx.hhstaging.dev/
2. CF Access redirects to its login page
3. Choose Google SSO (Workspace account) or email OTP (one-time code to your @hungryhub.com address)
4. After auth you land on the HyperDX UI

Configuration sources

All cluster-side config flows from:

The Atlas URI is read by clickhouse.tf via data "aws_secretsmanager_secret_version" at apply time and templated into clickhouse.yaml.tft as hyperdx.config.MONGO_URI. Never stored in TF state or git.

Re-apply procedure (sandbox dev)

# 0. Verify your AWS SSO + gh auth
aws sso login --profile sandbox
gh auth status  # should be saiqulhaq-hh

# 1. Update Cloudflare (tunnel + Access + DNS) — if any of those changed
gh workflow run terraform.yml --ref main \
  --field environment=dev \
  --field services=hh-cloudflare \
  --field cloudflare_deployment_target=hhstaging-dev \
  --field cloudflare_auto_approve=true \
  --field action=apply

# 2. Update EKS (cloudflared pods) — needs tunnel_token in tfvars infra
gh workflow run terraform.yml --ref main \
  --field environment=dev \
  --field services=eks-services \
  --field action=apply

# 3. Update ClickStack (Helm release) — needs CLICKSTACK_MONGO_URI_DEV in /hungryhub/dev
gh workflow run terraform.yml --ref main \
  --field environment=dev \
  --field services=helm-releases \
  --field action=apply

# 4. Force rollout restart to pick up new configmap (env vars are loaded at pod start)
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  rollout restart deployment -n clickhouse clickstack-app

Verification

# Tunnel healthy?
TOKEN_ID="79ea58f8-565f-465c-ad5d-7d6242770f84"
ACCOUNT_ID="6bacbc90386bf7634898c165b7dbc611"
CF_TOKEN="cfut_rZjinOgbMyqbpJAZ13yEgCHZH8yKEEJJOIBzwFMN25448ff5"
curl -s "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/cfd_tunnel/${TOKEN_ID}" \
  -H "Authorization: Bearer ${CF_TOKEN}" | python3 -c \
  "import sys,json; d=json.load(sys.stdin)['result']; print(d['status'], len(d['connections']))"

# Endpoints respond?
curl -sI https://clickhouse.hhstaging.dev/  # 302 → CF Access login (tunnel working)
curl -sI https://hyperdx.hhstaging.dev/     # 302 → CF Access login

# HyperDX pod is on Atlas?
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  exec -n clickhouse deploy/clickstack-app -- printenv MONGO_URI
# should print mongodb+srv://clickstack_dev:***@staging.owjxrkd.mongodb.net/hyperdx?...

# HyperDX connected to Atlas (look for "Connection established to MongoDB")
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  logs -n clickhouse deploy/clickstack-app --tail=50 | grep -E "MongoDB|Atlas"

Known caveats / pitfalls

1. Chart values path is hyperdx.config, not hyperdx.configmap. The v2.x chart reads non-sensitive env vars from hyperdx.config.<KEY>. The old v1.x path hyperdx.configmap.<KEY> is silently ignored — the chart falls back to its in-cluster MongoDB defaults, which would point at the orphaned clickstack-mongodb-svc:27017. See PR #362 + commit edd0095 for the fix.

2. mongodb.enabled: false does NOT remove the wait-for-mongodb init container in chart v3.0.0. Even with mongodb.enabled: false, the chart’s deployment template renders the init container until nc -z clickstack-mongodb-svc 27017, which never succeeds when the in-cluster MongoDB is gone. The pod gets stuck in Init:0/1 forever. Fix: set hyperdx.deployment.initContainers: [] explicitly in the values to drop the init container. See commit f08221a.

3. mongodb.enabled: false does not remove the in-cluster MongoDB subchart resources. The chart still renders the MongoDBCommunity CR, the StatefulSet, the service, and the operator. The MCK operator then creates the actual MongoDB pods. To get rid of in-cluster MongoDB you need to either delete the MongoDBCommunity CR (which the operator watches) or uninstall the chart. We chose the former — see “Cleanup” below.

4. hyperdx.config.FRONTEND_URL MUST match the public URL the browser uses (NOT hyperdx.frontendUrl — that key does not exist in the v3.0.0 chart, it’s silently ignored). HyperDX reads this env var for: (a) the session cookie’s Domain attribute, (b) constructing invite/reset links shown in the Copy Link button. If unset (chart default = http://localhost:3000), the browser refuses to send the cookie to hyperdx.hhstaging.dev (login spinner hangs) and the invite link points at localhost. Set it via hyperdx.config.FRONTEND_URL: "https://hyperdx.hhstaging.dev" in the values. See commit 1e282a4.

5. cloudflared needs --metrics 0.0.0.0:2000 for the liveness probe to reach the metrics/ready endpoint. The default is localhost:2000, which the kubelet can’t reach via the pod IP. The pod would be killed ~36s after start. See commit 226e462 for the fix.

6. ACM validation CNAMEs need lifecycle.ignore_changes = [content]. CloudFlare normalises the trailing dot on ACM tokens, causing a perpetual no-op diff. See commit a383599.

7. Pre-existing DNS records must be imported into Terraform state before the first apply. Otherwise the apply fails with CF error 81053 (“A, AAAA, or CNAME record with that host already exists”). Use terraform import cloudflare_dns_record.<name> <zone_id>/<record_id> for each existing record.

8. Ruleset name mismatches need lifecycle.ignore_changes = [name]. Rulesets created via API have "...via API" suffix; the module writes "...via Terraform". Combined with lifecycle.prevent_destroy, this would block the apply. See commit 91e4caf.

9. Helm timeouts: the clickstack chart deploys many CRDs/operators and can recreate StatefulSets, so the default 5min timeout is insufficient. Bump timeout = 900 (15 min) on the helm_release resource. If a helm upgrade still times out, run helm rollback clickstack <last-deployed-revision> to recover, then delete the half-applied release secret (kubectl delete secret -n clickhouse sh.helm.release.v1.clickstack.v<stuck-rev>) before re-applying. See commit 08b3f49.

10. Helm frontendUrl change doesn’t take effect on kubectl rollout restart alone — the values change only propagates to the configmap when the chart’s release is re-applied via Terraform. After a manual kubectl rollout restart you can verify by reading the configmap (kubectl get cm -n clickhouse clickstack-config -o yaml) and confirming the new keys are present.

SMTP / outbound email (team invites, alerts)

HyperDX uses nodemailer for outbound email. The SMTP transport only activates if the SMTP env vars are set; otherwise the system silently drops the email (e.g. team invites get created in MongoDB but no email is sent).

Configuration

• SMTP_HOST = smtp.sendgrid.net (SendGrid SMTP relay)
• SMTP_PORT = 587 (TLS)
• SMTP_USER = apikey (literal string)
• SMTP_PASSWORD = <SendGrid API key> — read from /hungryhub/{env} Secrets Manager key SENDGRID_API_KEY at apply time, templated into hyperdx.config.SMTP_PASSWORD in the values
• SMTP_FROM = no-reply@hungryhub.email — must be a verified Single Sender in SendGrid (Settings → Sender Authentication). If not verified, SendGrid will reject the send with 553 Unverified email address.

Verify in pod

NEW_POD=$(kubectl get pods -n clickhouse -l app=clickstack-app -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n clickhouse $NEW_POD -- printenv SMTP_HOST SMTP_PORT SMTP_USER SMTP_FROM
kubectl exec -n clickhouse $NEW_POD -- sh -c 'echo "SMTP_PASSWORD length: ${#SMTP_PASSWORD}"'
# Expected: smtp.sendgrid.net / 587 / apikey / no-reply@hungryhub.email / 69

Verify in configmap

kubectl get cm -n clickhouse clickstack-config -o yaml | grep -E "SMTP_"
# Should show SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, SMTP_FROM

Test the invite flow

1. Log in to HyperDX

2. Team Settings → Invite Team Member → enter your email

3. Check inbox (and spam) for an email from no-reply@hungryhub.email

4. If no email arrives, check the pod logs:

   kubectl logs -n clickhouse -l app=clickstack-app --tail=100 | grep -iE "smtp|email|nodemailer|invite"
   

   Common errors: Invalid login: 535 Authentication failed (bad API key), 553 Unverified email address (SMTP_FROM not verified in SendGrid), Connection timeout (EKS NAT egress blocked — already allowlisted to 13.228.95.112 but SendGrid IPs are different and may need allowlisting in your SG).

5. CF API token scopes vary by purpose. CLOUDFLARE_TOKEN_ACCOUNT_LEVEL (in /home/chucky/projects/github.com/hungryhub-team/hh-infra/.env) has the account-level cfd_tunnel + DNS scopes. It does NOT have Zone:Zone WAF (rulesets) or Account:Access:Policies by default. If you hit 403 on ruleset or access resources, add those scopes to the token in the Cloudflare dashboard.

Cleanup (in-cluster MongoDB)

If mongodb.enabled: false is set but the in-cluster MongoDB is still running (left over from a previous apply with enabled: true), the cleanest way to remove it is:

# 1. Delete the MongoDBCommunity CR — this releases the StatefulSet
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  delete mongodbcommunity clickstack-mongodb -n clickhouse

# 2. Now safely delete the rest
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  delete statefulset,svc,deployment -n clickhouse \
    clickstack-mongodb clickstack-mongodb-arb clickstack-mongodb-svc mongodb-kubernetes-operator

# 3. Secrets
for s in clickstack-mongodb-agent-password clickstack-mongodb-config \
         clickstack-mongodb-hyperdx-hyperdx clickstack-mongodb-keyfile \
         clickstack-mongodb-password clickstack-mongodb-scram-scram-credentials; do
  kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
    delete secret -n clickhouse $s
done

# 4. ClusterRoles + ClusterRoleBindings
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  delete clusterrole mongodb-kubernetes-operator-clickhouse-cluster-mongodb-role \
                    mongodb-kubernetes-operator-clickhouse-webhook-cr \
                    mongodb-kubernetes-operator-cluster-telemetry
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  delete clusterrolebinding mongodb-kubernetes-operator-clickhouse-cluster-mongodb-role-binding \
                           mongodb-kubernetes-operator-clickhouse-cluster-telemetry-binding \
                           mongodb-kubernetes-operator-clickhouse-webhook-crb

# 5. PVCs
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  delete pvc -n clickhouse data-volume-clickstack-mongodb-0 logs-volume-clickstack-mongodb-0

The order matters — delete the CR first, otherwise K8s garbage collection won’t release the StatefulSet-owned PVCs.

Issue tracking

EPIC: https://github.com/hungryhub-team/hungryhub-terraform/issues/339