ai-text-to-image — EKS Deploy Architecture and Operations

Background

ai-text-to-image is HungryHub’s restaurant menu image-generation service. It runs as a Python/FastAPI app on port 8000. As of 2026-06, the only active image-generation provider is Fal.ai (fal-ai/bytedance/seedream/v4/text-to-image); the Bedrock and Gemini providers remain in the codebase but are not in use. The earlier multi-provider design is documented historically in ai-text-to-image/MULTI_PROVIDER_IMAGE_API.md.

Before 2026-06-10, the service was deployed by hand to a VPS via docker compose (see the legacy ai-text-to-image/README.md §VPS Deployment). There was no CI/CD pipeline, no GitOps, and secrets were managed via .env files in the runtime container and a k8s Secret in the hungryhub namespace — the latter being lost in the 2026-06-08 namespace-deletion incident (see Incident_hungryhub_namespace_deletion_2026-06-08.md).

This document describes the new pipeline: GitHub Actions → ECR → EKS sandbox, with utility namespace-scoped admin for the CI role, and AWS Secrets Manager as the single source of truth for the service’s secrets.

Goals (and the why)

Auto-deploy to EKS sandbox on push to main — every push to the default branch that changes source files should reach the sandbox cluster without a human in the loop. Cuts the lead time on a code change from “ask DevOps” to “merge and see it land”. (Doc-only or k8s/** changes don’t trigger a rebuild — see paths-ignore in .github/workflows/deploy-eks.yml.)
Manual dispatch to EKS prod — prod is gated by the GitHub production environment (and a reviewer, on orgs where the protection rule is supported). Mirrors the hh-lion pattern (see HHLion deployment_setup.md).
Secrets in AWS Secrets Manager, not in k8s — fixes the 2026-06-08 incident root cause (k8s-only secrets get deleted with the namespace). Source-of-truth lives in SM; the deploy workflow mirrors it to a per-deploy k8s Secret.
Namespace-scoped admin, not cluster-admin — the CI role can only touch the utility namespace. Aligns with AGENTS.md §3.6 (“least privilege”) and matches the access-entries pattern already used in eks/iam_users.tf.

Architecture

 push to main      workflow_dispatch
      │                  │
      ▼                  ▼
 ┌───────────────────────────────┐
 │ docker-scan.yml                │
 │   - build final target         │
 │   - push to ECR                │
 │   - Trivy scan (advisory)      │
 └─────────────┬─────────────────┘
               │  079994049689.dkr.ecr.ap-southeast-1.amazonaws.com/ai-text-to-image:<sha>
               │  (sandbox account; prod uses 202255947274)
               ▼
 ┌───────────────────────────────┐
 │ deploy-eks.yml (sandbox)       │
 │   - OIDC → sandbox role        │
 │   - read /ai-text-to-image/    │
 │     sandbox from SM            │
 │   - mirror to k8s Secret       │
 │   - kustomize build overlay    │
 │   - kubectl apply -n utility   │
 │   - in-cluster /health smoke   │
 └─────────────┬─────────────────┘
               │
               ▼
   ┌────────────────────────┐
   │  EKS cluster           │
   │  eks-dev-262           │
   │  namespace: utility    │
   │                        │
   │  Deployments:          │
   │   - ai-text-to-image-  │
   │     web (gunicorn :8000)│
   │  (image-server         │
   │   dropped — see        │
   │   §"Image-server       │
   │   removal")            │
   │                        │
   │  Services / Ingress:   │
   │   - ClusterIP web      │
   │   - ALB ingress        │
   │     ai-image-sandbox   │
   │     .hungryhub.internal│
   │     (prod: ai-image    │
   │      .hungryhub.com)   │
   └────────────────────────┘

   Other in-cluster services (e.g. hh-menu) reach the
   ai-text-to-image service over the private cluster network
   via:
       http://ai-text-to-image-web.utility.svc.cluster.local/api/v1/hh-menu/generate-image

   This URL is configured in the `terraform/tfvars/{env}/infra`
   SM secret (the `ai_image_service_url` key). It was previously
   `https://menugram.hh-bee.my.id/...` (a separate external
   service that no longer exists); switching to the in-cluster
   URL removed the public-internet dependency and the related
   TLS / DNS / cert-rotation surface area.

Component design

1. CI roles (in `hungryhub-iam/stacks/secrets-manager-access/`)

PR: hungryhub-iam#35

Two new OIDC-trusted roles, ARN-scoped to the secrets they read:

Role	Account	Trust subject	SM read	EKS perms
`github-actions-ai-text-to-image-sandbox`	`079994049689` (sandbox)	`repo:hungryhub-team/ai-text-to-image:ref:refs/heads/main` + `:pull_request` + `:environment:sandbox`	`/ai-text-to-image/*`	`eks:DescribeCluster` on `eks-dev-262`
`github-actions-ai-text-to-image-prod`	`202255947274` (prod)	`repo:hungryhub-team/ai-text-to-image:environment:production`	`/ai-text-to-image/*`	`eks:DescribeCluster` on `eks-prod-21`

Both roles also have an inline ecr-push-ai-text-to-image policy granting ecr:GetAuthorizationToken (account-level) + the six ECR push actions + ecr:GetDownloadUrlForLayer (for Trivy) on arn:aws:ecr:ap-southeast-1:<account>:repository/ai-text-to-image.

Why OIDC subject also includes `:environment:sandbox`

When a workflow job declares environment: <name>, GitHub’s OIDC sub claim becomes repo:<org>/<repo>:environment:<name> — not the bare branch or PR. So a trust policy that only allows repo:<org>/<repo>:ref:refs/heads/main rejects sandbox-env jobs. Adding ...:environment:sandbox to the trust pattern fixes this without broadening access.

This is a non-obvious gotcha — if you copy the trust policy from github-actions-integrations (which doesn’t declare an environment), it will fail with Not authorized to perform sts:AssumeRoleWithWebIdentity in the new EKS deploy workflow.

2. EKS RBAC (in `hungryhub-terraform/eks/` + `eks-services/`)

PR: hungryhub-terraform#416

Two new EKS access entries, scoped to the utility namespace:

# eks/iam_users.tf (excerpt)
ai_image_access_entries_by_env = {
  dev = {
    "ai-text-to-image-sandbox" = {
      principal_arn = "arn:aws:iam::079994049689:role/github-actions-ai-text-to-image-sandbox"
      user_name     = "github-actions-ai-text-to-image-sandbox"
      policy_associations = {
        admin = {
          policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          access_scope = {
            type       = "namespace"
            namespaces = ["utility"]   # ← key: namespace-scoped, not cluster-wide
          }
        }
      }
    }
  }
  prod = { "ai-text-to-image-prod" = { ... } }
}

And the in-cluster RBAC in eks-services/ai-text-to-image-rbac.tf:

resource "kubernetes_role" "ai_image_admin" {
  metadata { name = "ai-text-to-image-admin", namespace = "utility" }
  rule {
    api_groups = ["*"]
    resources  = ["*"]
    verbs      = ["*"]
  }
}

resource "kubernetes_role_binding" "ai_image_admin" {
  metadata { name = "ai-text-to-image-admin", namespace = "utility" }
  role_ref { api_group = "rbac.authorization.k8s.io", kind = "Role", name = "ai-text-to-image-admin" }
  subject {
    api_group = "rbac.authorization.k8s.io"
    kind      = "User"
    name      = "github-actions-ai-text-to-image-sandbox"   # matches user_name in access entry
  }
}

Why both?

The EKS access entry controls IAM-side authentication to the cluster, with access_scope.type = "namespace" restricting what AWS considers “admin” (the entry’s policy only applies in the listed namespaces).
The Kubernetes Role + RoleBinding controls RBAC inside the cluster. The CI identity gets * on * in utility only.

This means the CI role cannot list cluster-scoped resources like nodes — the deploy workflow had to drop kubectl get nodes from the smoke test because it returned Forbidden. The kubeconfig write is the only cluster-scope action the workflow needs.

3. Image registry: ECR (not GHCR)

The original design used GitHub Container Registry (the docker-scan.yml workflow had cr-pat login). That hit two problems:

PAT expiration. The long-lived CR_PAT secret in the repo (set 2025-09-18) had expired by the time of this work — every build failed at Logging into ghcr.io → Error response from daemon: Get "https://ghcr.io/v2/": denied: denied.
ECR is more AWS-native. The EKS node IAM role (eks-dev-2622026030203480121050000000b) already has AmazonEC2ContainerRegistryReadOnly, so kubelets can pull ECR images without any imagePullSecret in k8s. GHCR required managing a per-deploy docker-registry Secret.

The migration: created 4 ECR repos (sandbox + prod, one image each — single image, not per-component), updated the build workflow to push via aws-actions/amazon-ecr-login@v1, and removed the GHCR pull-secret plumbing from the deploy workflow.

ECR push permission: an inline policy on each CI role with the seven ECR actions needed for docker buildx push plus ecr:GetAuthorizationToken on * (the auth-token call is account-scoped, not resource-scoped — so it can’t follow the ARN-scoped pattern; * is the standard practice for GetAuthorizationToken).

4. Secrets management (in AWS Secrets Manager)

Two SM secrets, one per env:

/ai-text-to-image/sandbox  →  ARN: arn:aws:secretsmanager:ap-southeast-1:079994049689:secret:/ai-text-to-image/sandbox-xxxx
/ai-text-to-image/prod     →  ARN: arn:aws:secretsmanager:ap-southeast-1:202255947274:secret:/ai-text-to-image/prod-xxxx

Each holds a JSON object with the keys the app reads from os.getenv: FAL_KEY, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, GEMINI_API_KEY, S3_BUCKET, S3_PREFIX. As of 2026-06-10, only FAL_KEY is required (the others are placeholder for the providers the team isn’t currently using).

Why mirror SM → k8s Secret at deploy time? We initially planned to use the External Secrets Operator (ESO) — k8s resources (SecretStore + ExternalSecret) that pull from SM. But ESO isn’t installed in the EKS clusters. Rather than block on a Terraform PR to install ESO cluster-wide, the deploy workflow does the mirror itself: read SM with the CI role’s secretsmanager:GetSecretValue permission, render a k8s Secret manifest, apply it. The cost is a one-time read on every deploy (cheap) and no automatic rotation; the benefit is one fewer cluster component.

If the SM secret value is REPLACE_ME_* or empty, the workflow fails fast with a ::error:: annotation listing the missing key. The check is non-fatal for keys other than FAL_KEY (warnings, not errors) since the user said only fal.ai is active.

5. Kustomize layout

ai-text-to-image/k8s/:

base/
├── deployment.web.yaml       gunicorn on :8000, non-root, RO root FS
├── service.web.yaml          ClusterIP
├── config.yaml                ConfigMap (S3_PREFIX, LOG_LEVEL, etc.)
├── ingress.yaml               ALB ingress, host placeholder
└── kustomization.yaml         namespace: utility (overridden by overlays)
overlays/
├── sandbox/                   1 replica, dev hostname
│   ├── kustomization.yaml     namespace: utility
│   ├── replicas-patch.yaml
│   └── ingress-host-patch.yaml
└── prod/                      2 replicas, prod hostname
    ├── kustomization.yaml     namespace: utility, 202255947274 ECR
    ├── replicas-patch.yaml
    └── ingress-host-patch.yaml

The deploy workflow’s Set image tag in overlay step does kustomize edit set image <ecr-registry>/ai-text-to-image=<ecr-registry>/ai-text-to-image:<sha> in the env overlay’s directory, then kustomize build .. This pins both the per-component (currently just web) image tags to the commit SHA, even though the kustomize images: block defaults to latest (handy for kubectl diff --dry-run).

Image-server removal (gotcha)

The original Dockerfile had two targets — final (web) and image-server. The deploy workflow assumed both existed and pushed two ECR images. The Dockerfile only ever had the final target. The legacy image-server GHCR image was a separate build that no longer exists in this repo.

The fix: use a single ECR image with a k8s pod-spec command: override on what would have been the image-server Deployment. But then we discovered the image_server.py source file itself doesn’t exist in the repo — the image-server service was never actually used, and the team confirmed Bedrock + Gemini are deprecated in favor of Fal.ai. So the image-server Deployment + Service were dropped entirely (see ai-text-to-image#29). If a future need arises, the legacy image_server.py will need to be added back from the git history first.

Failure modes we hit (and the fixes)

These are the bugs that were discovered during the first sandbox deploy. They’re documented here so the next person doesn’t have to retrace them.

1. IAM policy `Version: "2012-12-02"` typo

PR hungryhub-iam#36.

MalformedPolicyDocument: Syntax errors in policy on three inline role policies. Root cause: a typo in the new jsonencode({Version = "2012-12-02", ...}) blocks — the correct AWS policy version is 2012-10-17. (The pre-existing prod_eks_nodegroup_slr had 2012-10-17 before PR #35; the typo was introduced and propagated to the two new ai_image_*_eks policies.)

Fix: replace all three occurrences with 2012-10-17. The guardrail test in hungryhub-iam/tests/test_iam_guardrails.py only checks Resource = "*" patterns — not Version strings — so this slipped through. Future fix worth adding: extend the guardrail to also assert every jsonencode({Version = ..., ...}) block in the stack uses 2012-10-17.

2. GitHub OIDC TLS thumbprint mismatch

Symptom: every workflow run fails at aws-actions/configure-aws-credentials with Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity.

The IAM OIDC provider in the dev account was created by hungryhub-terraform/oidc-iam-rule/ on 2025-09-22 with thumbprint 22ff89586561fc2d52f77491e9f1eff1b80be33e. GitHub rotates its TLS cert periodically; the current cert has SHA-1 thumbprint 95:14:F4:ED:3C:84:1C:96:C4:3D:EF:0F:0A:CB:F1:77:40:5D:ED:12 (colon-free lowercase: 9514f4ed3c841c96c43def0f0acbf177405ded12). STS validates the OIDC token’s signature against the cert referenced by the IAM provider — so a stale thumbprint rejects all OIDC assumes.

Fix:

aws iam update-open-id-connect-provider-thumbprint \
  --open-id-connect-provider-arn "arn:aws:iam::<account>:oidc-provider/token.actions.githubusercontent.com" \
  --thumbprint-list 9514f4ed3c841c96c43def0f0acbf177405ded12 \
  --profile <profile>

Both home accounts (sandbox, prod) had the same stale thumbprint — fix in both. Future fix worth adding: monitor the OIDC cert and rotate the thumbprint on rotation. GitHub posts a notice in the community forum when this happens.

3. `pkg_resources` removed in setuptools 81+

PR ai-text-to-image#30, ai-text-to-image#32.

gunicorn==20.1.0 imports pkg_resources at startup. The setuptools>=78.1.1 constraint in requirements.txt was unintentionally future-versioned — it resolved to setuptools==82.0.1, which removed the pkg_resources package. The builder stage installs setuptools into the venv, but the runtime stage image doesn’t have it, so pods fail at ModuleNotFoundError: No module named 'pkg_resources'.

Two-part fix (both in the current Dockerfile on main):

Pin setuptools>=78.1.1,<81 in requirements.txt (and in the runtime stage’s pip install in the Dockerfile).
Add an explicit pip install 'setuptools>=78.1.1,<81' to the Dockerfile’s runtime stage so setuptools is present in the final image even if the COPY --from=builder /opt/venv step is racy. Defensive — the venv is copied fully in practice, but the explicit install makes the build reproducible.

4. OIDC trust: `:environment:` claim in `sub`

See the note in §1. The pre-existing OIDC roles don’t declare an environment: on the workflow job, so their trust policies work without the ...:environment:<name> pattern. The new EKS deploy workflow declares environment: sandbox (or production), so its sub claim changes accordingly. Easy to miss when copy-pasting the trust policy from another role.

5. Deployment selector is immutable

The first push to main applied manifests with kustomize commonLabels: {part-of: hungryhub}, which baked app.kubernetes.io/part-of: hungryhub into the Deployment selector. Selector is immutable in k8s. When the next deploy tried to remove the part-of label (because we removed commonLabels), the apply failed with field is immutable.

Fix: drop the existing deployment before re-applying so the selector can be re-created. The deploy workflow now does kubectl delete deployment ai-text-to-image-web --ignore-not-found before each apply. Idempotent (no-op when the deployment doesn’t exist).

6. ExternalSecret CRDs not installed in cluster

The original design used External Secrets Operator (external-secrets.io/v1beta1 SecretStore + ExternalSecret) to mirror SM → k8s Secret automatically. The deploy failed with no matches for kind "ExternalSecret". ESO isn’t installed in the EKS clusters.

Fix: switched to the workflow-mirrors-SM approach (see §4). A follow-up could install ESO cluster-wide (one Terraform PR to eks-services/) and revert the workflow back to ExternalSecret for the cleaner operator-managed model.

How to operate

First-time setup (one-time per env)

hungryhub-iam CI roles — applied via hungryhub-iam#35.
EKS RBAC — applied via hungryhub-terraform#416.

AWS Secrets Manager secrets — created manually:

aws secretsmanager create-secret \
  --name /ai-text-to-image/sandbox \
  --secret-string '{"FAL_KEY":"...","AWS_REGION":"ap-southeast-1",...}' \
  --profile sandbox --region ap-southeast-1

ECR repos — created manually per account (3 per account = 6 total): ai-text-to-image (the consolidated single image; this is the only one used in production), plus ai-text-to-image-web and ai-text-to-image-image-server (per-component repos from the early pipeline design that are no longer pushed to). The early per-component repos are unused but kept for history.
GitHub repo vars on hungryhub-team/ai-text-to-image:
- AWS_REGION=ap-southeast-1
- EKS_CLUSTER_NAME_SANDBOX=eks-dev-262
- EKS_CLUSTER_NAME_PROD=eks-prod-21
- OIDC_ROLE_ARN_SANDBOX=arn:aws:iam::079994049689:role/github-actions-ai-text-to-image-sandbox
- OIDC_ROLE_ARN_PROD=arn:aws:iam::202255947274:role/github-actions-ai-text-to-image-prod
- ECR_REGISTRY_SANDBOX=079994049689.dkr.ecr.ap-southeast-1.amazonaws.com
- ECR_REGISTRY_PROD=202255947274.dkr.ecr.ap-southeast-1.amazonaws.com
GitHub environments: sandbox (no protection) + production (required reviewers, on paid GitHub plans). The production env on the free plan is only declarative — the OIDC trust subject still restricts who can assume the prod role, so manual workflow_dispatch is the de-facto gate.

Sandbox deploy (auto)

git push origin main

This triggers docker-scan.yml (build + push to ECR + Trivy) followed by deploy-eks.yml (auto-targets sandbox). Watch the run from the Actions tab; rollout is gated by a kubectl rollout status timeout of 10 min.

Sandbox re-deploy (manual)

gh workflow run deploy-eks.yml --repo hungryhub-team/ai-text-to-image \
  -f environment=sandbox

Or with a specific image tag:

gh workflow run deploy-eks.yml --repo hungryhub-team/ai-text-to-image \
  -f environment=sandbox \
  -f image_tag=<commit-sha>

Production deploy (manual)

# 1. Build + push the prod image
gh workflow run docker-scan.yml --repo hungryhub-team/ai-text-to-image
# (the workflow_dispatch input 'target' defaults to dev — set
# to 'prod' to push to the prod ECR account)
gh workflow run docker-scan.yml --repo hungryhub-team/ai-text-to-image \
  -f target=prod

# 2. Wait for the build to finish, get the SHA from the run log

# 3. Deploy
gh workflow run deploy-eks.yml --repo hungryhub-team/ai-text-to-image \
  -f environment=production \
  -f image_tag=<commit-sha>

The production environment (on orgs where required-reviewers is enabled) blocks the run until a reviewer approves it.

Rotating secrets

# Edit the SM secret. The next deploy re-reads SM and re-mirrors
# the value into the k8s Secret, so no separate secret-rotation
# step is needed.
aws secretsmanager put-secret-value \
  --secret-id /ai-text-to-image/sandbox \
  --secret-string "$(cat new-secrets.json)" \
  --profile sandbox --region ap-southeast-1

# To pick up the new value on an already-running pod, restart
# the Deployment — the deploy workflow mirrors SM only as part
# of an apply, not on a timer.
kubectl --context "arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262" \
  rollout restart deployment/ai-text-to-image-web -n utility

If you rotate frequently, the same Fal.ai key example in §“Gotcha: fal.ai “User is locked” applies — the “no redeploy needed” line was a simplification.

hh-menu is the primary caller. It POSTs to the /api/v1/hh-menu/generate-image endpoint on ai-text-to-image, and ai-text-to-image calls fal.ai.

Wiring

The endpoint URL is configured in hh-menu’s ConfigMap (hungryhub-menu-config in the hungryhub namespace) via the AI_IMAGE_SERVICE_URL Terraform var, which is sourced from the terraform/tfvars/{env}/infra AWS Secrets Manager secret (the ai_image_service_url key — see hungryhub-terraform#426 for the docs change describing this).

Sandbox value:

{
  "ai_image_service_url": "http://ai-text-to-image-web.utility.svc.cluster.local/api/v1/hh-menu/generate-image",
  "ai_image_model": "fal-ai/bytedance/seedream/v4/text-to-image",
  "ai_image_provider": "fal.ai"
}

Important — the URL is an in-cluster Kubernetes DNS name, not a public URL. hh-menu runs in the hungryhub namespace; the target service is in the utility namespace. Kubernetes ClusterDNS resolves ai-text-to-image-web.utility.svc.cluster.local to the ClusterIP service IP, which forwards to the pod. The request stays on the cluster’s private network — no public internet round-trip, no TLS, no cert-rotation surface.

Before this migration (pre-2026-06-12), the URL was https://menugram.hh-bee.my.id/api/v1/hh-menu/generate-image — a separate external service that no longer exists. Switching to the in-cluster URL was a single SM-secret update + one terraform apply of hungryhub-apps.

End-to-end verification (2026-06-12)

After pointing hh-menu at the new in-cluster endpoint, we verified the full chain end-to-end from inside a pod in the hungryhub namespace:

kubectl run ai-image-test --rm -it --restart=Never \
  --image=curlimages/curl:8.10.1 -n hungryhub -- \
  curl -sS -m 30 -X POST \
  http://ai-text-to-image-web.utility.svc.cluster.local/api/v1/hh-menu/generate-image \
  -H "Content-Type: application/json" \
  -d '{"menu_id":"123","menu_name":"Margherita Pizza","menu_price":12.99,"restaurant_id":"6364","provider":"fal.ai","model":"fal-ai/bytedance/seedream/v4/text-to-image"}'

Response:

{
  "status": "accepted",
  "image": "https://v3b.fal.media/files/b/0a9df243/...png",
  "error": null
}

A real PNG (~1 MB) was returned. The chain hh-menu ns → in-cluster DNS → ai-text-to-image-web pod → SM-mirrored FAL_KEY → fal.ai API → image all works.

Gotcha: fal.ai “User is locked” is a billing state, not a balance state

When the fal.ai API returns User is locked. Reason: Exhausted balance. it does not mean the account has insufficient balance. It means the fal.ai account is in a locked state, which is set by fal.ai when the account has no payment method on file (regardless of any credits). Fix:

Log into fal.ai dashboard with the email tied to the API key
Add a payment method (credit card) — not just credits
Add credits / top up
The locked state is cleared once the payment method is on file
The API key itself is unchanged; no rotation needed

If a future “User is locked” error appears, check the payment method before checking the balance. Adding more credits without a payment method will not unlock the account.

References

ai-text-to-image#18 — original EKS deploy workflow PR
hungryhub-iam#35 — CI roles
hungryhub-iam#36 — Version typo fix
hungryhub-terraform#416 — EKS RBAC
hungryhub-terraform#426 — hh-menu AI image URL: in-cluster (this PR)
hh-menu#320 — .env.example AI image URL: in-cluster (local dev consistency)
ai-text-to-image#19 through #32 — series of fixes during the first deploy
HHLion/deployment_setup.md — the pattern this pipeline was modeled on
Incident_hungryhub_namespace_deletion_2026-06-08.md — the incident that motivated moving secrets to SM
EKS_Cost_Optimization_2026.md — related EKS work; ai-text-to-image uses the SPOT-preference pattern (PR ai-text-to-image#17)

HungryHub Knowledge Base