hungryhub-terraform — Architecture and Operations

Central infrastructure-as-code for all AWS resources across dev and prod environments.

• Repo: hungryhub-team/hungryhub-terraform
• Default branch: main
• Stack: Terraform ≥ 1.3, AWS Provider ~5.x

AWS accounts

State bucket: hungryhub-{env}-terraform-{account_id} DynamoDB lock table: terraform-backend-{env}-lock Region: ap-southeast-1

Module map (selected)

Each top-level directory is an independent module with its own S3-backed state.

CI/CD — GitHub Actions

All plan and apply runs go through GitHub Actions on self-hosted runners (arc-runner-set-hungryhub-terraform). AWS authentication uses OIDC — no long-lived keys.

Workflows

Running a plan or apply

GitHub → Actions → "Terraform Plan/Apply" → Run workflow
  module:      e.g. hungryhub-apps
  environment: dev | prod
  action:      plan | apply

Plan output is posted as a PR comment. Apply requires a separate manual trigger.

Secrets and tfvars

No *.tfvars files are committed to the repo. At CI start, scripts/build_tfvars_from_sm.py fetches secrets from AWS Secrets Manager and writes env/{env}.tfvars.

Secret path structure

terraform/tfvars/{env}/{suffix}

Available suffixes:

Adding a new terraform variable

1. Declare it in the module’s variable.tf.
2. Add the key/value to the correct suffix in AWS Secrets Manager:

# Read current value, add key, write back
aws secretsmanager get-secret-value \
  --secret-id terraform/tfvars/prod/<suffix> \
  --profile prod --region ap-southeast-1 \
  --query SecretString --output text | \
python3 -c "
import json, sys
d = json.load(sys.stdin)
d['my_new_variable'] = 'value'
print(json.dumps(d))
" | aws secretsmanager put-secret-value \
  --secret-id terraform/tfvars/prod/<suffix> \
  --profile prod --region ap-southeast-1 \
  --secret-string file:///dev/stdin

# Repeat for dev:
# --secret-id terraform/tfvars/dev/<suffix> --profile sandbox

3. Reference it by name in configmap.tf or wherever needed.

OIDC subject pinning + guardrail test

GitHub Actions OIDC trust subjects must be pinned — never use the repo:<org>/<repo>:* wildcard. Pin to a specific event: ...:ref:refs/heads/main, ...:pull_request, or ...:environment:production. A typo or wildcard in a hand-edited trust policy silently grants any branch or fork the ability to assume the role.

scripts/test_oidc_subject_pinning.py is the guardrail test for this repo’s OIDC roles (the mirror of tests/test_iam_guardrails.py in hungry-hub-iam). It scans the trust blocks and fails on any unpinned :* subject. Run it before opening a PR that touches oidc-iam-rule/ or any aws_iam_role trust policy.

See OIDC Subject Pinning Patterns for the good-vs-bad examples and the 2026-05-20 typo incident that motivated the test.

Cross-repo IAM ownership

Not all IAM that this repo consumes is managed here. The Terraform CI role and the weekend-scheduler Lambda IAM are being migrated to hungry-hub-iam/stacks/oidc-permissions/ (issue #312); lambda/iam.tf reads them back as data sources. The migration is per-env — done in dev, skipped in prod:

• dev — these resources are owned by oidc-permissions/; they are no longer managed in oidc-iam-rule/.
• prod — the Terraform CI role + scheduler roles still live in oidc-iam-rule/ here; oidc-permissions/ deliberately skips them (create_*=false) so its prod apply stays a clean no-op.

So a new operator editing oidc-iam-rule/ should know: in dev the scheduler/lambda IAM has moved out, but in prod it is still owned here. See HungryHub IAM — Architecture and Operations → Known incomplete migration.

Local deployment (build.sh)

For emergency or one-off applies outside of CI:

# Ensure correct AWS profile is active first
aws sts get-caller-identity --profile prod

# Plan a single module
./build.sh plan prod hungryhub-apps

# Apply a single module (normal day)
./build.sh apply prod hungryhub-apps false false

# Apply with event-day scaling (normal_event=true)
./build.sh apply prod hungryhub-apps true false

# Apply with big event scaling
./build.sh apply prod hungryhub-apps false true

# Plan all core modules
./build.sh plan prod all

Build.sh auto-creates the S3 state bucket and DynamoDB lock table if they don’t exist.

hungryhub-server ConfigMap (hungryhub-apps)

The hungryhub-server-config ConfigMap (~280 keys) is the main mechanism for injecting env vars into all Rails pods. It is fanned out across every namespace (hungryhub, hh-end-user-public, hh-syn-public, hh-cosmos-public, hh-vendor-public, and dev-preview namespaces) via for_each.

Production namespaces and what they serve:

To live-patch a value without waiting for terraform (e.g. during an incident):

kubectl patch configmap hungryhub-server-config -n hh-end-user-public \
  --patch '{"data": {"MY_VAR": "value"}}' \
  --context arn:aws:eks:ap-southeast-1:202255947274:cluster/eks-prod-21

# Restart the deployment to pick up the change
kubectl rollout restart deployment/hungryhub-server -n hh-end-user-public

Note: kubectl patches are overwritten the next time terraform apply hungryhub-apps runs. Always follow up with an AWS Secrets Manager update + terraform PR to make the change permanent.

Event-day scaling

hungryhub-apps has two boolean toggles that change replica counts across all HPA and KEDA resources:

Coordinate with marketing before a campaign. Apply the toggle, then revert after the event:

# Before event
./build.sh apply prod hungryhub-apps true false   # normal_event
# or
./build.sh apply prod hungryhub-apps false true   # big_event

# After event — re-enables CronJobs
./build.sh apply prod hungryhub-apps false false

Cloudflare (hh-cloudflare)

DNS and tunnels are managed separately from the main AWS modules:

• Dev environment must run from the develop branch (enforced in CI).
• Prod runs from main.
• Secrets sourced from scripts/fetch-cf-secrets.py (reads from AWS Secrets Manager /hungryhub/prod and /hungryhub/dev).

Related

• AGENTS.md in repo: detailed module map, operating notes, review checklist
• HungryHub IAM — Architecture and Operations — the separate IAM/OIDC repo
• OIDC Subject Pinning Patterns
• AWS IAM Orphan Cleanup runbook
• Terraform Remote Lock State — Troubleshooting
• How to Disable Weekend Scale Scheduler
• ClickStack EKS Dev Runbook