hungryhub-iam — Architecture and Operations

Terraform-managed IAM roles, Secrets Manager access policies, and GitHub Actions OIDC trust relationships for all HungryHub services.

• Repo: hungryhub-team/hungryhub-iam (was hungry-hub-iam)
• Default branch: main
• Stack: Terraform (HCL)

Why a dedicated repo (not hungryhub-terraform)

IAM/OIDC config used to live in hungryhub-terraform. It was moved to a dedicated repo to fix a privilege escalation risk.

The problem: the CI role that runs hungryhub-terraform workflows needs broad AWS permissions (EKS, RDS, S3, etc.). If that same repo also contains the code defining OIDC trust relationships and IAM roles, a compromised PR or workflow could silently modify its own permissions — granting itself broader access or backdooring the trust policy. The role controls the code that defines the role.

The fix: hungryhub-iam is a separate repo with a separate, tighter CI role. hungryhub-terraform’s CI role has no iam:CreateRole or iam:PutRolePolicy permissions. IAM changes require a deliberate PR in a separate repo, reviewed by a smaller group.

This matches the AWS Well-Architected recommendation to separate the identity plane from the infrastructure control plane.

Staying disciplined

Three things that must hold for the separation to remain effective:

1. No IAM resources in hungryhub-terraform. Any aws_iam_role, aws_iam_policy, or OIDC resource appearing in a hungryhub-terraform PR is a red flag — it should live here instead. Audit periodically or add a CI check.

2. Human gate on prod applies. The production GitHub environment is configured on the prod CI role, requiring manual approval before any workflow can assume it. Keep this gate; do not remove it for convenience.

3. Apply this repo before hungryhub-terraform when a new CI role is needed. Changes across both repos are sequenced: merge and apply hungryhub-iam first, then open the hungryhub-terraform PR that references the new role ARN.

What it manages

Repository layout

stacks/
  github-oidc/              OIDC provider (lookup or create)
  oidc-permissions/         Terraform CI role + weekend-scheduler IAM (issue #312)
  secrets-manager-access/   CI roles + scoped Secrets Manager policies
  engineer-access/          Squad groups + user bindings
  relay-ci/                 relay-eval-ci role — in the GENAI accounts
  secrets-manager-layout/   Empty per-domain secret skeleton
modules/
  github_oidc_role/         Reusable: OIDC-trusted IAM role
  secrets_access_policy/    Reusable: Secrets Manager read policy builder
  engineer_group/           Reusable: IAM group + attached policies
scripts/
  json_to_tfvars.py         Converts Secrets Manager JSON → .tfvars
examples/
  github-actions/           Reusable GitHub Actions workflow templates

Bootstrap order

Apply stacks in this order (home accounts — sandbox / prod):

1. stacks/github-oidc/
2. stacks/oidc-permissions/      (per-env: dev owns it, prod skipped — see below)
3. stacks/secrets-manager-layout/
4. stacks/secrets-manager-access/
5. stacks/engineer-access/

The relay-ci stack is applied separately, in the genai accounts — see Cross-account OIDC.

Each stack uses an empty backend "s3" {} and requires explicit backend config at init:

cd stacks/github-oidc
terraform init \
  -backend-config="bucket=<state-bucket>" \
  -backend-config="key=hungry-hub-iam/github-oidc.tfstate" \
  -backend-config="region=ap-southeast-1" \
  -backend-config="dynamodb_table=<lock-table>"
terraform apply

Cross-account OIDC

This repo manages OIDC trust across two account pairs:

The home-vs-genai split: hungry-hub-iam and hungryhub-terraform own the home accounts; hungryhub-ai and hh-relay own the genai accounts. The relay-ci stack is the exception — it lives in this repo but manages the relay-eval-ci OIDC role in the genai accounts (the home-account OIDC role has no cross-account IAM write perms into genai). All stacks dispatch through one workflow (.github/workflows/terraform-stacks.yml), which selects the OIDC role and the S3/DynamoDB backend per stack — home stacks read the default OIDC_ROLE_ARN_{PROD,DEV} secrets, relay-ci reads the _RELAY_CI_* overrides.

See:

• OIDC Subject Pinning Patterns — good-vs-bad trust subjects and the guardrail test
• Relay CI Stack Pattern — what relay-eval-ci grants and how its secrets are wired

Known incomplete migration — oidc-permissions ↔ oidc-iam-rule

The oidc-permissions stack was meant to take ownership of the Terraform CI role and the weekend-scheduler Lambda IAM that previously lived in hungryhub-terraform/oidc-iam-rule/ and lambda/iam.tf (issue #312). The migration is per-env — done in dev, deliberately skipped in prod:

• dev — oidc-permissions owns the role + scheduler roles (create_github_oidc_role=true, create_scheduler_roles=true).
• prod — the role still lives in oidc-iam-rule/ (a fresh recreate would hit EntityAlreadyExists, and the policy would exceed the 6144-char limit), so the workflow passes create_github_oidc_role=false + create_scheduler_roles=false. The prod apply then makes a clean 0 added/changed/destroyed plan — it is not blocked, it just doesn’t take prod ownership yet (until #312 is finished).

hungryhub-terraform/lambda/iam.tf reads the migrated resources as data sources. Before applying oidc-permissions, read stacks/oidc-permissions/README.md “Cross-repo ownership (prod)”.

CI roles and trust

The secrets-manager-access stack defines three GitHub Actions OIDC roles:

The production GitHub environment gate means no workflow can assume the prod role without a human approving the deployment.

Engineer access model

Prod secret access requires role assumption through a separate escalation path — it is not granted to individual engineers by default.

Common operations

Add a new secret key

# 1. Fetch the current secret
aws secretsmanager get-secret-value \
  --secret-id terraform/tfvars/dev/integrations \
  --query SecretString --output text > /tmp/secret.json

# 2. Edit /tmp/secret.json — add the new key (flat string/number/bool only)

# 3. Push the updated secret
aws secretsmanager put-secret-value \
  --secret-id terraform/tfvars/dev/integrations \
  --secret-string file:///tmp/secret.json

# 4. Shred the local file
shred -u /tmp/secret.json

# 5. Open a PR in the consuming repo adding variable "<new_key>" and wiring it

Rotate a secret (no Terraform change needed)

aws secretsmanager put-secret-value \
  --secret-id terraform/tfvars/prod/payments \
  --secret-string file:///tmp/new.json

Re-run the relevant terraform apply or app reload to pick up the new value.

Onboard an engineer

1. Create the IAM user (or SSO identity) out-of-band:

   aws iam create-user --user-name engineer.firstname.lastname
   

2. Open a PR adding to stacks/engineer-access/users.tf:

   resource "aws_iam_user" "engineer_firstname_lastname" {
     name = "engineer.firstname.lastname"
   }
   
   resource "aws_iam_user_group_membership" "engineer_firstname_lastname" {
     user   = aws_iam_user.engineer_firstname_lastname.name
     groups = [module.squad_integrations_group.group_name]
   }
   

3. Apply from stacks/engineer-access/ after merge.

Offboard an engineer

1. Open a PR removing the user’s blocks from stacks/engineer-access/users.tf.
2. Apply after merge.
3. Rotate every dev secret the user had write access to.
4. Verify no remaining access keys or login profile remain:

   aws iam list-access-keys --user-name engineer.firstname.lastname
   aws iam get-login-profile --user-name engineer.firstname.lastname
   

Related

• Full runbook (adding secrets, rotating, on/offboarding): ROTATION.md in the repo
• Testing guide: TESTING.md in the repo
• GitHub Actions reusable workflow examples: examples/github-actions/
• OIDC Subject Pinning Patterns
• Relay CI Stack Pattern
• AWS IAM Orphan Cleanup runbook — find and safely delete orphan roles/policies
• hungryhub-terraform — Architecture and Operations