Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

OIDC Subject Typo Incident (2026-05-20)

A one-character typo in a GitHub Actions OIDC trust subject created a role that silently never matched, spawned five orphan AWS resources, and stayed invisible until someone tried to reconcile it weeks later. This is the case study; the rule it produced lives in OIDC Subject Pinning Patterns.

Companion to Debugging a Real Issue with an AI Agent and AI Agent Lessons from Infrastructure Work.

What happened

On 2026-05-20, a hand-built OIDC role’s trust policy was given the subject:

repo:hungryhub-team/hungry-hub-iam:*

Two things are wrong with that one line:

  1. The repo name is a typo. The GitHub repository is named hungryhub-iam (no hyphen between hungry and hub). The local checkout directory is hungry-hub-iam, and the hyphenated form leaked into the trust subject. GitHub never mints a token with sub = repo:hungryhub-team/hungry-hub-iam:..., so the condition could never match.
  2. The suffix is the :* wildcard. Even with the right repo name, :* accepts any branch, tag, or fork PR — exactly the footgun the pinning rule forbids.

Because the role was created by hand in the AWS console (not in a reviewed PR), neither error was caught at write time. The subject was a free-text JSON blob, not a PR-reviewable Terraform attribute.

How it surfaced

The typo stayed invisible until the relay-ci reconcile work began — the effort to bring the hand-built relay-eval-ci role under Terraform management (see Relay CI Stack Pattern). Importing the live role exposed the malformed trust subject, and a sweep of the surrounding OIDC roles turned up the orphans the bad config had left behind.

The five orphan resources

The bad config and its siblings left 5 physical AWS resources in main prod (202255947274) that no Terraform state owned:

  • prod-oidc-hh-relay-eval-role + inline hh-relay-eval-ci-policy
  • prod-oidc-hh-relay-agentcore-deploy-role + inline hh-relay-agentcore-deploy-policy
  • relay-eval-ci (a main-prod duplicate) + inline relay-eval-ci-policy

State was removed first via terraform state rm (hungryhub-terraform PR #378, the cleanup-orphan-oidc-relay-state.yml workflow). The physical destroy of all five happened on 2026-06-05 as a follow-up (aws iam delete-role + aws iam delete-role-policy per role). See the AWS IAM Orphan Cleanup runbook for the safe procedure.

The fix

Two changes corrected the trust config and made the class of bug impossible to reintroduce silently:

  1. Pinned, correctly-named subject. The reconciled relay-ci stack (hungry-hub-iam PR #22, with the remote-state parameterization in PR #24) pins the subject to repo:hungryhub-team/hh-relay:ref:refs/heads/main and the terraform-side IAM group fix landed in hungryhub-terraform PR #373.
  2. A guardrail test. hungry-hub-iam/tests/test_iam_guardrails.py and its mirror hungryhub-terraform/scripts/test_oidc_subject_pinning.py (PR #383) fail loudly on any :* wildcard and on any malformed org/repo prefix — the exact hungry-hub-iam typo class.

The lesson

Subject patterns must be PR-reviewable attributes, not hand-typed JSON blobs in the AWS console. Both errors here were invisible because the role was hand-built: a reviewer never saw the diff, and no test ran. Moving the trust config into Terraform made the subject a reviewable line; adding the guardrail test made the bad shapes fail CI. A typo that survived weeks in the console would now fail a PR check in seconds.