Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

AWS IAM Orphan Cleanup

Purpose

Use this runbook to find and safely delete orphan IAM resources — roles, policies, OIDC providers, and access keys that no Terraform state owns and no workload uses. Orphans accumulate from hand-built roles, abandoned migrations, and incomplete terraform destroy runs. They are a security and audit liability.

This runbook follows the same shape as AWS SES Pause: Manual Notification Recovery: preconditions → safety → dry-run → approve → execute → verify → close-out.

Scope

  • Accounts: any HungryHub account — sandbox 079994049689, prod 202255947274, genai-sandbox 965444437277, genai-prod 512438352490 (see AWS Account and Profile Map)
  • Resource types: IAM roles, managed + inline policies, OIDC providers, access keys
  • Out of scope: anything still referenced in a live Terraform state — remove it via terraform state rm / the owning module first, never delete from under live state

When to use this runbook

Use this runbook if any of these are true:

  1. A terraform state rm left physical AWS resources behind (state cleaned, AWS side not).
  2. A hand-built role is being retired or has been superseded by a Terraform-managed one.
  3. A periodic audit found roles/policies/keys with no attachment or no recent use.

Preconditions

  1. You have an SSO session for the target account (aws sso login --profile <name>).
  2. You have verified the active account with aws sts get-caller-identity --profile <name> — orphan cleanup is a destructive, account-specific operation.
  3. You have confirmed the resource is not in any live Terraform state (terraform state list | grep <name> in every stack that could own it).
  4. You have human approval from an IAM owner (@saiqulhaq-hh or @irfanwicaksana-hh) before any delete.

Safety notes

  • Always run the dry-run / discovery queries first and review the full list before deleting anything.
  • IAM deletes are irreversible. A wrongly-deleted role breaks every workflow that assumes it.
  • Delete in dependency order (detach/delete policies before the role) or the delete call fails.
  • Never delete the shared OIDC provider token.actions.githubusercontent.com — many roles trust it.
  • Record the account, operator, and timestamp of every delete for the audit trail.

Procedure

1) Confirm the active account

aws sts get-caller-identity --profile <profile>
# Confirm Account matches the intended target before continuing.

2) Find orphan IAM roles (no attachments + no recent AssumeRole)

# List roles, then for a candidate check for attached/inline policies:
aws iam list-attached-role-policies --role-name <role> --profile <profile>
aws iam list-role-policies          --role-name <role> --profile <profile>

# Check whether the role has been assumed recently (CloudTrail, last N days).
# Run once per assume-role event name — a role can be assumed via plain
# AssumeRole (STS) and/or AssumeRoleWithWebIdentity (OIDC), so check both:
for EVENT in AssumeRole AssumeRoleWithWebIdentity; do
  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=$EVENT \
    --profile <profile> --region <region> \
    --query "Events[?contains(CloudTrailEvent, '<role>')]"
done

A role with no policy attachments and no AssumeRole / AssumeRoleWithWebIdentity traffic in the lookback window is a strong orphan candidate.

3) Find unattached managed policies and stale OIDC providers

# Customer-managed policies with zero attachments:
aws iam list-policies --scope Local --only-attached false --profile <profile>

# OIDC providers (confirm the GitHub one is still needed before touching any):
aws iam list-open-id-connect-providers --profile <profile>

4) Find access keys unused 90+ days

aws iam list-access-keys --user-name <user> --profile <profile>
aws iam get-access-key-last-used --access-key-id <key-id> --profile <profile>
# If LastUsedDate is older than 90 days (or null), flag for rotation/removal.

5) Approval checkpoint (before any delete)

  1. The full candidate list is reviewed.
  2. Each candidate is confirmed absent from every live Terraform state.
  3. An IAM owner has approved the specific list.

6) Execute the safe delete order

Inline policies first, then attachments, then the role:

# 1. Delete inline policies on the role
aws iam delete-role-policy --role-name <role> --policy-name <inline-policy> --profile <profile>

# 2. Detach any managed policies
aws iam detach-role-policy --role-name <role> --policy-arn <arn> --profile <profile>

# 3. Delete the role itself
aws iam delete-role --role-name <role> --profile <profile>

# For a managed policy: delete every non-default version first, then the
# policy itself (delete-policy fails if any non-default version remains).
aws iam list-policy-versions --policy-arn <arn> --profile <profile> \
  --query "Versions[?IsDefaultVersion==\`false\`].VersionId" --output text
# For each VersionId returned above:
aws iam delete-policy-version --policy-arn <arn> --version-id <version-id> --profile <profile>
# Then delete the policy (this removes the default version too):
aws iam delete-policy --policy-arn <arn> --profile <profile>

7) Verify results

aws iam get-role --role-name <role> --profile <profile>   # expect NoSuchEntity

Collect for the record: account ID, each deleted resource name, the operator, and the timestamp.

8) Close-out checklist

  • Active account verified before each delete.
  • Every deleted resource confirmed absent from live Terraform state first.
  • Delete list approved by an IAM owner.
  • Deletes ran in dependency order (inline → attached → role).
  • Post-delete get-role returns NoSuchEntity for each.
  • Shared OIDC provider untouched.
  • Cleanup summary (account, resources, operator, timestamp) recorded.

Worked example — 2026-06-05 relay OIDC orphan cleanup

The OIDC Subject Typo Incident left 5 orphan resources in main prod (202255947274):

  • prod-oidc-hh-relay-eval-role + inline hh-relay-eval-ci-policy
  • prod-oidc-hh-relay-agentcore-deploy-role + inline hh-relay-agentcore-deploy-policy
  • relay-eval-ci (main-prod duplicate) + inline relay-eval-ci-policy

The state was removed earlier via terraform state rm (hungryhub-terraform PR #378, cleanup-orphan-oidc-relay-state.yml). On 2026-06-05 the 5 physical resources were destroyed using exactly the step-6 order above (aws iam delete-role-policy then aws iam delete-role per role), outside any PR. This is the canonical example of the state-side-then-AWS-side two-step.

Two distinct relay cleanups — don’t conflate them. cleanup-orphan-oidc-relay-state.yml does terraform state rm on the orphan OIDC roles (the ones above). A separate workflow, destroy-orphaned-relay-iam.yml, destroys a leftover relay-bot-* IAM user (residual from removed access-key resources) — a different resource, with its own confirm_destroy input. The repo-side runbook documents both: hungryhub-terraform/docs/runbooks/orphan-oidc-relay-state-cleanup.md.