Eagle Eye + Hunger Games — Migrate Staging from DOKS/DOCR to AWS EKS/ECR

Status: planning — not yet started. Epic issues: eagle-eye#174, hunger-games#32.

Why

Both apps currently ship staging to DigitalOcean Kubernetes (DOKS) with images in DOCR. Two motivations for the move:

1. Cost & cluster consolidation — most other HungryHub services already run on eks-dev-262 in the sandbox account. Migrating eagle-eye and hunger-games staging means a single staging plane to operate.
2. Path to production — production already runs on eks-prod-21. Having dev/staging on the same EKS flavour means a single mental model, one set of RBAC + IAM plumbing, one set of OIDC trust relationships.

Production (eks-prod-21, GHCR) is out of scope for this document. We are not touching the prod pipeline for either app.

Current state (verified 2026-06-11)

Confirmed by repo maintainer (2026-06-11): the intended trigger model for hunger-games is push to main → staging, with workflow_dispatch for prod. The auto-prod bug is unintended.

eks-dev-262 namespaces inspected — no eagle-eye or hunger-games namespace yet. sandbox ECR — no eagle-eye-* or hunger-games-* repos yet. GitHub OIDC provider token.actions.githubusercontent.com is already in the sandbox account (owned by hungryhub-terraform/oidc-iam-rule/). We reuse it; we do not create a second one.

Target state

Old DOKS staging clusters and DOCR images are decommissioned after a 1–2 week observation window. The DOCR namespaces are drained, not deleted on day 1.

ECR repository layout

One repo per image, prefixed by service. Retention policy applied to each.

ECR lifecycle policy (applies to all of the above)

“Keep last 2 months” — codified as the following rules, evaluated in priority order. Untagged images (failed builds, intermediate layers) are cleared after 7 days to bound storage. Tagged images are kept up to 60 days OR the most recent 50, whichever expires first. Always-preserved tags (*-latest, semver-like vX.Y.Z) are exempt.

[
  {
    "rulePriority": 1,
    "description": "Expire untagged images after 7 days",
    "selection": {
      "tagStatus": "untagged",
      "countType": "sinceImagePushed",
      "countUnit": "days",
      "countNumber": 7
    },
    "action": { "type": "expire" }
  },
  {
    "rulePriority": 2,
    "description": "Keep at least 50 tagged images (covers bursty build cadence)",
    "selection": {
      "tagStatus": "tagged",
      "countType": "imageCountMoreThan",
      "countNumber": 50
    },
    "action": { "type": "expire" }
  },
  {
    "rulePriority": 3,
    "description": "Expire tagged images after 60 days (~2 months)",
    "selection": {
      "tagStatus": "tagged",
      "tagPrefixList": ["stg-", "prod-"],
      "countType": "sinceImagePushed",
      "countUnit": "days",
      "countNumber": 60
    },
    "action": { "type": "expire" }
  },
  {
    "rulePriority": 4,
    "description": "Always preserve floating tags and semver tags",
    "selection": {
      "tagStatus": "tagged",
      "tagPrefixList": ["staging-latest", "production-latest", "v"],
      "countType": "sinceImagePushed",
      "countUnit": "days",
      "countNumber": 365
    },
    "action": { "type": "expire" }
  }
]

Rules are evaluated in order — rule 4 keeps a long tail, so the 60-day cap in rule 3 only applies to non-protected prefixes.

IAM & OIDC (per repo)

Per hungryhub-iam/AGENTS.md rules, OIDC subject patterns are pinned and prod is isolated:

ARN scoping follows hungryhub-iam/AGENTS.md §3.4 — no Resource = "*".

The OIDC provider (arn:aws:iam::079994049689:oidc-provider/token.actions.githubusercontent.com) is not recreated. We add the new roles in a new stack stacks/eagle-eye-hunger-games-staging/ under hungryhub-iam, using the existing modules/github_oidc_role/ module.

Bootstrap order

1. Infra (manual / IaC):

   1. Add stacks/eagle-eye-hunger-games-staging/ to hungryhub-iam with the two OIDC roles and their scoped inline policies. Apply against sandbox.
   2. Create the 8 ECR repos in sandbox (ap-southeast-1) with the lifecycle policy above. Either via a new Terraform stack in hungryhub-terraform (ecr/eagle-eye-hunger-games-staging/) or via aws ecr create-repository for now and codified in a follow-up. Decide before the PR.
   3. Create the eagle-eye namespace on eks-dev-262 (no separate hunger-games namespace — the two apps share by design, see Eagle Eye + Hunger Games namespace note below). No special RBAC needed; namespaces are cheap.
   4. Add the GitHub Actions role ARNs to the cluster’s aws-auth ConfigMap so the CI role can call eks:DescribeCluster and write to the namespace. This is a one-time kubectl edit cm aws-auth -n kube-system plus a restart of the node groups (or use eksctl create iamidentitymapping).

2. Workflow changes (eagle-eye):

   1. Add new GitHub Environment eks-staging and mirror all existing STAGING_* secrets into it, OR rename staging → eks-staging and add an staging alias. Decision: rename to eks-staging to make the cluster name explicit, and remove the old staging env once the rename ships.
   2. Update .github/workflows/deploy.yml:
      • validate job: drop the KUBECONFIG_DO_STAGING requirement, add AWS_ROLE_TO_ASSUME_STAGING and AWS_EKS_CLUSTER_NAME_STAGING requirements.
      • build-images matrix: change the registry prefix to sandbox ECR registry for staging runs.
      • deploy job: when environment == eks-staging, assume the OIDC role and aws eks update-kubeconfig (mirroring the existing prod path).
   3. Update deploy/k8s/staging/ manifests if they have any DOKS-specific annotations or service types.

3. Workflow changes (hunger-games):

   1. Same shape of changes as eagle-eye.
   2. Prerequisite: remove auto-deploy to prod on push to main (issue #34). Without this, the new staging deploy can race with prod.

4. Cutover (both apps):

   1. Land infra + workflow PRs in dry-run mode first.
   2. Manually trigger a staging deploy on each app via workflow_dispatch. Watch kubectl get pods -n <ns> until all 5–9 deployments are healthy.
   3. Sanity-check the eagle-eye frontend (<env>-latest tag) renders, API responds, scheduler runs, BullMQ workers consume.
   4. Repeat for hunger-games.
   5. Leave DOKS staging running for 1–2 weeks. If no rollback needed, drain workloads and delete the DOKS clusters via DigitalOcean control panel.

5. Cleanup (post-cutover):

   1. Delete the DOKS clusters.
   2. Remove the legacy cleanup-registries.yml logic for DOCR namespaces of these two apps.
   3. Update Domain_and_Namespace_List.md to list the eagle-eye namespace on eks-dev-262 (it’s a shared namespace hosting both apps, not a new namespace).
   4. Update Eagle_Eye_Architecture_and_Operations.md and Hunger_Games_Architecture_and_Operations.md to reflect the new cluster/registry.

Why both apps share the eagle-eye namespace

Verified 2026-06-11 against prod EKS (eks-prod-21) and DOKS staging: both namespaces host both apps’ deployments. Eagle Eye’s BFF calls the hunger-games API over same-namespace DNS using HUNGERGAMES_API_BASE_URL (visible in the eagle-eye-env secret on prod). Splitting them would require a cross-namespace NetworkPolicy and DNS rewrite and is out of scope for the staging migration.

The hard-coded NAMESPACE=eagle-eye in hunger-games/.github/workflows/deploy-kubernetes.yml:260 is therefore correct, not a copy-paste bug. (An earlier version of this plan and epic #33 incorrectly claimed otherwise; both have been corrected.)

Rollback

• Before cutover: revert the workflow PR. No infra destroyed; nothing to undo.
• After cutover, within 2 weeks: re-point the workflow at DOKS (the old kubeconfig + DOCR credentials are still valid). DOKS clusters kept running for this reason.
• After DOKS decommission: restore from ECR. Tagged images are kept for 60 days (rule 3); if rollback is needed past that, the manifests in deploy/k8s/<env>/ are the source of truth for reconstruction.

Cross-cutting design decisions to revisit after the migration

These are the open issues this plan is not solving — file separately:

1. Image-baked CA bundle (eagle-eye prod). Stale CA risk. Plan: move to AWS Secrets Manager + job-time fetch.
2. Hunger-games maturity — auto-prod on main + wrong namespace. Tracked separately.
3. Drift detection on K8s secrets vs GitHub Environment secrets. Tracked separately.

Open questions for the team

• Are the eagle-eye and hunger-games repos OK being added as new stacks in hungryhub-iam, or should they live in their own repo? Default: new stack in hungryhub-iam to follow the existing pattern.
• Do we want a single shared ECR registry prefix (eagle-eye-* and hunger-games-* repos under the same registry), or separate registries? Default: single registry, prefixed repo names. Cheaper and simpler.
• Should the eagle-eye and hunger-games IAM stacks be one stack or two? Default: one stack, since they share the OIDC provider and apply together.