Incident: hungryhub + utility Namespace Deletion — 2026-06-08

Summary

Triggering terraform apply for eks-services with new code on branch fix/409-ca-priority-expander deleted the hungryhub and utility Kubernetes namespaces, taking down all workloads in those namespaces for ~30–45 minutes.

Timeline (UTC+7 / Bangkok)

Root Cause

namespace.tf in eks-services declares kubernetes_namespace resources for all prod namespaces. These namespaces had been created manually (pre-Terraform) and were never imported into Terraform state. Terraform therefore treated them as resources to create.

On the first apply after this discrepancy was exposed, Terraform attempted to create all namespaces. For namespaces already in state (dev-preview-*), it refreshed correctly. For the 10 that were NOT in state, it tried to create them. hungryhub and utility completed the delete step before the conflict errors terminated the apply — the others got “already exists” because their create was attempted while they still existed.

Why only hungryhub and utility were deleted: Terraform processes resources in parallel. Those two happened to complete delete before the conflict errors terminated the apply run.

Services Affected

Recovery Steps Taken

1. Recreated namespaces:

   kubectl create namespace hungryhub
   kubectl create namespace utility
   

2. Copied hungryhub-server-config ConfigMap from hh-cosmos-private (pods need it to start):

   kubectl get configmap hungryhub-server-config -n hh-cosmos-private -o json | \
     python3 -c "
   import json,sys
   cm = json.load(sys.stdin)
   cm['metadata'] = {'name': 'hungryhub-server-config', 'namespace': 'hungryhub'}
   print(json.dumps(cm))
   " | kubectl apply -f -
   

3. Pods rescheduled automatically once namespace + ConfigMap existed.

4. CodePipeline prod-hh-server-main (already running) completed the hungryhub deploy step, restoring remaining workloads.

What Was NOT Recovered Automatically

ai-text-to-image Secret — contained AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, GEMINI_API_KEY, FAL_KEY. Only existed as a Kubernetes Secret, not backed by SSM or Secrets Manager. Must be recreated manually before ai-text-to-image can start.

Fix Applied

Added namespace-imports.tf to eks-services with native Terraform import blocks (TF 1.5+) for all 10 untracked namespaces. On next successful terraform apply, these namespaces will be imported into state and Terraform will manage them without deleting.

PR: https://github.com/hungryhub-team/hungryhub-terraform/pull/411

Prevention / Follow-up Actions

Lessons Learned

1. Never apply terraform on a module with unimported live resources. Run terraform plan first; gate on zero unexpected destroys.
2. Terraform delete+create is not atomic. A failed apply can leave resources partially destroyed.
3. k8s Secrets must be backed by an external secret store. A Secret living only in etcd is one namespace deletion from permanent loss.
4. Add prevent_destroy to namespace resources to require explicit override before deletion.

Related

• EKS Cost Optimization 2026 — context for the CA expander change (#409) that triggered the apply
• hungryhub-terraform PR#411