How to update an env value (deploy key / credential) for hh-server

:no_entry_sign: Never edit a ConfigMap manually (kubectl edit configmap / kubectl patch) for values that Terraform owns. A manual edit causes drift: the live value no longer matches what Terraform has in state, so the next plan shows a - (remove) line for that key, and the following apply deletes the value you just set. This happens most often in dev/staging — drift piles up until the plan is hard to read and the review becomes ambiguous. Source of truth = AWS Secrets Manager → Terraform apply. Pods read the ConfigMap, but Terraform is what manages its values.

5 steps. If any one is skipped, the pod stays “healthy” while still holding the old value — so every pod has to be checked.

0. AWS account & context

Check first: aws sts get-caller-identity --profile prod + kubectl config current-context. Wrong account = you just changed prod.

1. Find the env in Secrets Manager

Every env lives in terraform/tfvars/{env}/. Pick to match the domain: infra, payments, notifications, analytics, rails, database, observability, social, ota-partners, hh-puma, hh-tiger, etc. (Not every suffix exists in both environments — hh-venus/hh-ballbot/hh-engineering/hh-syn/chrome-service are dev-only.)

# prod
aws secretsmanager get-secret-value \
  --secret-id terraform/tfvars/prod/<suffix> \
  --profile prod --region ap-southeast-1 \
  --query SecretString --output text | python3 -m json.tool

# dev
aws secretsmanager get-secret-value \
  --secret-id terraform/tfvars/dev/<suffix> \
  --profile sandbox --region ap-southeast-1 \
  --query SecretString --output text | python3 -m json.tool

2. Update the value in Secrets Manager

Each secret is one JSON object holding many envs. Update only the one key — <ENV_KEY> is the name of the env you are changing.

# dev
aws secretsmanager get-secret-value \
  --secret-id terraform/tfvars/dev/<suffix> \
  --profile sandbox --region ap-southeast-1 \
  --query SecretString --output text | \
python3 -c "import json,sys; d=json.load(sys.stdin); d['<ENV_KEY>']='NEW_VALUE'; print(json.dumps(d))" | \
aws secretsmanager put-secret-value \
  --secret-id terraform/tfvars/dev/<suffix> \
  --profile sandbox --region ap-southeast-1 \
  --secret-string file:///dev/stdin

prod: swap prod/ and –profile prod. If the env exists under more than one suffix, update it in every one of them.

3. Plan → check the drift → apply

CI: hungryhub-terraform → Actions → “Terraform Deployment Workflow” → environment dev/prod, services hungryhub-apps, action plan (read the diff first) → apply.

Check in the diff:

Only the env you updated should change. Anything else changing is drift and has to be resolved first (see the drift note below). There should be no destroy. A - line pointing at an env that is actually in use is drift left behind by a manual edit. Fix: add that value to SM and to hungryhub-apps/configmap.tf, then apply again.

4. Restart the pods

envFrom reads the ConfigMap once, at container start — updating the ConfigMap does not change the environment of a running pod, so a rollout is mandatory:

kubectl --context <eks-dev-262-arn> -n <ns> rollout restart deployment/<name>

Restart only the deployments that consume that ConfigMap

5. Verify

kubectl --context <ctx> -n <ns> exec <pod> -c <container> -- printenv <ENV_KEY> | cut -c1-14

Every pod that consumes the ConfigMap must show the new value. Still the old value = still running on the old value. (The container name is not always the deployment name — get it with kubectl get deploy -o jsonpath=‘{.spec.template.spec.containers[*].name}’.)