EKS Cost Optimization 2026

Background

After migrating from Auto Scaling Groups (ASG) to EKS Managed Node Groups (MNG) in early 2026, monthly EKS costs increased ~116%. Root cause: all workloads had a node.kubernetes.io/lifecycle: on-demand nodeSelector baked in, so pods refused to schedule on SPOT instances. MNG provides both OD and SPOT node groups, but only OD was being used.

Fix Chain

Step 1 — Remove hard on-demand nodeSelector (hh-server#8186)

Replaced nodeSelector: { node.kubernetes.io/lifecycle: on-demand } with a soft preferredDuringSchedulingIgnoredDuringExecution affinity:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 60
        preference:
          matchExpressions:
            - key: node.kubernetes.io/lifecycle
              operator: In
              values:
                - spot
      - weight: 40
        preference:
          matchExpressions:
            - key: node.kubernetes.io/lifecycle
              operator: In
              values:
                - normal

Weight 60/40 soft-prefers SPOT but falls back to OD if no SPOT available.

Step 2 — Force rollout restart (hh-infra#399)

Existing pods were already scheduled on OD nodes and were not retroactively rescheduled after the affinity change. A kubectl rollout restart was needed to cycle pods onto SPOT nodes.

Step 3 — CPU request rightsizing (hh-server#8187, hh-infra worktree hh-server-405-rightsize-cpu)

hungryhub-server had 850m CPU request vs ~200m average actual usage. Lowered to 300m (limit unchanged at 2000m). See Kubernetes CPU Request Rightsizing for the full procedure.

Why CA Didn’t Scale Down After Step 1

Cluster Autoscaler only considers newly scheduled pods when making bin-packing decisions. Pods that are already running on OD nodes stay there until they restart naturally or are manually restarted. CA will drain a node only when:

1. The node has been “unneeded” for scale-down-unneeded-time (default 10 min).
2. All non-DaemonSet pods on the node can be rescheduled elsewhere.

DaemonSet pods are non-evictable — a node that only has DaemonSet pods remaining after other pods are evicted shows “No candidates” in CA logs. CA adds a new node to schedule the DaemonSet pods, then removes the drained node.

Why SPOT Is Still Underused

CA uses expander=random (current config). When scaling out, it picks a node group randomly — OD and SPOT have equal probability. To prefer SPOT, the expander should be priority with a config map that ranks SPOT MNGs above OD MNGs. Tracked in hh-infra#409.

Vantage Savings Identified (June 2026 Report)

Additional top savings: hunger-games scrapers ($313), airbyte ($53), airflow ($57).

SPOT Affinity Pattern (2026-06-07)

Background workers should prefer SPOT. Web/critical queues should require ON_DEMAND. The canonical affinity blocks used throughout this project:

Prefer SPOT (background workers, scrapers, karafka, ai workloads):

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80
      preference:
        matchExpressions:
        - key: eks.amazonaws.com/capacityType
          operator: In
          values: ["SPOT"]

Require ON_DEMAND (sidekiq-critical, web tier):

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/capacityType
          operator: In
          values: ["ON_DEMAND"]

Note: eks.amazonaws.com/capacityType label values are SPOT and ON_DEMAND (not spot/normal). The old node.kubernetes.io/lifecycle label used spot/normal — wrong for MNG clusters.

After adding affinity, a kubectl rollout restart is required to move existing pods. CA drains vacated nodes after scale-down-unneeded-time (10 min default).

Rightsizing Wave 2 (2026-06-07/08)

Following the initial hungryhub-server fix (#405), a second wave rightsized background workloads:

airflow memory was increased — workers were consuming 1486–1549Mi vs 1024Mi request (OOM eviction risk). Always check actual memory usage before reducing.

Also: ai-text-to-image k8s manifest added to repo (k8s/deployment.yaml) — previously only existed as a live kubectl apply with no git source-of-truth. Airflow helm values saved to hungryhub-terraform/airflow/helm/values.yaml for the same reason.

OD MNG min_size Fix (hh-infra#401, 2026-06-07)

Terraform code had min_size=3 but AWS live showed min=12. Root cause: terraform apply hadn’t run since the code change. Fixed by triggering Terraform Deployment Workflow via GitHub Actions (environment=prod, action=apply, services=eks). AWS live now matches Terraform.

Fractional Gi Memory Bug (hh-server#8189, 2026-06-07)

3.2Gi and 2.5Gi in Kubernetes YAML resolve to fractional byte values (e.g. 3435973836800m) — rejected by Kubernetes as invalid. Kubernetes only accepts integer byte values.

Affected: sidekiq-kafka, sidekiq-inv, sidekiq-critical across prod/legacy, prod/private, staging/legacy (9 files).

Fix: convert fractional Gi to exact Mi:

• 3.2Gi → 3277Mi
• 2.5Gi → 2560Mi

Formula: N × 1024 = Mi (must be integer).

Helm Values as GitOps Source-of-Truth

When a Helm release has no values file in git, the live cluster becomes the only source-of-truth. To recover and version-control:

# Export current live values
helm get values <release-name> -n <namespace> -o yaml > helm/values.yaml

# Edit as needed, then apply
helm upgrade <release-name> <repo/chart> \
  --namespace <namespace> \
  --version <chart-version> \
  -f helm/values.yaml \
  --timeout 15m

Airflow-specific: uses airflow-helm/airflow chart (community chart, NOT the official airflow/airflow). Chart version 8.9.0 = Airflow 2.8.4. Workers run 4 init containers including install-pip-packages which takes 5–8 min — use --timeout 15m minimum. Do NOT use --atomic (auto-rollback fires on timeout before pods are ready).

Sandbox MNG Migration (2026-06-08, PR#411)

Migrated sandbox cluster eks-dev-262 from 7 legacy self-managed ASGs to a single SPOT-only Managed Node Group. Expected ~70% EC2 cost reduction.

Changes

• eks/main.tf: Added dev/prod ternary on eks_managed_node_groups. Dev = spot-only (SPOT, m5/m5a/m6a/m6i/m6in.large, min=2 desired=4 max=20). Prod unchanged (spot-baseline + on-demand-scaleout).
• eks-services/namespace.tf: Added lifecycle { prevent_destroy = true } on all namespaces (prevents #413-style accidental deletion).
• eks-services/namespace-imports.tf: Imported 10 dev-preview-* namespaces that existed in cluster but not in TF state.
• eks-services/ca-priority-expander.yaml: Updated priority expander ConfigMap to include spot-only at priority 10.

Incidents During Migration

• Reserved label error: eks.amazonaws.com/capacityType cannot be set in MNG labels block — AWS sets it automatically. Removed from config.
• Invalid lifecycle block: lifecycle { ignore_changes = [...] } cannot be inside a module block. Replaced with ignore_scaling_changes = true per node group.
• IAM access entry 409 drift: Partial apply deleted access entries from TF state but AWS still had them. Fixed by adding eks/access-entry-imports.tf with import block for iam-role-5 (dev-oidc-github-actions-terraform-role).
• Missing OIDC provider: enable_irsa = false in eks/main.tf means TF never manages the OIDC provider. CA and EBS CSI use IRSA — they crashed with No OpenIDConnect provider found. Fix: created OIDC provider manually via AWS CLI, then kubectl rollout restart on both deployments.

OIDC Provider Note

Both sandbox and prod OIDC providers are now managed as standalone aws_iam_openid_connect_provider.eks_oidc resources in eks/oidc-provider-imports.tf (outside the EKS module because enable_irsa = false).

• Sandbox (id/525D2368DDDC9693D79CAC104F484254): manually created 2026-06-08, imported to TF state via PR#411
• Prod (id/2E3245339DB22108B99E80633C3D7AD1): pre-existing, imported to TF state via PR#418 applied 2026-06-09

If either cluster is ever recreated, the OIDC provider must be re-created manually or enable_irsa set to true. CA and EBS CSI use IRSA — without the OIDC provider they crash with No OpenIDConnect provider found.

Result

• 19 nodes, 100% SPOT, single MNG eks-dev-262-mng-spot-only-*
• CA running, priority expander active (spot-only priority=10)
• EBS CSI running
• All namespaces protected with prevent_destroy = true

Open Issues

Known Remaining Issues

• CA expander=priority (sandbox): Sandbox now uses priority expander with spot-only at priority 10. CA will prefer spot-only MNG on scale-out.
• Airflow workers land on OD: No workers.affinity.nodeAffinity in helm values — workers scheduled randomly. Add SPOT preference to hungryhub-terraform/airflow/helm/values.yaml.
• sidekiq invalid memory limit: some sidekiq deployments have limits.memory < requests.memory. #402 tracks sidekiq-default specifically.
• Private namespaces: hh-end-user-private, hh-cosmos-private have separate base manifests. SPOT affinity and rightsizing need applying there independently (#404).
• Memory is the binding CA constraint: OD nodes at 42–68% memory utilization as of 2026-06-08. CPU rightsizing alone does not unlock CA draining — memory requests also need reviewing.

Cluster Autoscaler Verification

To confirm CA is working:

# Check which CA pod holds the leader lease
kubectl get lease cluster-autoscaler -n kube-system -o jsonpath='{.spec.holderIdentity}'

# Check leader pod logs for scaling decisions
kubectl logs -n kube-system <leader-pod> --tail=100 | grep -E "scale_down|unneeded|No candidates|Removing node"

CA logs key phrases:

• "scale_down_candidates" — nodes being considered for removal
• "Node ... has been unneeded for" — countdown to drain
• "No candidates" — no nodes eligible (often DaemonSet-only nodes)
• "Removing node" — drain fired

Related

• Kubernetes CPU Request Rightsizing
• Sidekiq KEDA Autoscaling