Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Failed Pods When Using Graviton

Problem

After migrating workloads to AWS Graviton (ARM64), several pods entered CrashLoopBackOff. This is critical because API and worker services become unstable and may drop jobs/requests.

Environment

  • Platform: AWS EKS
  • Architecture: mixed x86_64 and ARM64 (Graviton)
  • Namespace impact: hungryhub, inventory, kube-system, utility
  • Region / EKS version / node type: fill in from the affected cluster runbook before incident closure

Expected vs actual behavior

  • Expected: pods should remain Running and pass readiness checks.
  • Actual: multiple pods restarted repeatedly and showed CrashLoopBackOff.

Symptoms

# Example command
kubectl get pods -A

Common pattern observed:

  • Some pods run normally on one node group.
  • The same deployment crashes on another node group.
  • Restarts increase quickly (CrashLoopBackOff).

Diagnosis steps

  1. Identify failing pods:
    • kubectl get pods -A | grep CrashLoopBackOff
  2. Check logs:
    • kubectl logs <pod> -n <namespace> --previous
  3. Check events:
    • kubectl describe pod <pod> -n <namespace>
  4. Validate image architecture:
    • Confirm image supports linux/arm64.
  5. Check node architecture:
    • kubectl get nodes -o wide
  6. Verify native dependencies (gems/libs/binaries) are ARM-compatible.

Likely causes

  • Container image built only for amd64.
  • Native gem/library incompatibility on arm64.
  • Startup command depends on binaries unavailable in ARM image.
  • Misconfigured nodeSelector/affinity scheduling pods to unsupported nodes.

Resolution

  1. Build and push multi-arch images (amd64 + arm64).
  2. Pin unsupported workloads temporarily to amd64 node group.
  3. Rebuild native dependencies in ARM-compatible base image.
  4. Redeploy and monitor restart count for at least 30 minutes.

Reproduction notes

  • Deploy a service image without ARM build to a Graviton node pool.
  • Observe pod status transitions to CrashLoopBackOff.

Metadata to record in incident report

  • Failure start timestamp
  • Affected namespaces/deployments
  • Error signatures from logs
  • Node architecture mapping
  • Fix commit / image tag used