Kubernetes CPU Request Rightsizing
Concepts
| Setting | What It Does | Who Uses It |
|---|---|---|
resources.requests.cpu | Reserved CPU on the node; affects scheduling | Cluster Autoscaler, scheduler |
resources.limits.cpu | Hard ceiling; pod is throttled if it exceeds this | Linux cgroups (runtime) |
Key insight: a pod can burst up to its CPU limit regardless of its request. The request is purely a scheduler hint. Overprovisioning requests wastes node capacity and causes Cluster Autoscaler to provision more nodes than needed.
Why Over-Requesting Costs Money
Cluster Autoscaler scales nodes based on requests, not actual usage. If all pods request 850m but only use 200m on average, CA thinks the cluster is full at ~50% actual load and adds nodes unnecessarily.
Example: hungryhub-server had 850m request, 2000m limit. Typical usage was 150–400m. With 56 replicas in hh-end-user-public alone, that’s 56 × 850m = 47.6 vCPU reserved vs ~22 vCPU actually consumed.
Identifying Overprovisioned Workloads
Method 1: kubectl top
kubectl top pods -n <namespace> --sort-by=cpu
kubectl top nodes
Compare requests (from kubectl describe node) vs actual usage from kubectl top.
Method 2: Vantage cost report Vantage flags Kubernetes workloads with “Workload overprovisioned” and shows estimated savings. Download the Provider Resource Report CSV from Vantage → Cost Reports.
Method 3: Grafana Query actual CPU usage over 7–30 days:
rate(container_cpu_usage_seconds_total{container="service", namespace="hh-end-user-public"}[5m])
Set request to ~P95 actual usage with headroom.
Safe Rightsizing Procedure
- Measure: gather 7+ days of P95 CPU usage from Grafana/Prometheus.
- Set request to P95 + 20% headroom. Never set below P50.
- Keep limit unchanged — this preserves burst capacity.
- Apply via PR to the base deployment manifest (changes propagate to all namespaces sharing that base).
- Monitor for 30 min post-deploy: watch for OOMKilled, CrashLoop, or CPU throttling alerts.
kubectl patch (quick test on staging)
kubectl patch deployment hungryhub-server -n hh-end-user-public \
--type=json \
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"300m"}]'
Worked Example: hungryhub-server (June 2026)
- Before: 850m request, 2000m limit
- After: 300m request, 2000m limit (limit unchanged)
- Namespaces affected:
hh-end-user-public,hh-syn-public,hh-cosmos-public,hh-vendor-public(shared base manifest) - PR: hh-server#8187 + hh-infra worktree branch
hh-server-405-rightsize-cpu - Estimated savings: ~$1,016/mo across all 4 namespaces (Vantage)
- Observed: CPU bursts to 1100m during peak traffic (within 2000m limit) — normal and expected
minReplicaCount also reduced
hh-end-user-public ScaledObject minReplicaCount was also lowered from 20 → 10, allowing KEDA to scale down during off-peak hours.
Memory Request Caveat
Memory requests are the binding Cluster Autoscaler constraint in most HungryHub namespaces. After the CPU change, CA still reported nodes blocked by memory utilization (52–95%). Address memory requests separately after confirming CPU change is stable.
Scope Note
manifest/base-aws/prod/public/deployments/hungryhub-server.yaml is shared across all 4 prod public namespaces. A single change propagates everywhere. Private namespaces (hh-end-user-private, etc.) have separate base manifests.