infra-finops — AWS Reserved Instances & Savings Plans

Repo: hungryhub-team/infra-finops AWS account: prod (202255947274) Region: ap-southeast-1 First purchase: 2026-06-10 — 1-year No Upfront RIs for 7 ElastiCache Valkey nodes + 2 RDS Aurora MySQL instances EC2 Compute Savings Plan: 2026-06-10 — 1-year No Upfront, $2.50/hr (prod/ec2-csp, SP id b3074e90-b219-4950-ba2c-0834ee67e79c)
Topped up from $1.70 on 2026-07-06 (issue #21 — see Top-up log below).

Why this repo exists

On-demand pricing for long-running production nodes has no upside — we always need them. Reserved Instances (1-year No Upfront) cut costs ~30–40% for ElastiCache and ~30% for RDS with zero commitment beyond the year term. A dedicated repo keeps the financial purchases separate from application infrastructure and makes the irreversibility explicit.

What is reserved

ElastiCache — Valkey clusters (prod)

RDS — Aurora MySQL (prod)

Autoscale replicas (up to 10× db.r6g.xlarge) are not covered — purchase additional db.r6g.xlarge RIs separately if scaling is permanent.

Source of truth for cluster configs: hungryhub-terraform/valkey/local.tf and hungryhub-terraform/rds/aurora.tf.

EC2 — Compute Savings Plan (prod)

Two Compute Savings Plans cover prod EC2 on-demand compute. Unlike an RI, a Savings Plan is a $/hr commitment that auto-applies across instance families/sizes (and Fargate/Lambda) — it is not pinned to an instance type.

| Setting | Value | |—|—|—| | Plan type | Compute | | Term / payment | 1 year / No Upfront | | Commitment (base) | $1.70/hr — SP b3074e90 (2026-06-10) | | Commitment (top-up) | $0.80/hr — SP 33eebfdc (2026-07-08, issue #21) | | Total | $2.50/hr | | Offering ID | cea057b3-71c5-47cc-8d5b-307019b0c6ba | | Active | 2026-06-10 → 2027-06-10 (base), 2026-07-08 → 2027-07-08 (top-up) |

Why CSP over EC2 RI: for the prod EKS on-demand fleet (m5.xlarge) + non-EKS always-on (c6a.2xlarge, t3a.small), the Compute SP rate beats the 1yr-No-Upfront RI rate by ~15–17 pp on every type (≈53% vs ≈37% off on-demand), and the SP follows the fleet through Graviton migration / right-sizing without re-purchase. Full analysis: infra-finops/analysis/ri-recommendation.md.

Why two-tier approach (issue #21): an m5-only EC2Instance SP ($0.80/hr) was purchased manually via the AWS console on 2026-07-06, bypassing IaC. It was cancelled within the 7-day return window and replaced by a second additive Compute SP at the same $0.80/hr — achieving identical m5 coverage with Compute SP’s broader scope (EC2/Fargate/Lambda, all families/sizes). See Top-up log below.

Excluded: EKS spot baseline (Spot already discounted, SPs don’t apply), workerenv-* sandbox, EKS control plane, RDS/ElastiCache (already RI’d above).

Repo structure

prod/
  elasticache-ri/   7 Valkey nodes — aws_elasticache_reserved_cache_node (for_each)
  rds-ri/           2 Aurora MySQL — aws_rds_reserved_instance + data sources
  ec2-csp/          Compute Savings Plan — aws_savingsplans_savings_plan
analysis/           cost-analysis snapshots (NOT terraform) — ri-recommendation.md,
                    csp-resolution-plan.md, ec2_raw.json
scripts/
  lookup-offerings.sh      queries AWS CLI for ElastiCache offering IDs (no TF data source)
  lookup-csp-offering.sh   queries AWS CLI for the Compute SP offering ID (no TF data source)
.github/workflows/
  terraform.yml     plan on PR; apply on workflow_dispatch with prod-ri-purchase gate

Provider version: prod/ec2-csp requires AWS provider ≥ 6.x (the aws_savingsplans_savings_plan resource does not exist in 5.x). All three modules are pinned to 6.49.0. The RI resources re-validated clean on 6.x once the ignore_changes below were added.

Terraform state

Bucket: hungryhub-prod-terraform-202255947274 (same as hungryhub-terraform prod state).

GitHub Actions workflow

Triggers

OIDC authentication

The workflow assumes the infra-finops-ci IAM role (created by hungryhub-iam/stacks/infra-finops-ci). No stored credentials — GitHub’s OIDC provider issues short-lived JWT tokens.

Trusted OIDC subjects:

• repo:hungryhub-team/infra-finops:pull_request
• repo:hungryhub-team/infra-finops:ref:refs/heads/main
• repo:hungryhub-team/infra-finops:environment:prod-ri-purchase

Secrets required

IAM role — infra-finops-ci

Managed in hungryhub-iam/stacks/infra-finops-ci/. Deployed via the IAM Terraform Stacks workflow (stack=infra-finops-ci, environment=prod).

Policy grants:

• elasticache:Describe*/List* on * (read-only offering lookup)
• elasticache:PurchaseReservedCacheNodesOffering + AddTagsToResource scoped to arn:aws:elasticache:*:202255947274:reserved-instance:*
• rds:Describe*/List* on *
• rds:PurchaseReservedDBInstancesOffering + AddTagsToResource scoped to arn:aws:rds:*:202255947274:ri:*
• savingsplans:Describe* + ListTagsForResource on * (read — offering lookup + plan)
• savingsplans:CreateSavingsPlan + TagResource on * (purchase — SPs are account-level, no resource-level scoping)
• S3 read/write on hungryhub-prod-terraform-202255947274/prod/*
• DynamoDB lock on terraform-backend-prod-lock

The savingsplans:* grant was added in hungryhub-iam PR #42 (2026-06-10). CreateSavingsPlan/TagResource are mutating but cannot be ARN-scoped, so they are allow-listed in hungryhub-iam’s tests/test_iam_guardrails.py (STAR_ALLOWED_ACTIONS).

Non-obvious design decisions

ElastiCache + Savings Plan offering IDs must be looked up at runtime

Unlike RDS, there is no Terraform data source for ElastiCache RI offerings or Savings Plan offerings. The workflow runs the AWS CLI and exports the result as a TF_VAR_* environment variable:

• ElastiCache: aws elasticache describe-reserved-cache-nodes-offerings → TF_VAR_offering_id_* (scripts/lookup-offerings.sh)
• Compute SP: aws savingsplans describe-savings-plans-offerings --plan-types Compute --durations 31536000 --payment-options "No Upfront" → TF_VAR_savings_plan_offering_id (scripts/lookup-csp-offering.sh)

The ec2-csp job also injects TF_VAR_hourly_commitment_usd (from the CSP_COMMITMENT_USD workflow env, default 1.70) — the hourly_commitment_usd variable has no default so CI must supply it.

aws_savingsplans_savings_plan provider-v6 quirks — ignore_changes is mandatory

The aws_savingsplans_savings_plan resource (AWS provider v6, see terraform-provider-aws#46381) has two behaviours that will silently try to re-purchase an active commitment if left unhandled:

1. Decimal normalization — the API returns commitment = "1.70000000" (8 decimals) and upfront_payment_amount = "0.00000000", while config sets "1.70" / null. Without ignore_changes, plan shows a perpetual diff and upfront_payment_amount "0.00000000" -> null forces replacement.
2. Inconsistent result on create — the apply that purchases the SP succeeds in AWS but Terraform then errors with Provider produced inconsistent result after apply (the same decimal mismatch) and taints the resource. The next plan wants to destroy + recreate (= cancel + re-buy), blocked only by prevent_destroy.

The module sets:

lifecycle {
  prevent_destroy = true
  ignore_changes  = [savings_plan_offering_id, commitment, upfront_payment_amount]
}

If you ever see the SP tainted / marked for replacement after an apply:

1. Confirm the SP is live: aws savingsplans describe-savings-plans --profile prod --region ap-southeast-1 — if state=active, the purchase already happened, do not re-apply blindly.
2. The resource is already in state (the failed apply still wrote it). Run terraform untaint aws_savingsplans_savings_plan.ec2_compute.
3. terraform plan should then show only Changes to Outputs: (0 resource changes). If it still forces replacement, an attribute is missing from ignore_changes.

This was the actual sequence on the 2026-06-10 first purchase (fixed in infra-finops PR #9).

prevent_destroy = true on every resource

All RI resources have lifecycle { prevent_destroy = true }. A terraform destroy or accidental resource removal will fail with a hard error — deliberate protection since RIs are irreversible financial purchases.

Purchases are idempotent after the first apply

AWS does not allow cancelling or refunding RIs after purchase. Terraform state tracks each purchase — re-running apply with the same offering ID is a no-op (resource already exists in state). If the state is lost and apply is re-run, AWS will create a second RI for the same node type.

Runbook: buying more RIs (e.g. adding a new cluster)

1. Identify the node type of the new cluster from hungryhub-terraform.
2. Run scripts/lookup-offerings.sh --profile prod locally to get the offering ID.
3. Add an entry to local.reservations in prod/elasticache-ri/main.tf (or add a new data source + resource in prod/rds-ri/).
4. Open a PR — the plan job runs automatically and shows cost impact.
5. After review, trigger Actions → Terraform RI → Run workflow with action=apply. Approve the prod-ri-purchase environment gate.

Runbook: topping up the Compute Savings Plan

SP commitments are additive — to increase coverage you buy a second plan, you do not modify the first.

1. Re-run the steady-state analysis (issue #4 playbook): pull 2+ weeks of EKS on-demand node count + AWS Compute Optimizer, confirm the floor.
2. Decide the increment (cap at the full-fleet $3.73/hr from analysis/ri-recommendation.md unless a fresh analysis raises it).
3. Bump CSP_COMMITMENT_USD in the workflow (or hourly_commitment_usd in tfvars) and add a second aws_savingsplans_savings_plan resource — do not edit the existing one (its commitment is in ignore_changes and it is prevent_destroy).
4. PR → plan → apply through the prod-ri-purchase gate.
5. Record the new SP id + commitment here.

Issue #21 took the additive approach — UpdateSavingsPlan is unavailable via SDK/CLI, so a second aws_savingsplans_savings_plan resource was created at $0.80/hr alongside the existing $1.70 SP. The cancelled EC2Instance SP ($0.80/hr) was returned within its 7-day window. Total effective commitment: $2.50/hr across two SPs. This additive approach is the canonical pattern for all future top-ups.

HOLD the top-up if a Graviton migration / Rails-upgrade fleet reshape is imminent — re-evaluate post-migration so you don’t strand commitment on instance types you’re about to retire.

Top-up log

Verifying purchases in AWS console

• ElastiCache: AWS Console → ElastiCache → Reserved Cache Nodes
• RDS: AWS Console → RDS → Reserved Instances
• Compute SP: AWS Console → Cost Explorer → Savings Plans

Or via CLI:

aws elasticache describe-reserved-cache-nodes --profile prod --region ap-southeast-1
aws rds describe-reserved-db-instances --profile prod --region ap-southeast-1
aws savingsplans describe-savings-plans --profile prod --region ap-southeast-1

Related

• infra-finops/analysis/ri-recommendation.md — full RI-vs-CSP-vs-on-demand cost analysis (2026-06-10)
• infra-finops/analysis/csp-resolution-plan.md — CSP rollout plan + issue dependency graph (#2–#7)
• Vantage MCP for AI Agents — cost monitoring (re-eval cadence, budget alerts)
• hungryhub-iam/stacks/infra-finops-ci/ — the CI role + its savingsplans:* grant