infra-finops — AWS Reserved Instances & Savings Plans

Repo: hungryhub-team/infra-finops AWS account: prod (202255947274) Region: ap-southeast-1 First purchase: 2026-06-10 — 1-year No Upfront RIs for 7 ElastiCache Valkey nodes + 2 RDS Aurora MySQL instances EC2 Compute Savings Plans: Base — 2026-06-10, 1-year No Upfront, $1.70/hr (prod/ec2-csp, SP id b3074e90-b219-4950-ba2c-0834ee67e79c)
Top-up: 2026-07-08, $0.80/hr (SP id 33eebfdc); $2.50/hr total (issue #21 — see Top-up log below).

Why this repo exists

On-demand pricing for long-running production nodes has no upside — we always need them. Reserved Instances (1-year No Upfront) cut costs ~30–40% for ElastiCache and ~30% for RDS with zero commitment beyond the year term. A dedicated repo keeps the financial purchases separate from application infrastructure and makes the irreversibility explicit.

What is reserved

ElastiCache — Valkey clusters (prod)

RDS — Aurora MySQL (prod)

Autoscale replicas (up to 10× db.r6g.xlarge) are not covered — purchase additional db.r6g.xlarge RIs separately if scaling is permanent.

Source of truth for cluster configs: hungryhub-terraform/valkey/local.tf and hungryhub-terraform/rds/aurora.tf.

EC2 — Compute Savings Plan (prod)

Two Compute Savings Plans cover prod EC2 on-demand compute. Unlike an RI, a Savings Plan is a $/hr commitment that auto-applies across instance families/sizes (and Fargate/Lambda) — it is not pinned to an instance type.

| Setting | Value | |—|—|—| | Plan type | Compute | | Term / payment | 1 year / No Upfront | | Commitment (base) | $1.70/hr — SP b3074e90 (2026-06-10) | | Commitment (top-up) | $0.80/hr — SP 33eebfdc (2026-07-08, issue #21) | | Total | $2.50/hr | | Offering ID | cea057b3-71c5-47cc-8d5b-307019b0c6ba | | Active | 2026-06-10 → 2027-06-10 (base), 2026-07-08 → 2027-07-08 (top-up) |

Why CSP over EC2 RI: for the prod EKS on-demand fleet (m5.xlarge) + non-EKS always-on (c6a.2xlarge, t3a.small), the Compute SP rate beats the 1yr-No-Upfront RI rate by ~15–17 pp on every type (≈53% vs ≈37% off on-demand), and the SP follows the fleet through Graviton migration / right-sizing without re-purchase. Full analysis: infra-finops/analysis/ri-recommendation.md.

Why two-tier approach (issue #21): an m5-only EC2Instance SP ($0.80/hr) was purchased manually via the AWS console on 2026-07-06, bypassing IaC. It was cancelled within the 7-day return window and replaced by a second additive Compute SP at the same $0.80/hr — achieving identical m5 coverage with Compute SP’s broader scope (EC2/Fargate/Lambda, all families/sizes). See Top-up log below.

Excluded: EKS spot baseline (Spot already discounted, SPs don’t apply), workerenv-* sandbox, EKS control plane, RDS/ElastiCache (already RI’d above).

Repo structure

prod/
  elasticache-ri/   7 Valkey nodes — aws_elasticache_reserved_cache_node (for_each)
  rds-ri/           2 Aurora MySQL — aws_rds_reserved_instance + data sources
  ec2-csp/          Compute Savings Plan — aws_savingsplans_savings_plan
  csp-monitoring/   Vantage EC2 budget and alerts
analysis/           cost-analysis snapshots (NOT terraform) — ri-recommendation.md,
                    csp-resolution-plan.md, ec2_raw.json
scripts/
  lookup-offerings.sh      queries AWS CLI for ElastiCache offering IDs (no TF data source)
  lookup-csp-offering.sh   queries AWS CLI for the Compute SP offering ID (no TF data source)
.github/workflows/
  terraform.yml     plan on PR; apply on workflow_dispatch with prod-ri-purchase gate

Provider version: prod/ec2-csp requires AWS provider ≥ 6.x (the aws_savingsplans_savings_plan resource does not exist in 5.x). All three modules are pinned to 6.49.0. The RI resources re-validated clean on 6.x once the ignore_changes below were added.

Terraform state

Bucket: hungryhub-prod-terraform-202255947274 (same as hungryhub-terraform prod state).

GitHub Actions workflow

Triggers

OIDC authentication

The workflow assumes the infra-finops-ci IAM role (created by hungryhub-iam/stacks/infra-finops-ci). No stored credentials — GitHub’s OIDC provider issues short-lived JWT tokens.

Trusted OIDC subjects:

• repo:hungryhub-team/infra-finops:pull_request
• repo:hungryhub-team/infra-finops:ref:refs/heads/main
• repo:hungryhub-team/infra-finops:environment:prod-ri-purchase

Secrets required

IAM role — infra-finops-ci

Managed in hungryhub-iam/stacks/infra-finops-ci/. Deployed via the IAM Terraform Stacks workflow (stack=infra-finops-ci, environment=prod).

Policy grants:

• elasticache:Describe*/List* on * (read-only offering lookup)
• elasticache:PurchaseReservedCacheNodesOffering + AddTagsToResource scoped to arn:aws:elasticache:*:202255947274:reserved-instance:*
• rds:Describe*/List* on *
• rds:PurchaseReservedDBInstancesOffering + AddTagsToResource scoped to arn:aws:rds:*:202255947274:ri:*
• savingsplans:Describe* + ListTagsForResource on * (read — offering lookup + plan)
• savingsplans:CreateSavingsPlan + TagResource on * (purchase — SPs are account-level, no resource-level scoping)
• S3 read/write on hungryhub-prod-terraform-202255947274/prod/*
• DynamoDB lock on terraform-backend-prod-lock

The savingsplans:* grant was added in hungryhub-iam PR #42 (2026-06-10). CreateSavingsPlan/TagResource are mutating but cannot be ARN-scoped, so they are allow-listed in hungryhub-iam’s tests/test_iam_guardrails.py (STAR_ALLOWED_ACTIONS).

Non-obvious design decisions

ElastiCache + Savings Plan offering IDs must be looked up at runtime

Unlike RDS, there is no Terraform data source for ElastiCache RI offerings or Savings Plan offerings. The workflow runs the AWS CLI and exports the result as a TF_VAR_* environment variable:

• ElastiCache: aws elasticache describe-reserved-cache-nodes-offerings → TF_VAR_offering_id_* (scripts/lookup-offerings.sh)
• Compute SP: aws savingsplans describe-savings-plans-offerings --plan-types Compute --durations 31536000 --payment-options "No Upfront" → TF_VAR_savings_plan_offering_id (scripts/lookup-csp-offering.sh)

The ec2-csp job also injects TF_VAR_hourly_commitment_usd (from the CSP_COMMITMENT_USD workflow env, default 1.70) — the hourly_commitment_usd variable has no default so CI must supply it.

aws_savingsplans_savings_plan provider-v6 quirks — ignore_changes is mandatory

The aws_savingsplans_savings_plan resource (AWS provider v6, see terraform-provider-aws#46381) has two behaviours that will silently try to re-purchase an active commitment if left unhandled:

1. Decimal normalization — the API returns commitment = "1.70000000" (8 decimals) and upfront_payment_amount = "0.00000000", while config sets "1.70" / null. Without ignore_changes, plan shows a perpetual diff and upfront_payment_amount "0.00000000" -> null forces replacement.
2. Inconsistent result on create — the apply that purchases the SP succeeds in AWS but Terraform then errors with Provider produced inconsistent result after apply (the same decimal mismatch) and taints the resource. The next plan wants to destroy + recreate (= cancel + re-buy), blocked only by prevent_destroy.

The module sets:

lifecycle {
  prevent_destroy = true
  ignore_changes  = [savings_plan_offering_id, commitment, upfront_payment_amount]
}

If you ever see the SP tainted / marked for replacement after an apply:

1. Confirm the SP is live: aws savingsplans describe-savings-plans --profile prod --region ap-southeast-1 — if state=active, the purchase already happened, do not re-apply blindly.
2. The resource is already in state (the failed apply still wrote it). Run terraform untaint aws_savingsplans_savings_plan.ec2_compute.
3. terraform plan should then show only Changes to Outputs: (0 resource changes). If it still forces replacement, an attribute is missing from ignore_changes.

This was the actual sequence on the 2026-06-10 first purchase (fixed in infra-finops PR #9).

prevent_destroy = true on every resource

All RI resources have lifecycle { prevent_destroy = true }. A terraform destroy or accidental resource removal will fail with a hard error — deliberate protection since RIs are irreversible financial purchases.

Purchases are idempotent after the first apply

AWS does not allow cancelling or refunding RIs after purchase. Terraform state tracks each purchase — re-running apply with the same offering ID is a no-op (resource already exists in state). If the state is lost and apply is re-run, AWS will create a second RI for the same node type.

Runbook: buying more RIs (e.g. adding a new cluster)

1. Identify the node type of the new cluster from hungryhub-terraform.
2. Run scripts/lookup-offerings.sh --profile prod locally to get the offering ID.
3. Add an entry to local.reservations in prod/elasticache-ri/main.tf (or add a new data source + resource in prod/rds-ri/).
4. Open a PR — the plan job runs automatically and shows cost impact.
5. After review, trigger Actions → Terraform RI → Run workflow with action=apply. Approve the prod-ri-purchase environment gate.

Runbook: topping up the Compute Savings Plan

SP commitments are additive — to increase coverage you buy a second plan, you do not modify the first.

1. Re-run the steady-state analysis (issue #4 playbook): pull 2+ weeks of EKS on-demand node count + AWS Compute Optimizer, confirm the floor.
2. Decide the increment (cap at the full-fleet $3.73/hr from analysis/ri-recommendation.md unless a fresh analysis raises it).
3. Bump CSP_COMMITMENT_USD in the workflow (or hourly_commitment_usd in tfvars) and add a second aws_savingsplans_savings_plan resource — do not edit the existing one (its commitment is in ignore_changes and it is prevent_destroy).
4. PR → plan → apply through the prod-ri-purchase gate.
5. Record the new SP id + commitment here.

Issue #21 took the additive approach — UpdateSavingsPlan is unavailable via SDK/CLI, so a second aws_savingsplans_savings_plan resource was created at $0.80/hr alongside the existing $1.70 SP. The cancelled EC2Instance SP ($0.80/hr) was returned within its 7-day window. Total effective commitment: $2.50/hr across two SPs. This additive approach is the canonical pattern for all future top-ups.

HOLD the top-up if a Graviton migration / Rails-upgrade fleet reshape is imminent — re-evaluate post-migration so you don’t strand commitment on instance types you’re about to retire.

Top-up log

Verifying purchases in AWS console

• ElastiCache: AWS Console → ElastiCache → Reserved Cache Nodes
• RDS: AWS Console → RDS → Reserved Instances
• Compute SP: AWS Console → Cost Explorer → Savings Plans

Or via CLI:

aws elasticache describe-reserved-cache-nodes --profile prod --region ap-southeast-1
aws rds describe-reserved-db-instances --profile prod --region ap-southeast-1
aws savingsplans describe-savings-plans --profile prod --region ap-southeast-1

Monitoring: post-CSP budget alerts (issue #7, 2026-06-22)

Two Vantage budget alerts were configured to signal fleet-growth or other over-baseline on-demand spend that the CSP does not cover.

What was created (2026-06-22)

The alert thresholds are managed as percentages of the budget amount, not as dollar amounts; $3,000 is only a round gloss on 107% of $2,800. Budget and both alerts are evaluated for the full month (period_to_track = start_of_the_month, no duration filter).

The concrete Vantage object tokens (Cost Report, Budget, Budget Alerts) are omitted from this doc — look them up via the Vantage API if needed:

curl -s -H "Authorization: Bearer $VANTAGE_API_TOKEN" \
  "https://api.vantage.sh/v2/cost_reports" | jq '.cost_reports[] | select(.title | contains("AWS Prod EC2"))'
budget_token="$(
  curl -s -H "Authorization: Bearer $VANTAGE_API_TOKEN" \
    "https://api.vantage.sh/v2/budgets" | jq -r '.budgets[] | select(.name | contains("Post-CSP Ceiling")) | .token'
)"
curl -s -H "Authorization: Bearer $VANTAGE_API_TOKEN" \
  "https://api.vantage.sh/v2/budget_alerts" | jq --arg budget_token "$budget_token" \
  '.budget_alerts[] | select(.budget_tokens[]? == $budget_token)'

Or via the Vantage web UI: Cost Reports → “AWS Prod EC2 (MTD)” → Budgets tab.

Where the alerts go. The only notification destination the managed configuration declares is the recipient email saiqulhaq@hungryhub.com — both alerts set recipient_emails = ["saiqulhaq@hungryhub.com"] and nothing else. No Slack destination is declared in Terraform: hungryhub-team/infra-finops contains no Slack reference at all, and the module sets no recipient_channels.

The Vantage objects themselves were last observed carrying integration_provider: slack with recipient_channels = [] — an integration flag with no channel behind it — in the import plan captured on 2026-09-10 in infra-finops PR #14. That is a point-in-time read, not a current fact, and whether the Vantage org-level Slack integration is still connected is not recorded anywhere in this repository. Treat email as the confirmed delivery path. To settle it, a Vantage account admin must open Settings → Slack in the Vantage web UI and confirm the connection and the target channel; until someone does that and the answer is recorded here, do not assume a budget alert reaches a Slack channel.

Whether any of these alerts has actually fired is likewise unknown from this repository — nothing records delivery history. Checking it means reading the Vantage budget/alerts UI (or the Cost Report) for the current month, which is a live-account check and is deliberately not performed when this doc is updated.

Why these thresholds

The managed values are the budget’s monthly amount ($2,800, single period starting 2026-07-01) and the two alert thresholds (100% and 107% of that amount). They are exactly what prod/csp-monitoring/main.tf declares, and they are retained as-is — they have not been re-tuned and are not re-tuned by this doc.

What the alerts actually watch. Both track total monthly AWS EC2 spend for account 202255947274, not the Savings Plan commitment. The 100% alert fires at the budgeted ceiling; the 107% alert is the approximate $3,000 guard above it, kept as it was when the alerts were first created — infra-finops’ 2026-09-09 design note records it that way. The intent of the guard is to catch fleet drift the SP no longer covers (a node group scaled out, or non-EKS EC2 added without being in the plan). Both alerts are monthly aggregates; per-hour monitoring is AWS Cost Explorer and the prod/ec2-csp module’s plan output. See “Open follow-up” below for a per-hour alarm.

What this doc deliberately does not claim any more. The $2,719/mo “theoretical floor” and $5,775/mo on-demand figures come from the 2026-06-10 analysis snapshot in infra-finops/analysis/ri-recommendation.md. They no longer justify $2,800: the commitment this repo actually manages is the deliberately conservative two-tier Savings Plan in the EC2 — Compute Savings Plan table above (base $1.70/hr + $0.80/hr top-up = $2.50/hr, issue #21), which does not by itself derive a $2,800/mo spend ceiling. No re-derivation of $2,800 has been performed, and no owner for this threshold is recorded anywhere — that gap is the reason the value stands unchanged rather than being rewritten from a superseded derivation.

If you change it. Re-derive the ceiling from a fresh EC2 spend analysis first, get it reviewed, then change periods[0].amount / the threshold values in prod/csp-monitoring/main.tf. Editing this doc changes nothing that Vantage evaluates.

Do not delete these objects — they are Terraform-managed

All four Vantage objects are declared and imported by hungryhub-team/infra-finops → prod/csp-monitoring/ (one vantage_cost_report, one vantage_budget, two vantage_budget_alerts), and every one of them carries lifecycle { prevent_destroy = true }. A terraform destroy against that module fails with a hard error instead of tearing the objects down, and a Vantage apply is a manual dispatch from main only.

So: do not delete them — from the Vantage UI or with DELETE calls to the Vantage API. Deleting them by hand drifts the module out of state, and the budget is the only record of the spend ceiling this repo monitors against.

The Vantage object tokens are recorded in the module’s Terraform state: prod/csp-monitoring/state.tfstate in the hungryhub-prod-terraform-202255947274 bucket (see the Terraform state section above).

The one-shot provisioning script that originally created these objects no longer exists. Infra-finops PR #14 (merged 2026-09-11) replaced it with the Terraform module; there is no script to re-run.

Open follow-up — CSP utilization alert (task #4 in issue #7)

Not done. The plan is a Terraform module for a CloudWatch metric alarm on AWS/SavingsPlans Utilization, fanning out through an SNS topic to a notification channel. This catches under-utilisation (fleet shrank → commitment being wasted) which the Vantage budget alert does not.

That module does not exist yet. No CSP-utilisation alarm is declared anywhere in hungryhub-team/infra-finops today. Pick a module path that is not prod/csp-monitoring/ — that path already exists and belongs to the budget module described above, which owns Vantage objects rather than a CloudWatch alarm.

The proposed <90% utilisation sustained for one week is an unratified proposal, not an approved threshold: it is recorded here without a recorded owner or derivation, and the metric/scrape mechanism it assumes has not been verified against AWS documentation by this repo. Whoever picks this up should confirm both the metric source and the threshold before treating them as the design.

Calendar reminders (task #5 in issue #7)

Manual, non-code. Set these on the team’s FinOps calendar:

• Monthly (1st of month) — Vantage dashboard check vs commitment; confirm no alert fired last month (nothing in this repo records delivery history — read the Vantage budget/alerts view for the closed month); spot-check the AWS bill in Cost Explorer.
• Quarterly — re-pull aws ec2 describe-instances, re-run the analysis (issue #4 playbook). Fleet shape changes shift the recommendation. Commit a new dated snapshot under infra-finops/analysis/.
• Pre-2027-07-08 — the top-up Compute Savings Plan ($0.80/hr, added 2026-07-08 in issue #21) ends its 1-year term. Renewing means a new additive SP, not an edit of the existing one — see “Runbook: topping up the Compute Savings Plan” above.
• Pre-2027-06-10 — renew / switch-to-3yr / drop-to-OD decision on the base $1.70/hr SP. Do not auto-rollover if fleet shrank — the 1yr term expires and you’ll have data by then on whether the savings are worth the lock-in.

Related

• infra-finops/analysis/ri-recommendation.md — full RI-vs-CSP-vs-on-demand cost analysis (2026-06-10)
• infra-finops/analysis/csp-resolution-plan.md — CSP rollout plan + issue dependency graph (#2–#7)
• Vantage MCP for AI Agents — cost monitoring (re-eval cadence, budget alerts)
• hungryhub-iam/stacks/infra-finops-ci/ — the CI role + its savingsplans:* grant
• infra-finops issue #7 — post-CSP monitoring (this section tracks the work)