Sandbox: the “orphaned” available EBS volume is a LIVE ClickHouse PV — DO NOT DELETE

TL;DR for cost-audit agents: In the sandbox account (079994049689, ap-southeast-1) there is a 100 GiB gp3 EBS volume in available (detached) state that looks like an orphan. It is not. It is the EBS CSI PersistentVolume backing the ClickHouse StatefulSet on eks-dev-262. Deleting it = permanent ClickHouse data loss. Skip it in every cost cleanup.

Why this doc exists

A cost-reduction pass (hungryhub-terraform#441, 2026-06-11) flagged an available EBS volume as an “orphaned volume left over from the ASG→MNG migration.” It is a false positive. The volume is a Kubernetes CSI-provisioned PV for a StatefulSet — available just means the ClickHouse pod was rescheduling / scaled down at the moment of the scan. This doc records the verification so the next agent does not re-flag and delete it.

The volume

How to recognise a k8s PV (the general rule)

An EBS volume is a Kubernetes-managed PersistentVolume — never delete it as an orphan — if it carries any of these tags:

• kubernetes.io/created-for/pv/name
• kubernetes.io/created-for/pvc/name
• kubernetes.io/created-for/pvc/namespace
• kubernetes.io/cluster/<cluster-name> = owned
• ebs.csi.aws.com/cluster = true
• CSIVolumeName

A truly orphaned volume has none of these — typically only a Name tag or no tags at all, and no eks-* / cluster association.

A detached (available) CSI volume for a StatefulSet almost always means the pod is rescheduling or scaled to zero, not that the data is abandoned. The PV/PVC objects still exist in the cluster and will rebind. Deleting the EBS volume orphans the PV and loses the data.

Verify before touching ANY available EBS volume in a cluster account

# 1. Inspect the volume's tags — look for the kubernetes.io / ebs.csi.aws.com markers above
aws ec2 describe-volumes --profile sandbox --region ap-southeast-1 \
  --filters Name=status,Values=available \
  --query 'Volumes[].{id:VolumeId,size:Size,created:CreateTime,tags:Tags}'

# 2. If it has a kubernetes.io/created-for/pvc/* tag, confirm the PVC/PV/StatefulSet in-cluster
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  -n clickhouse get pvc,pv,statefulset,pods

# 3. Match the PV by its EBS volume handle before concluding anything
kubectl --context=arn:aws:eks:ap-southeast-1:079994049689:cluster/eks-dev-262 \
  get pv -o json | \
  python3 -c "import json,sys; [print(p['metadata']['name'], p['spec'].get('csi',{}).get('volumeHandle')) for p in json.load(sys.stdin)['items']]"

If step 2/3 shows the PV/PVC still exist (even if the pod is not currently running), the volume is in use. Do not delete.

Related

• ClickHouse on dev EKS: ClickStack_EKS_Dev_Runbook_20260605100000.md, ClickHouse_Access_Plan.md
• EKS cost work: EKS_Cost_Optimization_2026.md
• Origin: hungryhub-terraform#441 (the cost item that mis-flagged this volume)