How to Access the Hetzner k3s Cluster

The team runs a k3s cluster on Hetzner Cloud (provisioned with the kube-hetzner Terraform module). It hosts the GitHub Actions self-hosted runners (ARC) and a few supporting workloads. This page is how to get kubectl access to it.

For the runner architecture and failure modes, see Hetzner k3s Self-Hosted Runners. To add a runner for a new repo, see How to Add a Self-Hosted Runner to a Repo.

At a glance

The cluster’s kubectl auth is certificate-based. It does not use the node SSH key (id_ed25519) — that key is only for Terraform to provision the Hetzner VMs. So having the kubeconfig is enough to use the cluster; you do not need the SSH key for kubectl.

1. Get the kubeconfig

There are two sources. Prefer Terraform output (always current); the committed file is a convenience copy.

Option A — from Terraform (authoritative)

The kube-hetzner module exposes the kubeconfig as a Terraform output (kube.tf output "kubeconfig"):

cd hungryhub-terraform/github-runners/hetzner-k3s-arc-runners

# Terraform needs the Hetzner API token. Set it as an env var (do NOT hardcode):
export TF_VAR_hcloud_token=<hetzner-cloud-api-token>

terraform init   # first time only
terraform output --raw kubeconfig > k3s_kubeconfig.yaml
chmod 600 k3s_kubeconfig.yaml

Option B — the committed convenience copy

hungryhub-terraform/github-runners/hetzner-k3s-arc-runners/k3s_kubeconfig.yaml holds a working kubeconfig already. Use it directly:

export KUBECONFIG="$PWD/k3s_kubeconfig.yaml"

2. Use it

The kubeconfig defines a single context named k3s. Point kubectl at it via KUBECONFIG (recommended — keeps it isolated from your other clusters):

cd hungryhub-terraform/github-runners/hetzner-k3s-arc-runners
export KUBECONFIG="$PWD/k3s_kubeconfig.yaml"

kubectl get nodes
# k3s-autoscale-arm-medium-...   Ready   ...
# k3s-autoscale-large-...        Ready   ...

kubectl get pods -n arc-systems      # ARC controller + per-set listeners

Or merge it into your ~/.kube/config and switch contexts:

KUBECONFIG=~/.kube/config:./k3s_kubeconfig.yaml kubectl config view --flatten > /tmp/merged
mv /tmp/merged ~/.kube/config
kubectl config use-context k3s

Watch your context. Most engineers’ default context is the EKS dev cluster. Always confirm with kubectl config current-context (should print k3s) before running commands against the runner cluster. The ARC setup scripts pin KUBECONFIG to k3s_kubeconfig.yaml automatically via _arc_github_app.sh so they don’t hit the wrong cluster.

3. Common checks

export KUBECONFIG=.../k3s_kubeconfig.yaml

# Nodes (autoscaling pool — count varies)
kubectl get nodes

# ARC controller + all runner-set listeners
kubectl get pods -n arc-systems | grep -E "controller|listener"

# Runner scale sets and their min/max
kubectl get autoscalingrunnerset -A

# Live runner pods for a given set
kubectl get pods -n arc-runners-hh-server-cpu4

SSH to the nodes (rarely needed)

kubectl covers almost everything. If you genuinely need to SSH a node (debug k3s itself, inspect the host), that uses the Hetzner node SSH key referenced in kube.tf (ssh_private_key = file("~/.ssh/id_ed25519")). Node IPs are visible in the Hetzner Cloud console or hcloud server list.

The node SSH key is being moved to AWS Secrets Manager so the DevOps squad can retrieve it without depending on one person’s laptop — see the follow-up in hungryhub-terraform#420.

Security notes

• The kubeconfig embeds a client cert + key — treat it as a credential. It is currently committed for convenience; do not paste it into tickets/Slack.
• kube.tf historically hardcoded the Hetzner API token. It should be supplied via TF_VAR_hcloud_token (env var) and the hardcoded value rotated — tracked as a security follow-up.

Related

• Hetzner k3s Self-Hosted Runners — runner pool layout, auth (GitHub Apps), failure modes.
• How to Add a Self-Hosted Runner to a Repo
• Terraform: hungryhub-terraform/github-runners/hetzner-k3s-arc-runners/kube.tf