AI Agent Lessons from Infrastructure Work

A collection of real patterns from running Claude Code on hh-infra. This is a companion to Debugging a Real Issue with an AI Agent (which covers a single linear session) and Agentic Coding with OpenCode and MiniMax (the setup guide).

These examples are drawn from infra and ops work — terraform, kubernetes, multi-repo coordination — rather than application debugging.

1. Prod bugs can hide in infra config with no error

What happened: Nag messages (⚠️ No tests included, 🔗 Missing tracking link) were appearing in #github-pr-reviews with a “⚙️ dev mode” footer. The app was working correctly from the Rails side — the bug was in terraform.

RELAY_ENV was never set in the ECS task definition in hungryhub-ai/ecs-relay/main.tf. The env var defaults to "dev" in the Python code:

RELAY_ENV = os.environ.get("RELAY_ENV", "dev")  # "prod" in ECS prod task definition

The comment described the intent. The terraform never implemented it. Both dev and prod containers ran as "dev", silently routing author DMs to the public channel.

Fix: one line in hungryhub-ai/ecs-relay/main.tf, PR merged, terraform apply, rolling restart. Total time: ~15 minutes once the agent found the root cause.

Pattern: when a behaviour difference between dev and prod is hard to explain, check whether env vars that gate that behaviour are actually set in the prod task/deployment definition.

2. Stale docs produce confident wrong answers

What happened: while adding a new env var to hh-server, the agent gave detailed, plausible instructions to “add the key to the TFVARS_PROD GitHub Secret.” That secret no longer exists — tfvars have been built from AWS Secrets Manager since PR #182 (feat: build terraform tfvars from aws secrets manager).

The agent wasn’t hallucinating. It was reading stale documentation that described the old workflow accurately. The instructions looked correct because they matched the old pattern exactly.

Fix: found the real path (terraform/tfvars/{env}/{suffix} in Secrets Manager), added the key there, then updated 10 stale references across AGENTS.md, module READMEs, .github/copilot-instructions.md, and .claude/agents/terraform-author.md.

Pattern: when an agent gives confident instructions that don’t work, check whether the docs it was reading describe the current system or a previous version. Fix the docs as part of the same task.

3. Multi-cluster routing is undocumented until it costs you 30 minutes

What happened: patching POSTHOG_ADMIN_API_KEY into the hh-server configmap. The agent patched hungryhub-server-config in the hungryhub namespace on EKS prod — a reasonable first guess. Wrong namespace. hungryhub.com (and /admin) is served by the hh-end-user-public namespace, not hungryhub.

Finding this required reading ingresses across three clusters: EKS prod, DigitalOcean prod, DigitalOcean staging.

The right answer:

Extra trap: kubectl patches to a configmap are overwritten by the next terraform apply hungryhub-apps. Always follow a live patch with an AWS Secrets Manager update + terraform PR to make it permanent.

Pattern: which cluster/namespace serves which URL is exactly the kind of tribal knowledge that costs 30 minutes every time it isn’t written down.

4. Code shows what was built. Only docs preserve why.

What happened: hungryhub-iam is a separate repo from hungryhub-terraform. Nothing in the code explains why. An agent (or engineer) looking at the two repos might reasonably propose merging them back “to reduce complexity.”

The reason they’re separate is a deliberate security decision: if IAM/OIDC config lived in hungryhub-terraform, the CI role that runs that repo’s workflows could modify its own trust policy via a PR — a privilege escalation path. Separating the repos means hungryhub-terraform’s CI role has no iam:CreateRole permissions.

Fix: one paragraph in the hungryhub-iam knowledge-base doc captures the decision permanently. See hungryhub-iam — Architecture and Operations.

Pattern: architectural decisions that were made for security reasons are the most dangerous to lose. The code is the outcome; the doc is the reason. Without the reason, the outcome gets reversed.

5. An agent is as good as its starting point

The single highest-leverage habit across all of these sessions: search knowledge-base before starting any task.

grep -ri "<keyword>" knowledge-base/src --include="*.md"

In every example above, the investigation either found a gap (and filled it) or found stale docs (and corrected them). The sessions that started with a knowledge-base search finished faster. The ones that skipped it re-discovered things that were already known.

This applies to every task — not just explicit debug requests. Setting up a new integration? Search it. Touching terraform variables? Search it. Working in a namespace you haven’t touched before? Search it.

Related

• Debugging a Real Issue with an AI Agent — single linear walkthrough (application debugging)
• Agentic Coding with OpenCode and MiniMax — setup and workflow guide
• hungryhub-terraform — Architecture and Operations
• hungryhub-iam — Architecture and Operations