OpenRouter API Key Leak — Incident and Remediation (2026-11-03)

Status: Contained. Keys rotated, model allowlist and daily caps applied. Old leaked keys are disabled but not yet deleted — deletion waits on deployment of the new secrets.

Workspace: OpenRouter Default Workspace (d0cf37df-13fd-5d36-8a74-92ed6fa210c7)

Summary

Three OpenRouter API keys were found being used by third-party LLM reseller gateways instead of HungryHub services. The abuse was visible from 2026-10-27 and peaked on 2026-11-02.

• $666.33 of reseller-driven spend in the 31 days to 2026-11-03, across 94,101 requests.
• $142.04 of that was openai/gpt-6-astra-20260903 (GPT-6 Astra).
• Three of seven keys were compromised: hh-server prod, litellm, pma.

The controlling weakness was not IP access. At the time of the incident there was no model allowlist, no guardrail, no key expiry, and spending limits reset monthly at 20–100× real usage — so $602.54 of spend on 2026-11-02 did not trip any limit.

Impact

Direct monetary loss was bounded at roughly $0.67K. The material risk was unbounded going forward: the key material stayed in the attacker’s possession, so a reinstated key could have been resold immediately.

How the abuse was detected

OpenRouter’s analytics origin dimension (the HTTP Referer reaching OpenRouter) was the decisive signal. Legitimate HungryHub traffic shows origin in https://litellm.ai/, https://hungryhub.com/, or empty. The abuse showed:

newapi.ai (“New API”) and git.break.wtf/xenside/oxcode are third-party inference gateways that pool upstream provider keys and resell them to end users. Both attributed the traffic to country US, while our own traffic resolves to Singapore, Germany and elsewhere.

Note that the user dimension attributes requests to the credential owner, not the caller — it pointed at our own engineers and was a red herring. The origin dimension was the signal.

Timeline (UTC)

The Astra exposure window is narrow: total Astra spend across the whole 2026-08-20 → 2026-11-03 history is $142.04 (27,143 requests), all of it inside the four days above.

The Eagle Eye keys were disabled alongside the genuinely compromised keys but were not part of the abuse. Their traffic profile (Singapore origin, no Referer, model mix limited to a cheap open-weight model plus a search model) is consistent with Eagle Eye service traffic.

Affected keys

Full hashes (needed for key rotation and guardrail assignment):

Root cause

1. No model allowlist. Any key could call any model, including the expensive ones the reseller targeted.
2. No guardrail of any kind. GET /api/v1/guardrails returned total_count: 0; the workspace default guardrail was not configured and there were no per-key assignments.
3. Limits reset monthly and were oversized. $500/$300/$200 against real usage of ~$5–25/day meant a single bad day could not trip a limit.
4. No key expiry. expires_at was absent on every key, so a leaked key stayed valid forever.
5. Shared catch-all keys. hh-server prod and litellm each served many model families; one leak exposed everything.

Remediation applied (2026-11-03)

Model allowlist

The workspace default guardrail af250252-64ec-5449-b575-bc6de8d9a4bc was materialized. It applies to all workspace traffic, including keys not visible in the key list.

The four ignored models are 100% reseller traffic — excluding them removes the reseller’s entire model menu. The allowed list covers only models HungryHub services actually call.

anthropic/claude-4.5-haiku-20251001        anthropic/claude-4.5-opus-20251124
anthropic/claude-4.5-sonnet-20250929       anthropic/claude-4.6-opus-20260205
anthropic/claude-4.6-sonnet-20260217       anthropic/claude-4.7-opus-20260416
anthropic/claude-4.8-opus-20260528         anthropic/claude-5-fable-20260609
anthropic/claude-fable-5.1-20260831        anthropic/claude-haiku-5.5-20261007
anthropic/claude-opus-5-20260723           anthropic/claude-opus-5.5-20260921
anthropic/claude-sonnet-5-20260630         anthropic/claude-sonnet-5.5-20260928
cohere/north-mini-code-20260617:free       deepseek/deepseek-v4-flash-20260731
deepseek/deepseek-v4.1-flash-20261017      dots-studio/dots-3-note-preview-20260813:free
google/gemini-2.5-flash                   google/gemini-2.5-flash-image
google/gemini-3.1-flash-lite-20260507      google/gemini-3.8-flash-20260902
google/gemma-4-26b-a4b-it-20260403         google/gemma-4-26b-a4b-it-20260403:free
nex-agi/nex-n2.5-mini-20260908             nex-agi/nex-n2.5-pro-20260907
openai/gpt-4o                              openai/gpt-5.4-20260305
openai/gpt-5.4-mini-20260317               openai/gpt-5.4-pro-20260305
openai/gpt-5.5-20260423                    openai/gpt-5.5-pro-20260423
openai/gpt-5.6-luna-20260709               openai/gpt-5.6-sol-20260709
openai/gpt-5.6-terra-20260709              openai/gpt-6-luna-20260922
openai/gpt-6-sol-20260922                  openai/gpt-6.1-terra-20261026
openai/gpt-6.1-terra-pro-20261026          perplexity/sonar-pro-search
qwen/qwen3.5-122b-a10b-20260224            qwen/qwen3.5-397b-a17b-20260216
qwen/qwen3.8-27b-20260814                  recursive/flint-v1.0-20261021:free

Key rotation and daily caps

All seven active keys now carry a daily limit sized against real usage (roughly 20–60× headroom over observed legitimate spend):

hh-server prod and pma were rotated (new key created, old one disabled). The Eagle Eye keys were not compromised, so they were re-enabled with a daily cap rather than rotated.

expires_at is only settable at key-creation time — PATCH /api/v1/keys/{hash} has no such field. Adding an expiry therefore requires rotating, which is why the two new keys were created rather than patched.

Where the secrets live

New key secrets were written to a chmod 600 file outside the repository — never in chat, never committed. Move each into the secret store, then delete the file.

• hh-server — OPENROUTER_KEY, read at hh-server/app/services/openrouter_client.rb. In Terraform this is var.openrouter_key → hungryhub-terraform/hungryhub-apps/configmap.tf. Prod values are set out-of-band, not in prod.tfvars — update the secret, don’t edit tfvars.
• hh-menu — OPENROUTER_API_KEY, read at hh-menu/app/services/openrouter_client.rb. In Terraform this is var.openrouter_api_key_menu → configmap.tf.

Guardrail matching semantics (read before editing the allowlist)

GET /api/v1/models exposes two identifiers per model:

• id — the short/floating slug. Services send these (anthropic/claude-sonnet-4.5, deepseek/deepseek-v4-flash-0731). Some services send a legacy alias instead (anthropic/claude-4.5-sonnet).
• canonical_slug — the dated permaslug. This is what the guardrail matches on.

The API stores and matches the resolved canonical slug plus the endpoint variant, not the string the client sent. This is undocumented and caused two rounds of false blocking of our own traffic:

Four rules that follow:

1. Provider-flag suffixes are ignored. :beta, :online, :batch are flags on the same endpoint set, so one plain canonical entry covers them.
2. :free is a distinct endpoint set. A plain canonical slug does not match a :free request. Free-only models must be allowlisted with their :free id; a model with both paid and free endpoints needs both entries.
3. Aliases float to dated permaslugs. deepseek/deepseek-v4-flash and deepseek/deepseek-v4-flash-0731 resolved to two different permaslugs (…-20260423 and …-20260731) — they are different models. The catalog today only carries the -0731 alias.
4. The API rewrites a submitted canonical slug. Sending anthropic/claude-haiku-5.5-20261007 stored anthropic/claude-haiku-5.5:batch — and there is no :batch entry in the catalog. It appends a variant suffix on its own.

Always read the stored list back after a patch rather than trusting what was sent.

Verification

Enforcement was verified functionally, not by reading the config back:

• 17/17 first-party model strings return HTTP 200 through the guardrail.
• 7/7 blocked models return HTTP 404 with Model blocked by guardrail (4 reseller targets + 3 deliberately dropped models).
• google/gemma-4-26b-a4b-it:free and recursive/flint-v1.0:free return provider 503 — that is upstream free-tier capacity, not a guardrail rejection. The guardrail passed them.
• hh-menu’s default model anthropic/claude-4.5-sonnet was probed live and returns HTTP 200 (OpenRouter resolves it to anthropic/claude-sonnet-4.5).

One gotcha: GET /api/v1/models/user still lists Astra and gpt-6.1-sol even with the guardrail applied. That endpoint reflects the account’s model list, not the guardrail. The 404 responses are the proof of enforcement.

Known consequences of the allowlist

• z-ai/glm-5.3 is intentionally excluded → pma’s glm-5.3 probe now 403s.
• mistralai/mistral-large-4-0 and minimax/minimax-m3 are intentionally excluded → the LiteLLM probe key’s calls to them 403. The deployed LiteLLM config still routes openrouter/minimax-m3 and openrouter/moonshotai/kimi-k2.7-code through OPENROUTER_API_KEY; moonshotai/kimi-k2.7-code is not in the allowlist and will 403 for those routes. LiteLLM should call minimax.io directly — see LiteLLM MiniMax Key Management.
• The three :free models plus enable_free_model_publication: true mean free endpoints may publish prompts to providers.
• Guardrail budgets are per key / per member, not a workspace total. The $600/month is a per-member ceiling; the workspace ceiling is the sum of the daily caps.
• google/gemini-2.0-flash-001 (hh-server admin_setting default) is retired on OpenRouter — “no provider serves this model anymore”. This breakage predates the guardrail and is unrelated to it.

Remaining actions

Open questions

• Whether pma’s oxcode traffic was the same actor as the newapi.ai traffic or an unrelated colleague using that relay.
• Whether other OpenRouter accounts or providers (LiteLLM, Anthropic, OpenAI direct, AWS) hold credentials from the same leak.
• Whether PROD - Eagle Eye’s high-volume qwen3.5-397b pattern (67,201 requests, 382M tokens) is entirely legitimate Eagle Eye load.

Rotating an OpenRouter key (runbook)

1. Read the current inventory. GET /api/v1/keys?include_disabled=true&include_expired=true.
2. Create the replacement. POST /api/v1/keys accepts name, limit, limit_reset and expires_at. Set expires_at here — it cannot be added later.
3. Capture the secret once into a chmod 600 file outside the repo. Never print it to chat, never commit it.
4. Deploy the secret to the consuming service and confirm it appears in analytics traffic.
5. Disable the old key (PATCH /api/v1/keys/{hash} with disabled: true) while the new key is being verified.
6. Delete the old key (DELETE /api/v1/keys/{hash}) once the new key shows traffic.
7. Re-check the guardrail — if the service calls a model that is not in allowed_models, the call now fails with HTTP 404 Model blocked by guardrail. Add the model’s canonical slug (see the matching-semantics section above) before deploying.

Related

• LiteLLM + Langfuse + Grafana — LLM Gateway Stack
• LiteLLM MiniMax Key Management
• Eagle Eye — Architecture and Operations
• Credential Access Monitoring, Logs & APM
• Incident — hungryhub + utility Namespace Deletion 2026-06-08 — same class of problem: secrets living only in Kubernetes
• hungryhub-team/hungryhub-terraform — var.openrouter_key, var.openrouter_api_key_menu