Managing MiniMax API Keys in LiteLLM

Overview

LiteLLM proxy load-balances requests across multiple MiniMax API keys using simple-shuffle routing. Each model group has N deployments (one per key), and requests are distributed evenly.

Current setup: 2 keys × 10 model groups = 20 deployments.

Subscription seats drive the key count. We buy N user seats on the MiniMax Plus plan and provision N API keys; the model pool scales linearly (2 keys → 2 replicas per model group → 2× aggregate throughput per model under the same per-key rate limit). See hungryhub-terraform#534 for the most recent 13 → 2 reduction.

Model groups (all on MiniMax Plus plan — same per-key rate limit):

Both keys are used in parallel per model group (order: 1 on every deployment) since they’re all the same Plus tier.

Source of Truth

The key set and per-key pool size are Terraform-managed, not runtime. The live Kubernetes secret litellm-env-secret is reconciled from:

AWS Secrets Manager:  /hungryhub/dev  (LITELLM_REDIS_PASSWORD, MINIMAX_API_KEY_*, ...)
        |
        v
helm-releases/litellm.tf           (kubernetes_secret_v1.litellm_env, K8s secret)
        |
        v
helm-releases/values/litellm.yaml.tft  (proxy_config.model_list, fed via helm release)
        |
        v
LiteLLM pods read both on startup; model_list lives in chart values, not in /config

Why this matters: the previous workflow (add to K8s secret + POST /config/update from a running pod) was effective in earlier LiteLLM versions, but in v1.88.1 the chart-loaded config takes precedence on pod restart and the runtime update is silently ignored (200 OK but no change to the router’s deployment list). The chart is now the single source of truth and must be updated for any change to take effect.

Adding a New Key

1. Append the new key to keys.txt (local, gitignored, operator reference). Line N becomes MINIMAX_API_KEY_N.

2. Add the key to AWS Secrets Manager (sandbox account for dev):

   aws --profile=sandbox secretsmanager get-secret-value \
     --secret-id /hungryhub/dev --region ap-southeast-1 \
     --query SecretString --output text | \
   python3 -c "
   import json, sys
   d = json.load(sys.stdin)
   d['MINIMAX_API_KEY_N'] = open('keys.txt').readlines()[N-1].strip()
   sys.stdout.write(json.dumps(d))" | \
   aws --profile=sandbox secretsmanager put-secret-value \
     --secret-id /hungryhub/dev --region ap-southeast-1 \
     --secret-string file:///dev/stdin
   

   Note: sys.stdout.write instead of print — print adds a trailing newline that AWS CLI persists in SecretString and breaks jsondecode on the next Terraform apply. See hungryhub-terraform/AGENTS.md §6.6 in the hh-infra workspace root.

3. Update helm-releases/litellm.tf to declare the new key in the kubernetes_secret_v1.litellm_env data block. Without this step, the next terraform apply would wipe the key from the live K8s secret.

4. Update helm-releases/values/litellm.yaml.tft to add one new deployment per model group under proxy_config.model_list (10 groups × 1 new entry each). Use the existing MINIMAX_API_KEY_{1..N-1} patterns in the file as a template; the rate limits per model group are listed in the overview table above.

5. Open a PR against hungryhub-team/hungryhub-terraform. The terraform-pr-plan.yml workflow runs terraform plan automatically and posts a comment. Review the plan diff (expect: 1 K8s secret update + 1 helm release update, 0 destroy).

6. Apply via the terraform.yml workflow (workflow_dispatch, env=dev, services=helm-releases, action=apply). The Helm upgrade rolls the LiteLLM pods; the new model_list is loaded on startup.

7. Verify by counting unique model_info.id values in the proxy log for any model — should match the new pool size (N distinct IDs once rollout settles).

Removing a Key

Reverse the add workflow:

1. Remove the key from AWS SM /hungryhub/dev (use the same put-secret-value pattern with the new JSON).
2. Remove the MINIMAX_API_KEY_N line from helm-releases/litellm.tf and from proxy_config.model_list in values/litellm.yaml.tft (one entry per model group, 10 groups total).
3. PR + apply as above.

Rotating a Key

In-place rotation (key N → new value, no model_list change):

1. Replace the value of MINIMAX_API_KEY_N in AWS SM.
2. Update line N in helm-releases/litellm.tf (the value comes from local._secrets["MINIMAX_API_KEY_N"] at apply time, so the .tf itself doesn’t need a literal value change, but the secret data will be reconciled).
3. PR + apply. Pods pick up the new env var via the secret volume mount within ~60s; the helm upgrade will roll pods explicitly anyway.

How Load Balancing Works

• Strategy: simple-shuffle — random distribution across all deployments sharing the same model_name.
• Failover: If a key returns 429 (rate limit) or 5xx, LiteLLM retries with another key (up to num_retries: 2).
• Cooldown: After 3 consecutive failures on a key, it’s cooled down for 30s. Cooldown state is shared across pods via the bundled Redis (subchart of litellm-helm, auth wired to LITELLM_REDIS_PASSWORD).
• Fallback chain (chat models only):
  • M3 → M2.1 → M2
  • M2.1-highspeed → M2.1
  • M2.5 → M2.1
  • M2.5-highspeed → M2.1-highspeed
  • M2.7 → M2.1
  • M2.7-highspeed → M2.1-highspeed

Key Source

MiniMax API keys are created at https://platform.minimax.io (developer console). Each key has its own per-key rate limit; more keys = higher aggregate throughput under simple-shuffle.

Where Keys Are Stored

For Terraform-managed deployments, update AWS SM and the chart values together in the same PR; re-run terraform apply on helm-releases.

History