DevLake — GitHub Apps Rate-Limit Distribution

Status: Live as of 2026-06-10.

Five org-owned GitHub Apps replace the single AppKey connection that was hitting the 5,000 req/hr GitHub API ceiling. Each App has its own installation quota; a refresh cron mints tokens every 30 min and loads them as a comma-separated round-robin in the DevLake GitHub connection.

Result: 5 × 5,000 = 25,000 req/hr effective rate limit.

Why this was needed

The nightly pipeline for 15 repos exhausted the 5,000 req/hr limit during catch-up backfill runs. The single GitHub App (AppKey auth, app_id 4006819) introduced after the 2026-06-04 incident solved the immediate regression but left no headroom.

Architecture

VPS cron (*/30 * * * *)
  └─ refresh-tokens.sh
       ├─ fetch hh-devlake-{1..5} private keys from SSM
       ├─ mint one GitHub App installation token per app (ghs_...)
       └─ PATCH DevLake connection 1
            authMethod: AccessToken
            token: "ghs_aaa,ghs_bbb,ghs_ccc,ghs_ddd,ghs_eee"
            rateLimitPerHour: 20000

DevLake round-robins across the 5 tokens. Each token carries its own 5,000 req/hr quota against the hungryhub-team org. Tokens expire after 1 hour; the 30-min cron keeps at least 30 min of life remaining at all times.

App registry

All 5 Apps are installed on All repositories in the hungryhub-team org with read-only permissions (contents, issues, pull_requests, actions, deployments, metadata, members).

SSM parameters (prod account, ap-southeast-1)

One SecureString per App, JSON-encoded:

/devlake/github-apps/hh-devlake-{1..5}
{
  "app_id": "...",
  "installation_id": "...",
  "private_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
}

Cron on the VPS

# /etc/cron.d/devlake-token-refresh
AWS_PROFILE=default
*/30 * * * * root /opt/devlake/ops/github-apps/refresh-tokens.sh >> /var/log/devlake-token-refresh.log 2>&1

Scripts live at /opt/devlake/ops/github-apps/ and in the repo at hungryhub-devlake/ops/github-apps/.

Day-2 operations

Check the refresh log

ssh root@62.238.41.27 "tail -50 /var/log/devlake-token-refresh.log"

A healthy run looks like:

[2026-06-10 10:09:43] refresh-tokens: Starting token refresh for 5 apps
[2026-06-10 10:09:43] refresh-tokens: Token minted for hh-devlake-1 ...
...
[2026-06-10 10:09:43] refresh-tokens: DevLake connection patched (HTTP 200). 5 tokens active, rate limit: 20000 req/hr.

Emergency manual refresh

ssh root@62.238.41.27 "/opt/devlake/ops/github-apps/refresh-tokens.sh"

Verify live token count in DevLake

ssh root@62.238.41.27 "curl -s http://localhost:8080/plugins/github/connections/1 | python3 -c \"
import json,sys; d=json.load(sys.stdin)
print('authMethod:', d.get('authMethod'))
print('tokens:', len(d.get('token','').split(',')))
print('rateLimitPerHour:', d.get('rateLimitPerHour'))
\""

Rotate a single App’s private key

1. Go to https://github.com/organizations/hungryhub-team/settings/apps → select the app → Private keys → Generate a private key → download the PEM.
2. From a laptop with prod SSO:

cd hungryhub-devlake/ops/github-apps
cp ~/Downloads/<app-name>.private-key.pem .
AWS_PROFILE=prod ./store-app-secret.sh <app-name>
rm <app-name>.private-key.pem

3. The next cron run picks up the new key automatically. Trigger a manual refresh to confirm immediately.

Rollback to single GitHub App

If something goes wrong with the multi-token setup:

ssh root@62.238.41.27

# Switch connection back to AppKey auth
curl -s -X PATCH http://localhost:8080/plugins/github/connections/1 \
  -H "Content-Type: application/json" \
  -d '{
    "name": "hungryhub-team",
    "endpoint": "https://api.github.com/",
    "authMethod": "AppKey",
    "appId": "4006819",
    "installationId": 139078097,
    "secretKey": "<private key from SSM /devlake/github-app-secret>",
    "enableGraphql": true,
    "rateLimitPerHour": 4500
  }'

# Remove the cron
/opt/devlake/ops/github-apps/refresh-tokens.sh --remove-cron

Provisioning scripts

Source: hungryhub-devlake/ops/github-apps/ (PR #18)

Follows the same pattern as hungryhub-terraform/github-runners/github-app-provisioning/.

Related

• Architecture: DevLake_Architecture_and_Operations.md
• Incident that triggered this: DevLake_Incident_2026-06-04_GraphQL_Regression.md
• Runbook: DevLake_Runbook.md