DevLake — Runbook

Day-2 operations, API reference, and known gotchas for the HungryHub DevLake deployment.

URL: https://devlake.hungryhub.com
VPS: root@62.238.41.27
Compose dir on VPS: /opt/devlake/

API authentication

All DevLake API calls require a Bearer token. The token is stored in .env as DEVLAKE_API_TOKEN.

# Local (from this repo .env)
source .env
AUTH="-H \"Authorization: Bearer $DEVLAKE_API_TOKEN\""

# From VPS (direct localhost, no auth needed for internal calls)
ssh root@62.238.41.27
curl http://localhost:8080/health

Important: The config-ui at :4000 proxies /api/* → devlake :8080 (strips the /api prefix). So /api/blueprints on port 4000 = /blueprints on port 8080. Public URL: https://devlake.hungryhub.com/api/blueprints

Cloudflare blocks Python’s urllib user-agent (HTTP 403, error 1010). Always make API calls via:

curl from your local machine, or
ssh root@62.238.41.27 curl http://localhost:8080/... for anything Cloudflare blocks

Check system health

# Container status
ssh root@62.238.41.27 "cd /opt/devlake && docker compose ps"

# DevLake API health
curl -s https://devlake.hungryhub.com/api/health

# Pipeline status (last 5)
curl -s "https://devlake.hungryhub.com/api/pipelines?pageSize=5" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" | \
  python3 -c "
import sys,json; d=json.load(sys.stdin)
for p in d.get('pipelines',[]):
    print(p['id'], p['name'], p['status'], p['finishedTasks'],'/',p['totalTasks'])
"

Trigger a manual sync

# Trigger blueprint 4 (HungryHub Nightly)
curl -s -X POST "https://devlake.hungryhub.com/api/blueprints/4/trigger" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

For a full resync from the beginning (ignoring timeAfter):

curl -s -X POST "https://devlake.hungryhub.com/api/blueprints/4/trigger" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"fullSync": true}'

Check a running pipeline

# Replace 4 with actual pipeline id
PIPELINE_ID=4
curl -s "https://devlake.hungryhub.com/api/pipelines/$PIPELINE_ID" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" | \
  python3 -c "
import sys,json; d=json.load(sys.stdin)
print('status:', d['status'])
print('tasks:', d['finishedTasks'], '/', d['totalTasks'])
print('stage:', d.get('stage'))
print('started:', d.get('beganAt'))
print('finished:', d.get('finishedAt'))
"

Check failed tasks:

curl -s "https://devlake.hungryhub.com/api/pipelines/$PIPELINE_ID/tasks" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" | \
  python3 -c "
import sys,json
d=json.load(sys.stdin)
tasks = d.get('tasks', [])
for t in tasks:
    if t.get('status') == 'TASK_FAILED':
        msg = t.get('message','')
        wraps = [l.strip() for l in msg.split('\n') if 'Wraps' in l]
        print('FAILED:', t.get('plugin'), '|', wraps[-1][:120] if wraps else msg[:120])
"

Restart containers

ssh root@62.238.41.27 "cd /opt/devlake && docker compose restart"

Restart a single container:

ssh root@62.238.41.27 "cd /opt/devlake && docker compose restart devlake"

View logs

# All containers
ssh root@62.238.41.27 "cd /opt/devlake && docker compose logs -f --tail=100"

# DevLake engine only
ssh root@62.238.41.27 "cd /opt/devlake && docker compose logs -f devlake --tail=100"

Add a new repo to track

Look up the GitHub repo ID:

curl -s "https://api.github.com/repos/hungryhub-team/<REPO_NAME>" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['id'])"

Add the scope (from VPS to avoid Cloudflare WAF):

ssh root@62.238.41.27 python3 - <<'EOF'
import json, urllib.request

TOKEN = open("/opt/devlake/.env").read()
# Parse token from .env
for line in TOKEN.splitlines():
    if line.startswith("DEVLAKE_API_TOKEN="):
        TOKEN = line.split("=",1)[1].strip()
        break

GITHUB_ID = 123456789  # replace with actual
FULL_NAME = "hungryhub-team/new-repo"  # replace

body = {"data": [{
    "connectionId": 1,
    "githubId": GITHUB_ID,
    "fullName": FULL_NAME,
    "name": FULL_NAME.split("/")[1],
    "HTMLUrl": f"https://github.com/{FULL_NAME}",
    "cloneUrl": f"https://github.com/{FULL_NAME}.git",
    "scopeConfigId": 1
}]}

req = urllib.request.Request(
    "http://localhost:8080/plugins/github/connections/1/scopes",
    data=json.dumps(body).encode(),
    headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
    method="PUT"
)
resp = urllib.request.urlopen(req)
print("Done:", resp.read().decode()[:200])
EOF

PATCH blueprint 4 to add the new scopeId:

# Get current blueprint connections/scopes, add new scopeId, PATCH
curl -s "https://devlake.hungryhub.com/api/blueprints/4" \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps(d['connections'][0]['scopes'], indent=2))"

Then PATCH with the updated scopes list (from VPS):

ssh root@62.238.41.27 curl -s -X PATCH http://localhost:8080/blueprints/4 \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"connections": [{"pluginName": "github", "connectionId": 1, "scopes": [...existing..., {"scopeId": "NEW_ID"}]}]}'

Known gotchas

1. Blueprint `connections` silently ignored on POST

POST /blueprints with a connections array creates the blueprint but stores connections: []. Always create the blueprint first, then PATCH /blueprints/{id} to set connections.

This is a known bug in DevLake v1.0.3-beta12: GeneratePlanJsonV200 calls GetPlugin("") when the planner reads connections during the POST flow.

Workaround: Create via POST /projects (embeds a blueprint), then PATCH the resulting blueprint.

2. Project creation via API does not attach blueprint connections

POST /projects with an embedded blueprint.connections creates the project and blueprint but the blueprint is stored with connections: []. Always PATCH the blueprint after.

3. DB URL must be written via Python, not sed

The Aiven MySQL DSN contains & characters. Writing it with sed causes shell expansion, silently corrupting the value (truncates at the first &).

Always use Python:

import re
content = open("/opt/devlake/.env").read()
content = re.sub(r'DB_URL=.*', f'DB_URL={new_url}', content)
open("/opt/devlake/.env", "w").write(content)

4. sql_require_primary_key must be OFF on Aiven MySQL

DevLake migrations create tables without primary keys. This setting must be disabled or migrations fail with ERROR 3750.

Check/fix:

avn service user-config-change devlake-mysql \
  --project hh-development \
  '{"mysql": {"sql_require_primary_key": false}}'

5. FORCE_MIGRATION=true wipes blueprint/project tables

Setting FORCE_MIGRATION=true in .env and restarting drops and recreates the schema. Plugin data (connections, scopes, scope configs) survives in separate tables, but blueprints and projects are wiped. Remove FORCE_MIGRATION=true after the first run.

Recovery: re-create the project + PATCH blueprint (connections survive in plugin tables).

6. Grafana dashboard JSON must be uploaded via file, not shell echo

The dashboard JSON files are >50 KB with special characters. Passing them via shell variable or echo corrupts the JSON. Always use curl -d @file.json.

7. Scope PUT requires `{"data": [...]}` wrapper

PUT /plugins/github/connections/{id}/scopes requires the body to be {"data": [<GithubRepo>, ...]}. Sending a bare array or a single object both return errors.

8. DORA plugin subtask names are camelCase, not PascalCase

The Meta struct’s Name field in the Go source is camelCase, e.g.:

var DeploymentGeneratorMeta = plugin.SubTaskMeta{
    Name: "generateDeployments",  // NOT "DeploymentGenerator"
    ...
}

The wrong-name error is opaque: subtask DeploymentGenerator does not exist.

Correct dora subtask names (DevLake v1.0.3-beta12):

generateDeploymentCommits
enrichPrevSuccessDeploymentCommits
generateDeployments
ConvertIssuesToIncidents (PascalCase — exception)
ConnectIncidentToDeployment (PascalCase — exception)
calculateChangeLeadTime

9. NORMAL-mode blueprint does not auto-include the DORA stage

In NORMAL mode, the blueprint plan only contains data-collection stages (github, gitextractor, etc.). The DORA plugin’s transformation stage is NOT auto-added, even when cicd_deployments data exists. This is why DORA panels showed “No data” despite 4,313 deployments being collected.

Fix: Switch the blueprint to ADVANCED mode and add an explicit dora stage:

dora_stage = [{
    "plugin": "dora",
    "subtasks": [
        "generateDeploymentCommits", "enrichPrevSuccessDeploymentCommits",
        "generateDeployments", "ConvertIssuesToIncidents",
        "ConnectIncidentToDeployment", "calculateChangeLeadTime"
    ],
    "options": {}
}]

Note: mode is immutable on a blueprint — you cannot PATCH it. To change NORMAL→ADVANCED, you must delete the blueprint (and its project) and recreate with the full plan + connections.

10. DORA scope config needs issueTypeIncident regex to populate incidents

The dora subtask ConvertIssuesToIncidents only picks up issues where type = 'INCIDENT'. The type field is set by the github Extract Issues task based on the scope config’s issueTypeIncident regex matched against issue labels.

Without this regex set, StdType stays empty for all issues and incidents table stays at 0 rows, which makes “Time to Restore Service” panels empty even when deployments exist.

Required scope config for incident tracking:

{
  "issueTypeIncident": "(?i)(incident|outage|sev[0-9]|p0|p1)",
  "issueTypeBug": "(?i)(bug|defect|regression)"
}

After updating the scope config, re-run the pipeline to re-extract issues with the new regex (the extractor only sets StdType on first extraction — it doesn’t update on subsequent runs).

11. Project DELETE panics with nil-deref if blueprint is already deleted

DELETE /projects/{name} calls thereAreUnfinishedPipelinesUnderProject(name), which calls GetBlueprintByProjectName(name). If the blueprint was already deleted (e.g. to recreate it), this returns a blueprint with ID=0, and the subsequent blueprint.ID use panics with runtime error: invalid memory address or nil pointer dereference.

Workaround: Rename the project in MySQL to free the unique name, then create a new project:

UPDATE projects SET name = 'HungryHub Engineering (old)' WHERE name = 'HungryHub Engineering';

This is fixed in newer DevLake versions but the bug is present in v1.0.3-beta12.

12. Pipeline DELETE doesn’t free the cron slot

DELETE /pipelines/{id} marks the pipeline as cancelled in the in-memory queue, but the devlake engine still holds a lock on the blueprint’s cron slot. Trying to trigger a new pipeline on the same blueprint returns:

there are pending pipelines of current blueprint already (400)

Workaround: Force-update the pipeline to TASK_FAILED in MySQL:

UPDATE _devlake_pipelines SET status='TASK_FAILED', finished_at=NOW(3) WHERE id=25;
UPDATE _devlake_tasks SET status='TASK_FAILED', finished_at=NOW(3)
  WHERE pipeline_id=25 AND status IN ('TASK_RUNNING','TASK_CREATED','TASK_RERUN','TASK_RESUME');

Note: this only updates the DB. The in-memory engine state still has the old pipeline active. A full container restart (docker compose restart devlake) clears the in-memory state.

13. GitHub connection authMethod=AppKey with bad secret fails 401

If a connection is set to authMethod=AppKey but the secretKey is wrong (e.g. rotated PEM), all graphql calls fail with 401 Bad credentials after 2 retries each = 240s per request.

Check current auth state:

curl -sf http://localhost:8080/plugins/github/connections/1 \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" | python3 -m json.tool

Revert to PAT auth:

curl -X PATCH http://localhost:8080/plugins/github/connections/1 \
  -H "Authorization: Bearer $DEVLAKE_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"authMethod":"AccessToken","token":"ghp_xxx","appId":"","installationId":0,"secretKey":""}'

The dora_report variable in the DORA dashboard queries:

SELECT dora_report FROM dora_benchmarks

Adding a new DORA report year (e.g. 2024) requires inserting 4 rows (one per metric) into dora_benchmarks AND updating the title_value Grafana variable’s CASE statement to map the new year to the correct metric name (e.g. “Failed Deployment Recovery Time” for 2023+).

The default dora_benchmarks from DevLake migrations only includes 2021 and 2023.

INSERT INTO dora_benchmarks (metric, low, medium, high, elite, dora_report) VALUES
  ('Deployment frequency', 'Fewer than once per month(low)', 'Between once per week and once per month(medium)', 'Between once per day and once per week(high)', 'On-demand(elite)', '2024'),
  ('Lead time for changes', 'More than one month(low)', 'Between one week and one month(medium)', 'Between one day and one week(high)', 'Less than one day(elite)', '2024'),
  ('Change failure rate', '> 15%(low)', '10%-15%(medium)', '5%-10%(high)', '0-5%(elite)', '2024'),
  ('Failed deployment recovery time', 'More than one week(low)', 'Between one day and one week(medium)', 'Less than one day(high)', 'Less than one hour(elite)', '2024');

15. GitHub Apps for rate-limit distribution (5 apps × 5,000 req/hr = 25,000/hr)

A nightly 15-repo backfill can hit the 5,000 req/hr ceiling on a single token. The HungryHub team now uses 5 org-owned GitHub Apps for round-robin distribution. See DevLake_GitHub_Apps_Rate_Limit.md for the full setup (app IDs, SSM layout, cron-based token refresh, key rotation, rollback).

When the connection’s rateLimitPerHour is 20000 and token starts with ghs_ (GitHub App installation token, not ghp_ PAT), the GitHub Apps distribution is active.

Restore from Aiven backup (verified 2026-06-10)

We run a quarterly restore test by forking the live devlake-mysql to a new test service, comparing row counts and data integrity, then terminating the test service. The full procedure is documented in the hungryhub-devlake repo at deploy/restore-test.sh. Quick recipe below.

1. Quick start

TEST_NAME="devlake-mysql-restore-test-$(date +%Y%m%d)"

avn service create "$TEST_NAME" \
  --project hh-development \
  --service-type mysql \
  --plan startup-4 \
  --cloud google-europe-north1 \
  --service-to-fork-from devlake-mysql \
  -f

# Wait for the service to reach RUNNING state (~30-60 min for a 10 GB DB)
avn service wait "$TEST_NAME" --project hh-development

2. Cloud-specific hostname pattern

Source cloud	Forked-to cloud	Hostname pattern
`do-fra`	`do-fra`	`<name>-<project>.f.aivencloud.com`
`do-fra`	`google-europe-north1`	`<name>-<project>.l.aivencloud.com`
`google-europe-north1`	`google-europe-north1`	`<name>-<project>.l.aivencloud.com`
`aws-us-east-1`	`aws-us-east-1`	`<name>-<project>.a.aivencloud.com`

DNS may take 5–10 min after RUNNING to propagate.

3. Verify data integrity

RST_HOST="<forked service hostname>"
SRC_HOST="devlake-mysql-hh-development.f.aivencloud.com"  # from /devlake/db-url SSM

# Compare row counts on the key tables
for host in "$SRC_HOST" "$RST_HOST"; do
  docker run --rm --network host mysql:8 mysql -h "$host" -P 15939 \
    -u avnadmin -p"$AIVEN_DB_PASSWORD" --ssl-mode=REQUIRED lake \
    -e "SELECT 'projects' AS t, COUNT(*) AS cnt FROM projects
        UNION ALL SELECT 'repos', COUNT(*) FROM repos
        UNION ALL SELECT 'pull_requests', COUNT(*) FROM pull_requests
        UNION ALL SELECT 'cicd_deployment_commits', COUNT(*) FROM cicd_deployment_commits
        UNION ALL SELECT '_devlake_blueprints', COUNT(*) FROM _devlake_blueprints;"
done

In the 2026-06-10 test, all 11 key tables matched exactly between source and restored (projects=1, repos=15, pull_requests=5894, cicd_pipelines=137343, etc.). The GitHub App credentials (AppKey auth, app_id=4006819) round-tripped correctly.

4. Gotchas

Aiven fork takes 30–60 min for a 10 GB DB; service is in REBUILDING state during this time.
DNS propagation lag. After RUNNING, the new hostname may return NXDOMAIN for 5–10 min. Wait, then retry.
Different hostname per cloud region. See the table above.
The forked service has a different password from the source. Aiven rotates the password on the fork; fetch the new one from the Aiven console under the service’s “Connection information” page.
SSL mode matters. Use --ssl-mode=REQUIRED (encrypt, no cert verify) for routine queries. Use --ssl-mode=VERIFY_CA only with the Aiven CA cert at --ssl-ca=<path>.
The avn CLI --format json flag is unreliable in aiven-client 4.15.0 (returns the literal string "json" for some commands). Use --format yaml or parse the table output.

5. Tear down

avn service terminate "$TEST_NAME" --project hh-development -f

The startup-4 plan is ~$74/mo — don’t leave the test service running.

Full restore (catastrophic recovery)

If the live service is corrupted or deleted:

# Fork the live service to a new name
avn service create "$NEW_NAME" \
  --project hh-development --service-type mysql --plan startup-4 \
  --cloud google-europe-north1 --service-to-fork-from devlake-mysql -f
avn service wait "$NEW_NAME" --project hh-development

# Point the app at it
NEW_URI=$(avn service connection-info "$NEW_NAME" --project hh-development)
aws ssm put-parameter --name /devlake/db-url --type SecureString \
  --value "$NEW_URI" --overwrite --profile prod --region ap-southeast-1
./deploy/pull-secrets.sh
./deploy/runbook.sh restart

Re-create project and blueprint from scratch

If the blueprint and project are accidentally wiped (e.g. by FORCE_MIGRATION=true):

ssh root@62.238.41.27 python3 - <<'PYEOF'
import json, urllib.request

TOKEN = "<DEVLAKE_API_TOKEN>"
BASE = "http://localhost:8080"

repos = [
  199970072, 733067847, 1006944319, 1077453560, 494021211, 496500431,
  219383404, 1204792079, 510972874, 1158898105, 1156890442, 539788675,
  1239594970, 1028203938, 1006956323
]

def api(method, path, body=None):
    req = urllib.request.Request(
        f"{BASE}{path}",
        data=json.dumps(body).encode() if body is not None else None,
        headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
        method=method
    )
    try:
        return json.loads(urllib.request.urlopen(req).read())
    except urllib.error.HTTPError as e:
        return {"_err": e.code, "_body": e.read().decode()[:300]}

# 1. Create project
proj = api("POST", "/projects", {
    "name": "HungryHub Engineering",
    "description": "DORA metrics and PR cycle time for the HungryHub team",
    "enable": True,
    "blueprint": {
        "name": "HungryHub Nightly", "mode": "NORMAL", "enable": True,
        "cronConfig": "0 2 * * *", "isManual": False, "skipOnFail": True,
        "timeAfter": "2025-01-01T00:00:00Z", "connections": []
    }
})
print("Project:", proj.get("name", proj))

# 2. Find blueprint id
bps = api("GET", "/blueprints?pageSize=20&page=1")
bp_id = next(b["id"] for b in bps.get("blueprints", []) if b["name"] == "HungryHub Nightly")
print("Blueprint id:", bp_id)

# 3. PATCH with connections
scopes = [{"scopeId": str(r)} for r in repos]
patched = api("PATCH", f"/blueprints/{bp_id}", {
    "projectName": "HungryHub Engineering",
    "connections": [{"pluginName": "github", "connectionId": 1, "scopes": scopes}]
})
print("Connections after patch:", len(patched.get("connections", [])))

# 4. Trigger backfill
pipeline = api("POST", f"/blueprints/{bp_id}/trigger", {})
print("Pipeline:", pipeline.get("id"), pipeline.get("status"), pipeline.get("totalTasks"), "tasks")
PYEOF

Architecture: DevLake_Architecture_and_Operations.md
Epic: hungryhub-team/hungryhub-terraform#195
Issues: hungryhub-team/hungryhub-devlake

Keyboard shortcuts

HungryHub Knowledge Base