DevLake — Architecture and Operations

DevLake provides DORA metrics and per-squad PR cycle time for the HungryHub team (~25 devs).

• Repo: hungryhub-team/hungryhub-devlake
• URL: https://devlake.hungryhub.com (Cloudflare Tunnel)
• Dashboards: grafana.hungryhub.com (Aiven Grafana — external)
• Version: Apache DevLake v1.0.3-beta12

Architecture

Internet
   │
   ▼
Cloudflare Tunnel (cloudflared)
   │
   └──► config-ui  (port 4000) — DevLake admin UI
              │  proxies /api/* → devlake :8080
              ▼
        devlake (port 8080) — pipeline engine + REST API
              │
              ▼
       Aiven MySQL (managed)
       hh-development project
       devlake-mysql
       google-europe-north1 (Finland)

Host: Hetzner CPX42 VPS (devlake-prod), Helsinki — 62.238.41.27

No local MySQL or Grafana containers — both are replaced by Aiven managed services.

Infrastructure details

Key components

Secrets (AWS SSM — prod account 202255947274, ap-southeast-1)

Pull secrets to the VPS before deploying:

./deploy/pull-secrets.sh

DB URL format: mysql://user:pass@host:port/db?charset=utf8mb4&parseTime=True&loc=UTC&tls=skip-verify Always write the DB URL via Python re.sub — never sed, as & in the value causes shell expansion corruption.

IAM

IAM user devlake-vps (key AKIAS6F3E2YFCI6SBIPK) in the prod account has a narrow inline policy:

• ssm:GetParameter + kms:Decrypt scoped to arn:aws:ssm:ap-southeast-1:202255947274:parameter/devlake/*

Credentials live at /root/.aws/credentials on the VPS.

GitHub connector configuration

Tokens are minted from 5 org-owned GitHub Apps by a cron on the VPS every 30 min. See DevLake GitHub Apps Rate-Limit Distribution for the full setup.

Scope config hungryhub-dora:

• deploymentPattern: (?i)(deploy|release|prod)
• productionPattern: (?i)(prod|main|master|cloudflare_zone_env) — includes Terraform CF deploys
• prBodyClosePattern: (?i)(fix|close|resolve)(\s*)(?:#)(\d+)

15 tracked repos (all with scopeConfigId: 1):

Projects and blueprints

Four NORMAL-mode blueprints run nightly at 00:00 UTC, one per squad:

All share scope config hungryhub-dora (id: 1, GitHub connection 1).

To add or move a repo: update the blueprint’s scope list via the DevLake API (PATCH /blueprints/<id>). The squad source of truth is hh-relay/squad_channels.yml.

See DevLake DORA Metrics Setup for detailed configuration, debugging, and known limitations.

Grafana dashboards

Grafana folder: DevLake DORA (uid: ffo08wszkd2wwf) Datasource: devlake (uid: dfo08jdx29z40f, MySQL, id: 15)

11 dashboards imported from official DevLake v1.0.3-beta12 JSONs:

• DORA, DORAByTeam, DORADebug
• DORADetails ×4
• EngineeringOverview, EngineeringThroughputAndCycleTime
• GitHub, Homepage

Common operations

Check container status

ssh root@62.238.41.27
cd /opt/devlake
docker compose ps
docker compose logs -f devlake

Deploy / update

# Initial deploy
./deploy/deploy.sh

# Update (pull new images, restart)
./deploy/update.sh

Trigger a manual pipeline run

Via the DevLake API (use DEVLAKE_API_TOKEN from .env):

# Trigger nightly sync for a project (replace 10-13 for other projects)
# 10=growth, 11=integrations, 12=automation, 13=platform
curl -X POST https://devlake.hungryhub.com/api/blueprints/10/trigger \
  -H "Authorization: Bearer <DEVLAKE_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'

SSH onto the VPS

ssh root@62.238.41.27

Update DevLake version

1. Edit docker-compose.yml — bump devlake and config-ui image tags.
2. Run ./deploy/update.sh.
3. Verify on https://devlake.hungryhub.com.

Database operations

Check disk usage

avn service get devlake-mysql --project hh-development --format json | \
  python3 -c "import json,sys; svc=json.load(sys.stdin); print(svc.get('disk_space_mb'), 'MB disk')"

Scale up if > 60% full:

avn service update devlake-mysql --plan startup-8 --project hh-development

Restore from Aiven backup

1. Go to Aiven Console → hh-development → devlake-mysql → Backups.
2. Fork to a new service using the desired backup point.
3. Export: mysqldump -h <new-host> -u avnadmin -p lake > lake_backup.sql
4. Import: mysql -h <live-host> -u avnadmin -p lake < lake_backup.sql
5. Restart: docker compose -f /opt/devlake/docker-compose.yml restart

ENCRYPTION_SECRET — do not rotate casually

All connector credentials stored in the DevLake DB are encrypted with this key. Rotating it without re-entering every connector secret will break all data pipelines.

Only rotate if the key is compromised:

1. openssl rand -base64 32 — generate new secret
2. aws ssm put-parameter --profile prod --region ap-southeast-1 --name /devlake/encryption-secret --type SecureString --value '<new>' --overwrite
3. ./deploy/pull-secrets.sh
4. Restart containers
5. Re-enter all connector API tokens in the config-ui

Infrastructure provisioning scripts

Related

• Epic: hungryhub-team/hungryhub-terraform#195
• Dashboards: grafana.hungryhub.com
• Runbook: DevLake_Runbook.md