DevLake — DORA Metrics Setup and Debugging

This document covers how DORA metrics data flows through DevLake, common failure modes, and how we’ve configured the system. It complements the Architecture & Operations and Runbook docs.

Current blueprint structure (as of 2026-06)

Four NORMAL-mode blueprints run nightly at 00:00 UTC, one per squad:

All four share scope config hungryhub-dora (id: 1) on GitHub connection 1.

Trigger a single project’s nightly sync manually:

# Replace 10 with 11, 12, or 13 for other projects
curl -s -X POST "https://devlake.hungryhub.com/api/blueprints/10/trigger" \
  -H "Content-Type: application/json" -d '{}'

Scope config (hungryhub-dora, id: 1)

The scope config controls how pipelines are classified as deployments and what counts as production:

Why cloudflare_zone_env is in productionPattern:
hungryhub-terraform GitHub Actions workflows use CLOUDFLARE_ZONE_ENV as the deployment environment name (not production). Without this, Terraform production deploys wouldn’t be counted. This was added in June 2026.

To update:

UPDATE _tool_github_scope_configs
SET production_pattern = '(?i)(prod|main|master|cloudflare_zone_env)'
WHERE id = 1 AND name = 'hungryhub-dora';

How DORA metrics data flows

GitHub API (workflow runs / deployments)
  ↓ github_graphql plugin
cicd_pipelines + cicd_pipeline_commits
  ↓ dora plugin: generateDeploymentCommits
cicd_deployment_commits (environment = 'PRODUCTION' for matching rows)
  ↓ dora plugin: enrichPrevSuccessDeploymentCommits
cicd_deployment_commits.prev_success_deployment_commit_id populated
  ↓ dora plugin: calculateChangeLeadTime (needs commits_diffs, often empty)
  OR manual SQL insert using merge_commit_sha matching
project_pr_metrics (pr_cycle_time in MINUTES)
  ↓ Grafana dashboards
DORA stats: Deployment Frequency, Lead Time, Change Failure Rate, FDRT

project_pr_metrics — manual population

When gitextractor fails (often due to SSH URL issues), commits_diffs stays empty and calculateChangeLeadTime produces nothing. The workaround: populate project_pr_metrics directly using merge_commit_sha matching.

Key facts:

• pr_cycle_time is stored in MINUTES (not seconds). Panel SQL uses < 24 * 60 (= 1 day) for the elite threshold.
• Link: pull_requests.merge_commit_sha ↔ cicd_deployment_commits.commit_sha

-- Check current state
SELECT project_name, COUNT(*) as cnt,
    MIN(pr_deployed_date) as earliest, MAX(pr_deployed_date) as latest
FROM project_pr_metrics
GROUP BY project_name;

-- Populate for a project (replace 'github:GithubRepo:1:XXXXXX' with actual scope IDs)
INSERT INTO project_pr_metrics (
    id, created_at, updated_at, _raw_data_params, _raw_data_table,
    project_name, first_commit_sha, pr_coding_time, pr_pickup_time, pr_review_time,
    deployment_commit_id, pr_deploy_time, pr_cycle_time,
    first_commit_authored_date, pr_created_date, pr_merged_date, pr_deployed_date
)
SELECT
    pr.id, NOW(), NOW(), '', '',
    pm.project_name,
    prc.commit_sha,
    GREATEST(0, TIMESTAMPDIFF(SECOND, prc.commit_authored_date, pr.merged_date)) / 60,
    0,
    GREATEST(0, TIMESTAMPDIFF(SECOND, pr.created_date, pr.merged_date)) / 60,
    dc.id,
    GREATEST(0, TIMESTAMPDIFF(SECOND, pr.merged_date, dc.finished_date)) / 60,
    GREATEST(0, TIMESTAMPDIFF(SECOND, prc.commit_authored_date, dc.finished_date)) / 60,
    prc.commit_authored_date, pr.created_date, pr.merged_date, dc.finished_date
FROM pull_requests pr
JOIN (
    SELECT pull_request_id, commit_sha, commit_authored_date,
           ROW_NUMBER() OVER (PARTITION BY pull_request_id ORDER BY commit_authored_date ASC) AS rn
    FROM pull_request_commits WHERE commit_authored_date IS NOT NULL
) prc ON prc.pull_request_id = pr.id AND prc.rn = 1
JOIN project_mapping pm ON pr.base_repo_id = pm.row_id AND pm.`table` = 'repos'
JOIN (
    SELECT commit_sha, id, finished_date,
           ROW_NUMBER() OVER (PARTITION BY commit_sha ORDER BY finished_date ASC) AS rn
    FROM cicd_deployment_commits
    WHERE result = 'SUCCESS' AND environment = 'PRODUCTION'
      AND prev_success_deployment_commit_id IS NOT NULL
      AND prev_success_deployment_commit_id != ''
) dc ON dc.commit_sha = pr.merge_commit_sha AND dc.finished_date >= pr.merged_date AND dc.rn = 1
WHERE pm.project_name = 'growth'   -- change per project
  AND pr.merged_date IS NOT NULL
  AND pr.merge_commit_sha IS NOT NULL AND pr.merge_commit_sha != ''
  AND prc.commit_authored_date < pr.merged_date
ON DUPLICATE KEY UPDATE updated_at = NOW(),
    pr_cycle_time = VALUES(pr_cycle_time),
    pr_deploy_time = VALUES(pr_deploy_time),
    pr_deployed_date = VALUES(pr_deployed_date),
    deployment_commit_id = VALUES(deployment_commit_id);

Enriching prev_success_deployment_commit_id

The DORA plugin subtask enrichPrevSuccessDeploymentCommits links each deployment to the previous successful one. If it produces 0 rows (common when running standalone), enrich manually:

UPDATE cicd_deployment_commits dc1
JOIN (
    SELECT id, LAG(id) OVER (
        PARTITION BY cicd_scope_id, environment
        ORDER BY finished_date
    ) as prev_id
    FROM cicd_deployment_commits
    WHERE result = 'SUCCESS' AND environment = 'PRODUCTION'
) ranked ON ranked.id = dc1.id
SET dc1.prev_success_deployment_commit_id = ranked.prev_id
WHERE ranked.prev_id IS NOT NULL;

This is idempotent — safe to re-run after adding new PRODUCTION rows.

PRODUCTION environment mapping

Only deployments with environment = 'PRODUCTION' count toward DORA metrics. Environments observed in our data and how they map:

To force a historical environment rename:

-- Only run after verifying the scope produces real production deploys
UPDATE cicd_deployment_commits
SET environment = 'PRODUCTION'
WHERE environment = 'CLOUDFLARE_ZONE_ENV' AND result = 'SUCCESS';

Grafana ANSI_QUOTES fix

The Aiven MySQL datasource is configured with ANSI_QUOTES mode (SQL standard). This means double quotes are identifier delimiters, not string literals. All string literals in Grafana panel SQL must use single quotes.

Symptom: Panels show “No data” or execute IS NULL checks that always fail.
Cause: SQL like = "PRODUCTION" silently fails (MySQL reads it as a column reference).
Fix: Replace = "value" with = 'value' in all panel SQL.

All 11 DevLake dashboards were audited and fixed in June 2026:

• DORA, DORA (by Team), DORA Validation — double-quoted string literals in CASE blocks
• Engineering Overview, GitHub — = "DONE" comparisons

To re-audit if new dashboards are imported:

import requests, re

GRAFANA_HOST = "grafana.hungryhub.com"
r = requests.get(f"https://{GRAFANA_HOST}/api/dashboards/uid/<UID>",
                 auth=("avnadmin", "<PASSWORD>"))
for panel in r.json()['dashboard']['panels']:
    for target in panel.get('targets', []):
        sql = target.get('rawSql', '')
        if re.search(r'(?:=|IN\s*\()\s*"[^"@{$][^"]*"', sql):
            print(f"Panel {panel['id']}: ANSI_QUOTES issue found")

Known gaps and limitations

Lead Time only covers April 2026 onward

The project_pr_metrics table was populated in June 2026 using merge_commit_sha matching. Data coverage: Apr-Jun 2026 for growth/automation, Jan/May-Jun 2026 for platform. Pre-April months show 0 in the Lead Time chart because no PRODUCTION deployments existed with matching commit SHAs.

Once gitextractor is fixed (see below), calculateChangeLeadTime can backfill via commit range matching.

commits_diffs table is empty

gitextractor fails with “Invalid Git URL” for SSH-format URLs (e.g. git@github.com:...). Without commits_diffs, calculateChangeLeadTime produces 0 rows. The project_pr_metrics manual population above is the current workaround.

Fix needed: Set USE_GO_GIT_IN_GIT_EXTRACTOR=true in /opt/devlake/.env and/or use HTTPS clone URLs.

Change Failure Rate and FDRT are N/A

No incident tracking integration (PagerDuty, OpsGenie, etc.) is configured. These metrics require incident data linked to deployments.

DORA (by Team) dashboard shows No data

This dashboard uses DevLake’s team membership tables (team_members, teams). Team configuration hasn’t been done — it requires mapping GitHub users to teams in DevLake’s UI. The project-based dashboards are the primary view.

Deployment Frequency troubleshooting

The Deployment Frequency stat uses a median calculation over the 6-month window. If fewer than half the months have deployments, the median is 0 and the stat shows “0 deployment days per month (low)” even if some months have data.

Symptom: Bar chart shows deployments but stat says “0”.
Root cause: Only 3 of 7 months (April-June) had PRODUCTION data.
Fix applied: Included CLOUDFLARE_ZONE_ENV (hungryhub-terraform) as PRODUCTION, adding Oct 2025 - Mar 2026 coverage. Now 6 of 7 months have data → median is non-zero.

Verification SQL:

SELECT DATE_FORMAT(finished_date, '%Y-%m') as month,
    COUNT(DISTINCT DATE(finished_date)) as deployment_days
FROM cicd_deployment_commits
WHERE result = 'SUCCESS' AND environment = 'PRODUCTION'
GROUP BY month ORDER BY month;

Related

• DevLake Architecture and Operations
• DevLake Runbook
• DevLake GitHub Apps Rate Limit
• Dashboards: grafana.hungryhub.com