Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DevLake — Incident 2026-06-04: github_graphql requestedIssues Regression (and follow-on rate limit + Grafana issues)

Status: Resolved for the original requestedIssues bug and the GitHub rate limit. Two upstream bugs remain in apache/incubator-devlake v1.0.4-alpha:

  1. release_convertor.go:76 Go panic (mitigated by publishing draft releases + cleaning the DB; upstream patch prepared, not yet merged)
  2. DORA dashboard SQL has a parse-time error from double-quoted string literals in CASE/ELSE (see DORA_Dashboard_SQL_Bug.md)

Full timeline: 2026-06-04 02:00 UTC → 2026-06-10 (multiple fixes deployed). The original incident is fixed; two follow-on issues remain as documented in the postmortem.

Severity: Medium — Grafana DORA panels were stale for the team’s 5 largest repos; no data loss, no security impact, no impact on customer-facing services.

Affected duration: 2026-06-04 02:00 UTC → 2026-06-08 ~10:30 UTC (~4.5 days, 4 nightly runs)

Severity: Medium — Grafana DORA panels were stale for the team’s 5 largest repos; no data loss, no security impact, no impact on customer-facing services.

Triggered by: GitHub removed the requestedIssues field from the Query type in the GraphQL API (announced 2025, removed in 2026). DevLake v1.0.3-beta12’s plugins/github_graphql/tasks/issue_collector.go:242 still referenced it.

Symptom

Nightly pipeline 4 (HungryHub Nightly, schedule 0 2 * * *) returned TASK_PARTIAL every night from 2026-06-04 onwards. All 5 of the 15 github_graphql tasks for the largest repos failed. The remaining 10 repos succeeded, which is why the pipeline was “partial” rather than “failed”.

User-visible impact: Aiven Grafana DORA dashboards (grafana.hungryhub.com, DevLake DORA folder) showed stale or blank values for:

  • Open issue age
  • Issue-to-PR linking
  • Per-squad open-issue counters
  • DORADetails panels

For these 5 repos:

  • hungryhub-team/hh-server (199970072)
  • hungryhub-team/hh-pegasus (733067847)
  • hungryhub-team/hh-pegasus-monorepo (1204792079)
  • hungryhub-team/hh-relay (1239594970)
  • hungryhub-team/hungryhub-terraform (1028203938)

Smaller repos were unaffected because their first page of issues fit in the initial 10-item page and never triggered the second InitGraphQLCollector block that contains the bug.

Detection gap

The pipeline has been quietly returning TASK_PARTIAL for 4 consecutive days (2026-06-04 → 2026-06-08) before anyone noticed. There is no CloudWatch alarm, Slack notifier, or health check on pipeline.status != TASK_COMPLETED. A user noticed because the Grafana dashboards started looking wrong; the first person to check the pipeline found the partial status.

Root cause

plugins/github_graphql/tasks/issue_collector.go in v1.0.3-beta12 has two InitGraphQLCollector blocks inside CollectIssues:

  1. First collector (line ~120-150): paginates repository(owner, name) { issues(first, after, orderBy) { ... } } to collect new issues since the previous run.
  2. Second collector (line ~190-240): reads the local DB cursor of OPEN issues and asks GitHub to refresh each one via repository(owner, name) { issue(number) { ... } }.

The second collector is the broken one. It builds a GraphqlQueryIssueDetailWrapper struct whose tag is:

Repository struct {
    Issues []GraphqlQueryIssue `graphql:"issue(number: $number)" graphql-extend:"true"`
} `graphql:"repository(owner: $owner, name: $name)"`

…and whose BuildQuery populates query.requestedIssues (a Go map of missingGithubIssueRef) and then sets the variable issue to a list of issue numbers. The merico-ai/graphql client can’t bind the unexported requestedIssues field to the GraphQL response, and there’s a singular/plural mismatch between the $number variable in the query and the list being passed in.

The fallback behaviour is for the client to emit a GraphQL query that asks for a top-level requestedIssues field, which GitHub returns:

Field 'requestedIssues' doesn't exist on type 'Query'
  graphql.DataError
  plugins/github_graphql/tasks/issue_collector.go:242

This only fires for repos whose OPEN-issue count is large enough to make the second collector iterate beyond the first page (≥10 OPEN issues, per the PageSize: 10 in the source). hh-server, hh-pegasus, hh-pegasus-monorepo, hh-relay, and hungryhub-terraform all qualify; the other 10 repos do not.

Fix

Bump the image tag from v1.0.3-beta12 to v1.0.4-alpha in hungryhub-devlake/docker-compose.yml. PR: hungryhub-team/hungryhub-devlake#10.

v1.0.4-alpha removes the broken code path entirely:

  • issue_collector.go: 399 → 208 lines
  • 0 references to requestedIssues, missingGithubIssue, cleanupMissingGithubIssues

The new code keeps both collectors (collect new + refresh OPEN) but drops the broken reconciliation map and the cleanupMissingGithubIssues plumbing that depended on it.

The same tag bump is applied to devlake-config-ui.

Deployment

# Local
scp .worktrees/hungryhub-devlake-upgrade-graphql-fix/docker-compose.yml \
    root@62.238.41.27:/opt/devlake/docker-compose.yml

ssh root@62.238.41.27 "cd /opt/devlake && docker compose pull && docker compose up -d"

# Verify containers
ssh root@62.238.41.27 "cd /opt/devlake && docker compose ps"
# NAME                    STATUS    IMAGE
# devlake-cloudflared-1   Up X      cloudflare/cloudflared:latest
# devlake-config-ui-1     Up X      devlake.docker.scarf.sh/apache/devlake-config-ui:v1.0.4-alpha
# devlake-devlake-1       Up X      devlake.docker.scarf.sh/apache/devlake:v1.0.4-alpha

# Verify API
curl -s https://devlake.hungryhub.com/api/health
# → 200 OK

# Check blueprint connections (per runbook gotcha #1 — alpha may drop them)
curl -s https://devlake.hungryhub.com/api/blueprints/4 | jq '.connections | length'
# → 1 (intact, no re-PATCH needed)

# Manual trigger to verify the fix end-to-end
curl -X POST https://devlake.hungryhub.com/api/blueprints/4/trigger \
  -H "Content-Type: application/json" -d '{}'
# → pipeline 15, status TASK_CREATED → TASK_RUNNING → TASK_COMPLETED

Rollback

ssh root@62.238.41.27
cd /opt/devlake
sed -i 's/v1.0.4-alpha/v1.0.3-beta12/' docker-compose.yml
docker compose pull && docker compose up -d

The TASK_PARTIAL state is recovered by simply running the next nightly pipeline with the new code — no DB cleanup required. The data that the second collector would have refreshed is regenerated on the next successful run.

Lessons learned

  1. No monitoring on pipeline.status != TASK_COMPLETED. The whole team missed 4 nights of partial failures because there’s no alarm. Add a CloudWatch alarm or simple cron health check:

    # Cron: 02:35 daily (after the 02:00 pipeline)
LAST=$(curl -s 'https://devlake.hungryhub.com/api/pipelines?pageSize=1' | jq -r '.pipelines[0].status')
if [ "$LAST" != "TASK_COMPLETED" ]; then
  aws sns publish --topic-arn arn:aws:sns:ap-southeast-1:202255947274:devlake-alerts \
    --subject "DevLake pipeline status: $LAST" \
    --message "$(curl -s 'https://devlake.hungryhub.com/api/pipelines/1/tasks' | jq -r '.tasks[] | select(.status==\"TASK_FAILED\") | .options.fullName')"
fi

    (Owns: infra squad. Tracked in follow-up issue.)

  2. v1.0.3-beta12 was tagged 2022-12-30 and used a merico-ai/graphql library with bugs that were fixed upstream. Worth pinning to a less-stale tag and subscribing to Apache DevLake security/release notifications.

  3. The 2-week-old full backfill on 2026-06-03 was the only recent TASK_COMPLETED — it ran for 10.6 hours (38,289s), so a full-resync is doable if needed but expensive. The post-fix nightly pipeline takes ~30-60 min per repo (heaviest: hh-server at ~80 min due to 68k+ workflow runs).

  4. GraphQL field removals are silent killers. When a major API deprecates a field, the client library usually fails noisily, but a tag mismatch like this one (an unexported Go field with the same name as the deprecated GraphQL field) emits a confusing error that requires reading the merico-ai/graphql source to diagnose. Worth adding a smoke test in CI that runs a representative blueprint against a non-prod DevLake and asserts TASK_COMPLETED.

Follow-ups

  • Add CloudWatch alarm on pipeline.status != TASK_COMPLETED (infra squad)
  • Add a post-deploy smoke test that triggers blueprint 4 and asserts green within 60 min
  • Subscribe to apache/incubator-devlake releases via RSS in Slack #infra-alerts
  • Review the skipOnFail: true setting on blueprint 4 — masking the failure for 4 days is what let this slip

Second regression: release_convertor nil-pointer panic

After upgrading to v1.0.4-alpha, the nightly pipeline started hitting a different panic:

runtime error: invalid memory address or nil pointer dereference
  plugins/github/tasks.ConvertRelease.func1
  plugins/github/tasks/release_convertor.go:76

  PublishedAt:  *githubRelease.PublishedAt,

This is a regression introduced by commit b58f6ece (“fix(github): fix zerotime issues for GraphQL”) which sits between v1.0.3-beta12 and v1.0.4-alpha. The commit:

  1. Changed github_releases.PublishedAt from time.Time to *time.Time (correct, to handle zero-time releases)
  2. Updated the GraphQL extractor to call utils.NilIfZeroTime (correct)
  3. But also changed the REST github plugin’s release_convertor.go to dereference the new nullable pointer without a nil check
  4. And removed the WHERE published_at IS NOT NULL filter that the convertor cursor used in v1.0.3-beta12

So any repo with at least one release where published_at IS NULL (e.g. is_draft = 1) will panic the entire ConvertRelease task. For the HungryHub team, only hungryhub-team/hh-pegasus has such records (2 draft releases: 1.73.0 and 1.75.0). The other 14 repos have all-published releases and process fine.

This bug was hidden under v1.0.3-beta12 because the requestedIssues failure in github_graphql happened earlier in the task. With the graphql fix in place, the pipeline now runs far enough to expose the new bug.

Mitigation (applied 2026-06-09)

  1. Published the 2 draft releases on GitHub via gh api PATCH /repos/hungryhub-team/hh-pegasus/releases/{id} -f draft=false:
    • 1.73.0 (id=305564434) — now published_at = 2026-06-09T03:44:02Z
    • 1.75.0 (id=306883054) — now published_at = 2026-06-09T03:44:14Z
  2. Deleted the 2 stale rows in _tool_github_releases so the next Convert Releases (which still has the panic) won’t fail. The rows will be re-created by the next Extract Releases with the new (non-null) published_at.

The reason for the DB cleanup: DevLake’s Collect Releases for hh-pegasus failed on the GitHub REST API rate limit in pipelines 16/17/18, so the local raw data (_raw_github_graphql_release) still shows the 2 records as drafts. Until Collect Releases succeeds for hh-pegasus with a fresh rate-limit window, the Extract Releases step will keep recreating the rows with the old (draft) data.

Permanent fix (upstream)

A patch is prepared at knowledge-base/src/DevOps/upstream-fix-release-convertor.patch. It restores the cursor filter and adds a defensive nil check:

--- a/backend/plugins/github/tasks/release_convertor.go
+++ b/backend/plugins/github/tasks/release_convertor.go
@@ -53,7 +53,8 @@ func ConvertRelease(taskCtx plugin.SubTaskContext) errors.Error {
 	rawDataSubTaskArgs, data := CreateRawDataSubTaskArgs(taskCtx, RAW_RELEASE_TABLE)
 	cursor, err := db.Cursor(
 		dal.From(&models.GithubRelease{}),
-		dal.Where("connection_id = ? and github_id = ?", data.Options.ConnectionId, data.Options.GithubId),
+		dal.Where("published_at IS NOT NULL AND connection_id = ? and github_id = ?",
+			data.Options.ConnectionId, data.Options.GithubId),
 	)
 	if err != nil {
 		return err
@@ -69,6 +70,11 @@ func ConvertRelease(taskCtx plugin.SubTaskContext) errors.Error {
 		RawDataSubTaskArgs: *rawDataSubTaskArgs,
 		Convert: func(inputRow interface{}) ([]interface{}, errors.Error) {
 			githubRelease := inputRow.(*models.GithubRelease)
+			// Skip releases that have no published_at (e.g. drafts, unpublished releases).
+			// Without this guard, dereferencing a nil *time.Time panics the whole task.
+			if githubRelease.PublishedAt == nil {
+				return nil, nil
+			}
 			release := &devops.CicdRelease{
 				DomainEntity: domainlayer.DomainEntity{
 					Id: releaseIdGen.Generate(githubRelease.ConnectionId, githubRelease.Id),

PR to be opened against apache/incubator-devlake — see upstream-fix-release-convertor.README.md in the same directory for context.

Additional follow-ups

  • Submit upstream PR for release_convertor.go nil-pointer fix
  • Until upstream lands, consider building a custom devlake image with the patch applied
  • Increase the GitHub PAT rate limit budget (5,000 req/hr is insufficient for 15 repos × daily × both REST + GraphQL) — options: split into multiple PATs per squad, or upgrade to a GitHub App with higher limits
  • Investigate why the github_graphql plugin uses REST endpoints for some subtasks (CollectPrReviewComments, CollectPrReviewComments → repos/{}/pulls/comments). This causes a 5x rate-limit amplification when running 15 repos.

Third follow-on: GitHub API rate limit (discovered 2026-06-09)

After fixing the requestedIssues and release_convertor bugs, the nightly pipeline started failing with a different error class: every night, 8–12 of 15 github_graphql tasks failed with 403 API rate limit exceeded from the same user (saiqulhaq-hh, our 5,000 req/hr PAT).

Root cause

The HungryHub team’s 5,000 req/hr PAT budget is insufficient for the workload:

  • 15 repos × 1 nightly run/day
  • Each task does both GraphQL (paginated, ~600 points per repo) and REST (some subtasks in github_graphql plugin use /repos/.../pulls/comments etc., which costs against the REST rate limit pool)
  • Total: ~9,000–15,000 requests/day vs. the 5,000/hour ceiling

This was always a latent problem but the earlier graphql bug masked it (failing at step ~5 of the task, before exhausting the budget).

Fix (applied 2026-06-09)

Switched the DevLake GitHub connection from PAT auth to a custom GitHub App with these credentials:

  • App ID: 4006819
  • Installation ID: 139078097 (on hungryhub-team org)
  • Required permissions: Contents / Issues / Pull requests / Actions / Deployments / Metadata — all read-only
  • Private key: stored in AWS SSM at /devlake/github-app-secret (SecureString)

Patched the DevLake connection via API:

curl -X PATCH "https://devlake.hungryhub.com/api/plugins/github/connections/1" \
  -H "Content-Type: application/json" \
  -d '{
    "name":"hungryhub-team",
    "endpoint":"https://api.github.com/",
    "authMethod":"AppKey",
    "appId":"4006819",
    "installationId":139078097,
    "secretKey":"<PEM contents from SSM>",
    "enableGraphql":true,
    "rateLimitPerHour":4500
  }'

Result

PipelineStatusNotes
15 (with App)TASK_PARTIAL 30/31Only the release_convertor panic on hh-pegasus failed
18 (with App)TASK_PARTIAL 31/31Same, no rate limit failures
21 (with App + actions/deployments perms)TASK_PARTIAL 30/31No rate limit failures, no graphql errors
22 (tonight’s nightly, with App)TASK_PARTIAL 31/31Single failure: release_convertor panic

Zero rate-limit failures since switching to the GitHub App. The DORA dashboards now have data flowing for 14 of 15 repos; only hh-pegasus is missing the release-frequency metric (pending the upstream fix).

Follow-ups

  • Submit the release_convertor.go upstream PR to Apache DevLake
  • If we still hit the 5,000 req/hr limit as the team grows, look at the multi-PAT option: DevLake’s github plugin supports comma-separated tokens in a single connection (round-robin), so 2-3 PATs from different team members would give us N×5,000/hr without the GitHub App
  • The github_graphql plugin’s reliance on REST endpoints for some subtasks is wasteful (5× rate-limit amplification). Worth filing an upstream issue

Fourth follow-on: Grafana datasource database: lake was empty (discovered 2026-06-10)

After the GitHub App fix, the pipeline ran cleanly. But the DORA dashboards still showed “No data” / errors. Investigating the Grafana datasource revealed a silent configuration error: the database field was empty.

Datasource "devlake" (uid dfo08jdx29z40f, id 15)
  url:       devlake-mysql-hh-development.f.aivencloud.com:15939
  database:  ""     ← BUG
  user:      avnadmin
  jsonData:  {tlsSkipVerify: true, connMaxLifetime: 14400, maxIdleConns: 100, ...}

Without database: lake, the MySQL plugin doesn’t route queries to the right schema. All table queries returned 500 db query error: query failed - please inspect Grafana server log for details. The Database Connection OK health check passed because it only verifies connectivity, not schema routing.

Fix: set database: lake in the Grafana UI (Connection → Databases → lake) and Save & test. The token-based Editor role is required to PATCH via API.

After the fix, all DORA panel queries return real data (pull_requests=5894, cicd_pipelines=137343, cicd_deployment_commits=4313 etc.).

Follow-ups

  • Add a CI/smoke test that runs a representative query against the DevLake datasource and asserts the response has data — would have caught this in minutes, not days
  • Note in the runbook that the database field is required (not optional) for the mysql plugin

See Grafana_DevLake_Datasource_Setup.md for the full setup checklist.

Fifth follow-on: DORA dashboard SQL parse error (discovered 2026-06-10)

Once the datasource was fixed, the dashboards still didn’t render. The error in Grafana was:

db query error: Error 1054 (42S22): Unknown column 'N/A. Please check if you have collected deployments.' in 'field list'

The DORA panel SQL (panels 11, 12, 14, 17, plus the “Overall DORA Metrics” panel 8) has CASE / ELSE branches that return literal strings using MySQL double-quote syntax:

ELSE "N/A. Please check if you have collected deployments." END

In MySQL, double-quoted identifiers are column names, not string literals. The parser fails at parse time (not runtime), so the dashboard never renders regardless of whether there’s data. This is an upstream bug in the DORA dashboard JSON exports from apache/incubator-devlake.

Workaround: replace double-quotes with single-quotes in the literal strings, or use a CONCAT('N/A. ...') form. Until upstream fixes it, the dashboards need a local patch.

See DORA_Dashboard_SQL_Bug.md for the full bug analysis, exact lines to fix, and a sed command to apply the workaround.