Kibana APM Error Reports — API Retrieval & Interpretation Runbook

Verified against Kibana 8.15.0 (2026-08-05). The internal /internal/apm/... routes below are version-specific and can change in any Kibana minor release. Always re-verify against the cluster you are reading from (see §2) instead of treating this document as a stable API contract.

This runbook explains how engineers and AI-assisted operators can pull an APM error group out of Kibana over its HTTP API and read the result safely. It covers authentication, route discovery, sample enumeration, detail retrieval, field selection, comparison across samples, and a causal checklist for interpreting stack traces. It deliberately avoids dumping raw event payloads to chat, logs, or commits: APM documents can carry request parameters, headers, URLs, user context, labels, or business data.

Companion docs:

• Setup & verification of the agent itself: APM Setup Guide
• Cluster & credential context: Credential Access Monitoring, Logs & APM
• Prior incident that surfaced these route shapes: Fixing Kibana APM Service Parameters and Environment Filtering

Table of contents

1. When to use this runbook
2. Safety rules
3. Vocabulary — what each ID means
4. Procedure
   • §4.1 Translate a Kibana APM UI URL to API inputs
   • §4.2 Discover routes and Kibana version on the cluster
   • §4.3 Authenticate without leaking the API key
   • §4.4 List sample IDs for the error group
   • §4.5 Fetch one sample detail (or all of them)
   • §4.6 Compare samples across the group
   • §4.7 Interpret stack traces causally
   • §4.8 Sanitize and clean up
5. Reference — sanitized case study
6. Causal checklist
7. Evidence limitations
8. Discovery warnings (HAR, browser DevTools)
9. Related docs

1. When to use this runbook

Use it when all of the following are true:

1. You have a Kibana APM Errors link from a teammate, a Grafana alert, a Datadog alert mirror, or your own investigation, and you want the data outside the browser (to grep it, diff it across timestamps, compare pods, feed it to a downstream tool, or capture it as incident evidence).
2. The Kibana UI does not give you what you need (e.g., you need every sample, not the top three; you need a kuery you cannot type in the UI).
3. You have an API key for the target cluster in a .env file outside this repo (see §2).

Do not use it for routine APM UI work — the browser is faster for one-off reads.

2. Safety rules

These rules are non-negotiable. They are the reason this runbook exists.

1. Never put the API key into a command-line argument. ps, /proc/<pid>/cmdline, shell history files, CI logs, and screen-sharing tools all surface argv. Use .env sourcing + stdin or a process-argument-safe curl config (see §4.3 and §4.5).
2. Never commit the API key, the .env file, or any HTTP response that embeds it. A successful response does not contain the key, but a mis-configured proxy error page can. If you accidentally capture one, rotate the key.
3. Never dump a raw APM error event into chat, a ticket, or a PR. APM documents can carry request parameters, headers, URLs, user context (user.id, user.email), labels, and business data. Select only what the analysis needs (see §4.5).
4. If you must keep a raw response for repro, store it under /tmp with umask 077, restrict permissions (chmod 600), inspect it once, redact anything outside the allowlist, and rm it before the next phase. Do not move it into the repo.
5. Use absolute UTC ISO timestamps in incident evidence. A Kibana UI range like now-15h is relative; reruns weeks later will pull a different window. Capture the resolved start and end once and reuse.
6. Treat every route shape under /internal/ as unstable. Kibana marks them as internal for a reason. Verify the cluster’s own copy (see §4.2) before relying on the routes listed here.

3. Vocabulary — what each ID means

APM talks use four different identifiers that look interchangeable but are not. Confusing them is one of the main ways incident analysis goes wrong.

What you should not read into these:

• A list of “worst samples” returned by Kibana is a sample of duration outliers — it is not a measure of prevalence. Prevalence requires main_statistics (occurrence count) and distribution (bucket histogram) for the group; see §4.4.
• A single sample’s stack trace is not the cause; the exception class on top of the stack only marks the operation that was interrupted (see §4.7).

4. Procedure

§4.1 Translate a Kibana APM UI URL to API inputs

Kibana APM Errors URLs follow this shape:

https://<kibana-host>/app/apm/services/<serviceName>/errors/<groupId>
  ?rangeFrom=<start>&rangeTo=<end>&environment=<environment>&kuery=<kuery>

Map each piece to an API input:

Example conversion (Kibana relative range → absolute UTC):

rangeFrom = now-15h  →  start = 2026-08-05T01:30:00.000Z
rangeTo   = now      →  end   = 2026-08-05T16:30:00.000Z

Keep start/end in your incident note as evidence; later reruns will hit the same window.

§4.2 Discover routes and Kibana version on the cluster

The routes in §4.4–§4.5 were verified on Kibana 8.15.0. Before relying on them:

# 1) Confirm version. The 'version.number' field is the cluster version.
set -a; . ./.env; set +a
curl --silent --show-error --config - \
  "${ELASTIC_APM_BASE_URL}/api/status" <<EOF
header = "Authorization: ApiKey ${ELASTIC_APM_API_KEY}"
EOF

Expected: a JSON document with a version.number field (e.g., 8.15.0).

If the version differs, do not assume the internal route shapes are identical — the response key names, the parameter names (start vs rangeFrom), and the supported query shapes have all changed across Kibana minor versions. Inspect the cluster’s own code or its currently-mounted browser network captures instead (with the credential caveats in §8).

To find the route a Kibana page actually calls: open the APM Errors page in your browser, open DevTools → Network, filter by Fetch/XHR, and trigger the action you want to script. Note the path and the response shape. Treat that as the source of truth for the cluster you are reading from.

§4.3 Authenticate without leaking the API key

The verified pattern uses Kibana’s ApiKey auth header. The header is never placed in argv. Two patterns, in order of preference:

Pattern A — Python with stdlib only (recommended). Reads .env itself, holds the key in a Python string, builds the header in memory, and prints only a sanitized allowlist of the response. See §4.4 and §4.5 for the copy-paste-ready code.

Pattern B — curl with config on stdin. Use this when the script is shell-only. The header is built from a shell variable and travels through stdin (curl --config -), so ps only shows curl --config -.

# Source the .env into the shell so values never enter curl's argv.
set -a
. ./.env
set +a

curl --silent --show-error --config - \
  "${ELASTIC_APM_BASE_URL}/internal/apm/services/${SERVICE_NAME}/errors/groups/main_statistics?start=${START}&end=${END}&environment=${ENVIRONMENT}" \
  <<EOF
header = "Authorization: ApiKey ${ELASTIC_APM_API_KEY}"
EOF

Do not write curl -H "Authorization: ApiKey ${ELASTIC_APM_API_KEY}". The shell will expand the variable into curl’s argv before exec, and the expanded value will appear in ps, /proc/<pid>/cmdline, and shell history.

If you need to keep a curl config file (e.g., for repeated calls in a shell script), create it with umask 077, then chmod 600, then rm -f it as the last step of the script.

§4.4 List sample IDs for the error group

This is the first call after you have a serviceName + groupId. It returns the IDs of the samples Kibana has retained for the group in the window. The default is the few longest-duration samples, not all occurrences.

#!/usr/bin/env python3
"""List retained sample IDs for an APM error group.

Reads ELASTIC_APM_BASE_URL and ELASTIC_APM_API_KEY from a .env file you
point it at via ENV_FILE. Holds the API key in memory only. Prints only
the sample IDs and a coarse occurrence count — no event fields, headers,
URLs, or exception attributes.
"""
import json
import os
import sys
import urllib.parse
import urllib.request

ENV_FILE = os.environ.get("ENV_FILE", "./.env")
SERVICE_NAME = os.environ["SERVICE_NAME"]      # e.g. "cosmos"
GROUP_ID = os.environ["GROUP_ID"]              # 32-hex error group ID
START = os.environ["START"]                    # ISO 8601 UTC, e.g. "2026-08-05T01:30:00.000Z"
END = os.environ["END"]
ENVIRONMENT = os.environ.get("ENVIRONMENT", "production")


def load_env(path):
    values = {}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith("#") or "=" not in line:
                continue
            k, v = line.split("=", 1)
            values[k.strip()] = v.strip().strip('"').strip("'")
    return values


def apm_get(env, path, query):
    url = env["ELASTIC_APM_BASE_URL"].rstrip("/") + path + "?" + urllib.parse.urlencode(query)
    req = urllib.request.Request(url, method="GET")
    # Header built in memory; never appears in argv or ps output.
    req.add_header("Authorization", "ApiKey " + env["ELASTIC_APM_API_KEY"])
    req.add_header("Accept", "application/json")
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            return resp.status, json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as e:
        # Intentionally do not echo e.headers or e.url. Both can leak
        # credentials if a misconfigured proxy intercepted the request.
        print(f"http error: status={e.code}", file=sys.stderr)
        sys.exit(2)


def main():
    env = load_env(ENV_FILE)

    query = {"start": START, "end": END, "environment": ENVIRONMENT}
    path = f"/internal/apm/services/{SERVICE_NAME}/errors/{GROUP_ID}/samples"
    status, body = apm_get(env, path, query)
    if status != 200:
        print(f"unexpected status: {status}", file=sys.stderr)
        sys.exit(2)

    # Strict allowlist: only sample IDs and coarse counts. No event fields.
    # Key names vary by Kibana version (see note below); check both spellings.
    sample_ids = body.get("errorSampleIds", body.get("sampleIds", []))
    occurrences = body.get(
        "occurrencesCount",
        body.get("occurences", body.get("occurrences")),
    )
    print(json.dumps({"sample_ids": sample_ids, "occurrences": occurrences}, indent=2))


if __name__ == "__main__":
    main()

Save as apm_list_samples.py, set the env vars, and run:

ENV_FILE=./.env \
SERVICE_NAME=<serviceName> \
GROUP_ID=<32-hex-group-id> \
START=<ISO-8601-UTC> \
END=<ISO-8601-UTC> \
ENVIRONMENT=<environment> \
python3 apm_list_samples.py

The samples response keys are version-specific like the routes (§4.2). On Kibana 8.15.0 the payload uses errorSampleIds and occurrencesCount; older/newer minors have used sampleIds and occurences/occurrences. The script reads both spellings, but if sample_ids comes back empty on a different cluster, re-verify the key names against that Kibana’s own apm/routes source before concluding the group has zero samples.

To get the full occurrence count and rate for the same window (rather than retained samples), call main_statistics on the group:

GET /internal/apm/services/{serviceName}/errors/{groupId}/main_statistics
    ?start={ISO}&end={ISO}&environment={environment}&kuery={kuery}

It returns a occurrencesPerMinute series and an occurrences total. Use this when “how bad?” is the question; use samples when “what did the failures actually look like?” is the question.

§4.5 Fetch one sample detail (or all of them)

Once you have a sample ID, pull its full event document:

GET /internal/apm/services/{serviceName}/errors/{groupId}/error/{errorId}
    ?start={ISO}&end={ISO}&environment={environment}

The verified response shape (Kibana 8.15.0) is { "error": { ...event... } }. The event itself has nested fields under error.exception[].stacktrace, plus error.transaction and error.trace (may be present even when no top-level transaction is returned). The other top-level event fields typically seen include @timestamp, service, parent, span, http, url, host, cloud, process, and error.grouping_key.

Strict allowlist for the Python reader. Anything outside the allowlist is intentionally dropped — if the analysis needs more, add a field by name, do not blanket-print the document:

#!/usr/bin/env python3
"""Read one APM error sample and emit only an allowlisted summary.

This deliberately drops:
- request headers and parameters
- exception messages (often contain user input, URLs, file paths)
- user.* fields (id, email, name)
- labels and custom context (free-form user data)
- raw URLs the agent captured
- raw cache keys
- stack frame `abs_path` (filesystem path) and `vars` (local variable snapshot)
"""
import json
import os
import sys
import urllib.parse
import urllib.request

ENV_FILE = os.environ.get("ENV_FILE", "./.env")
SERVICE_NAME = os.environ["SERVICE_NAME"]
GROUP_ID = os.environ["GROUP_ID"]
ERROR_ID = os.environ["ERROR_ID"]
START = os.environ["START"]
END = os.environ["END"]
ENVIRONMENT = os.environ.get("ENVIRONMENT", "production")

# Allowlist of (input-path, output-name) tuples. Add to this list only when
# the new field is known to be non-sensitive; do not use a blanket dump.
ALLOWLIST = [
    ("error.@timestamp",                         "timestamp"),
    ("error.service.name",                       "service"),
    ("error.service.environment",                "environment"),
    ("error.service.node.name",                  "service_node"),
    ("error.transaction.id",                     "transaction_id"),
    ("error.trace.id",                           "trace_id"),
    ("error.error.grouping_key",                 "grouping_key"),
    # Exception type identifies the terminating boundary (see §4.7).
    # Exception *message* is intentionally omitted — it often contains user
    # input, URLs, or file paths. Add it explicitly if you have a
    # sanitized source and have confirmed it carries no PII for this group.
    ("error.error.type",                         "error_type"),
    ("error.error.exception.0.type",             "exception_type"),
    # Stack frames: only filename + lineno + function. Drop abs_path
    # (filesystem path) and vars (local variable snapshot).
    ("error.error.exception.0.stacktrace.0.filename",   "frame0_file"),
    ("error.error.exception.0.stacktrace.0.lineno",     "frame0_line"),
    ("error.error.exception.0.stacktrace.0.function",    "frame0_function"),
    ("error.error.exception.0.stacktrace.1.filename",   "frame1_file"),
    ("error.error.exception.0.stacktrace.1.lineno",     "frame1_line"),
    ("error.error.exception.0.stacktrace.1.function",    "frame1_function"),
    ("error.error.exception.0.stacktrace.2.filename",   "frame2_file"),
    ("error.error.exception.0.stacktrace.2.lineno",     "frame2_line"),
    ("error.error.exception.0.stacktrace.2.function",    "frame2_function"),
    ("error.host.os.platform",                   "host_platform"),
    ("error.process.thread.id",                  "thread_id"),
]


def get_by_path(doc, dotted):
    cur = doc
    for part in dotted.split("."):
        if isinstance(cur, list):
            try:
                cur = cur[int(part)]
            except (ValueError, IndexError):
                return None
        elif isinstance(cur, dict):
            cur = cur.get(part)
        else:
            return None
        if cur is None:
            return None
    return cur


def load_env(path):
    values = {}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith("#") or "=" not in line:
                continue
            k, v = line.split("=", 1)
            values[k.strip()] = v.strip().strip('"').strip("'")
    return values


def apm_get(env, path, query):
    url = env["ELASTIC_APM_BASE_URL"].rstrip("/") + path + "?" + urllib.parse.urlencode(query)
    req = urllib.request.Request(url, method="GET")
    req.add_header("Authorization", "ApiKey " + env["ELASTIC_APM_API_KEY"])
    req.add_header("Accept", "application/json")
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            return resp.status, json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as e:
        print(f"http error: status={e.code}", file=sys.stderr)
        sys.exit(2)


def main():
    env = load_env(ENV_FILE)
    query = {"start": START, "end": END, "environment": ENVIRONMENT}
    path = f"/internal/apm/services/{SERVICE_NAME}/errors/{GROUP_ID}/error/{ERROR_ID}"
    status, body = apm_get(env, path, query)
    if status != 200:
        print(f"unexpected status: {status}", file=sys.stderr)
        sys.exit(2)

    out = {}
    for src, dst in ALLOWLIST:
        v = get_by_path(body, src)
        if v is not None:
            out[dst] = v

    print(json.dumps(out, indent=2))


if __name__ == "__main__":
    main()

If you need every retained sample, loop over sample_ids from §4.4 and call this script once per ID. Group and sort the resulting JSONs in memory — do not write each raw event to disk unless required for repro (and then follow §4.8).

§4.6 Compare samples across the group

Once you have the allowlisted summary for several samples, look for:

1. Same exception type, same top frame? Confirms they belong to the same group. If not, you have collected from the wrong group.
2. Same pod, or different pods? A single-pod cluster means the fault probably belongs to that pod’s environment (memory pressure, image regression, network). A multi-pod spread usually means an upstream dependency.
3. Same @timestamp cluster, or scattered? Bursts often correlate with deploys, cron jobs, or upstream provider incidents. Scattered usually means per-request variance (object size, payload, retry timing).
4. Same trace ID? If two samples share a trace ID, you are looking at the same distributed request captured twice — likely a retry. If they share no IDs, they are independent failures.
5. Different exception types in the same group? This is rare and means the grouping hash collided on something noisy (a message field). Treat the group ID as suspect and re-check the kuery.

The supporting routes for time-distribution analysis:

• GET /internal/apm/services/{serviceName}/errors/{groupId}/distribution → bucketed occurrence histogram across the window (offset, numBuckets body or query depending on the version). Use this to see whether the group is a steady drip, a single burst, or periodic.
• POST /internal/apm/services/{serviceName}/errors/groups/detailed_statistics → per-group statistics across many groups for the same window. Useful when comparing the suspect group against its peers.

§4.7 Interpret stack traces causally

Stack traces are read outside-in, not top-down. The exception type on top is what was interrupted, not what caused the failure.

Apply this in order:

1. Identify the terminating boundary. The top frame’s class is almost always the boundary that gave up. Examples:
   • Rack::Timeout::RequestTimeoutException — the Rack middleware killed the request because it ran longer than the configured timeout.
   • Net::OpenTimeout — the socket gave up before connecting.
   • Net::ReadTimeout — the socket gave up while reading.
   • Aws::S3::Errors::RequestTimeout — S3 returned an explicit timeout.
2. Read the next several frames outward until you reach application code. Note which dependency each frame belongs to.
3. Isolate application frames. The application frames are the ones where the team owns the code. The frames below them are the dependencies. The boundary between the two is where the timeout fired.
4. Compare all retained samples (see §4.6). The cause is whatever is common across samples with different pods and timestamps. The boundary is what is consistent across samples regardless of cause.
5. Require disconfirming evidence before naming a dependency as root cause (see §6). A dependency is not the cause just because it appears in the stack.

§4.8 Sanitize and clean up

When the investigation is done:

1. rm any raw event captures from /tmp (ls /tmp/apm-* is a fast audit). If you used a config file under /tmp, rm it too.
2. Unset shell-side secrets if you used Pattern B: unset ELASTIC_APM_API_KEY.
3. If you pasted anything into chat or a ticket, audit for accidentally pasted headers (Authorization: ApiKey …), user IDs, or full URLs.
4. If the API key was exposed (argv, commit, screenshot, ticket), rotate it before continuing — the key is now considered compromised.

5. Reference — sanitized case study

This is a sanitized summary of one real investigation. It is included here because the failure mode (a timeout boundary being mistaken for the cause) is one of the most common APM analysis mistakes. The original event documents are not embedded.

Inputs:

• Service: cosmos (production)
• Error group: b5f9d6f2895e2471e8c3af73a5d016ba
• Window: 15 hours (resolved to absolute UTC timestamps before analysis)
• Source URL: Kibana APM → Services → cosmos → Errors → group

Observed across 3 retained samples:

Supported conclusion:

• Rack Timeout killed each request at the 30-second boundary while it was blocked in Net::HTTP#wait_readable waiting for an S3 response during synchronous CarrierWave storage.
• The 30-second wall time, the exception class, the top frame, and the HTTP method are all consistent with that explanation.

Unsupported from the events alone:

• Whether the initiating cause was S3 service latency, network latency between the EKS pod and S3, retry behaviour inside the AWS SDK, or object size affecting per-request latency.
• Whether the same group is correlated with an unrelated mobile incident — it is not: service, endpoint, and timestamps differ.
• Whether the fix is “raise the Rack timeout”, “raise CarrierWave’s timeouts”, “make storage async”, “raise Puma worker count”, or “raise the S3 SDK read timeout”. Each is a different change with a different blast radius.

Why this case study is here: the terminating exception (Rack::Timeout::RequestTimeoutException in the actual document) is not the cause; it is the boundary that fired because the request was already too slow. Naming “Rack Timeout” as the root cause would be wrong; naming “AWS S3” as the root cause without disconfirming evidence would also be wrong. The correct next step is the disconfirming-evidence loop in §6.

6. Causal checklist

For every APM error group you investigate, answer these in writing before naming a root cause:

1. Initiating trigger — what request, event, or schedule started the failing path? (Inbound HTTP, Sidekiq job, cron, Karafka message, ActiveJob retry, …)
2. Masking / exposure condition — what made this normally-tolerated failure become visible? (Timeout boundary, retry exhausted, breaker open, alert fired, customer report.)
3. Timeout / termination boundary — which library or middleware actually raised the exception? (Rack::Timeout, Net::*, SDK timeouts, Puma worker timeout, Kubernetes pod readiness.)
4. Visible symptom — what the operator saw first (5xx, alert, dashboard spike, log noise).

Then require all of the following before naming a dependency as root cause:

• The dependency’s stack frame appears in every retained sample.
• The dependency’s timing in distribution matches the symptom window.
• The dependency has a vendor-side status event, an SDK error log, or an upstream latency metric that correlates with the window.
• At least one disconfirming check has been run (different object size → same error? smaller payload → succeeds? retry → succeeds? another service that calls the same dependency → unaffected?) and recorded in the incident note.

If any item is unchecked, the right next step is to gather that evidence, not to declare the cause.

7. Evidence limitations

What this runbook’s evidence cannot tell you:

• Prevalence. A list of “worst samples” is the longest-duration outliers, not the most-frequent occurrences. Use main_statistics or distribution for prevalence.
• Rates / percentiles. Per-route latency percentiles require the transactions route, not the errors route.
• Cross-service causation. An error on service A does not prove A’s dependency B caused it. B’s own error rate and latency must be checked independently.
• Customer impact. Kibana APM samples requests, not users. The user count behind a group has to come from elsewhere (request logs, the service’s own audit log, the database).
• Pre-window history. A group that started failing 30 minutes ago looks identical in API to one that has been failing for weeks unless you widen the window yourself.

8. Discovery warnings (HAR, browser DevTools)

You can find the route shapes Kibana uses by:

• Reading the Kibana source for your version (github.com/elastic/kibana/tree/8.15/x-pack/plugins/apm/routes).
• Capturing browser DevTools → Network as you click through the APM UI.

Both work, but DevTools captures are dangerous:

• A HAR file contains every request and response, including the Authorization: ApiKey … header on every fetch. Treat the HAR like a credential.
• A response body can contain raw event documents with user data, headers, URLs, or business values. Do not paste it into a ticket; redact the header line and the sensitive fields before sharing.
• If you commit a HAR to a repo, rotate the API key — assume it is compromised the moment the file lands on a remote.

The safer path for discovery is the Kibana source tree: same answer, no credential exposure.

9. Related docs

• APM Setup Guide — agent config, ConfigMap, per-namespace service_name/environment mapping.
• Credential Access Monitoring, Logs & APM — Kibana hosts and per-environment access (UI credentials).
• Fixing Kibana APM Service Parameters and Environment Filtering — prior incident that surfaced the current param shape (environment is single-value, not an array).
• Fix for Production Config-Path IndexError in apm/apm_alerts.py — related APM-alerts pipeline change.
• Kibana Staging Log Access (logs.hhstaging.dev) — Kibana logs (Discover), unrelated to the APM errors API in this runbook. Same host family, different UI.