Platform-Monitoring Alertmanager Slack Routing Guide

Source of record: hungryhub-team/platform-infra on main — the runbook docs/runbooks/platform-monitoring-alertmanager-slack.md and the alertmanager block it documents in gitops/services/platform-monitoring/kube-prometheus-stack/values.yaml. This page is a reading aid and a pointer, not a second copy of the mapping. If this page and the runbook on main disagree, the runbook wins and this page is stale. Transcribed from main on 2026-10-23 (PR hungryhub-team/platform-infra#173).

This page describes configuration, not observed behaviour. Nothing here asserts that a notification has been seen arriving in any channel. Read What a repository check cannot prove before treating any green signal here as proof of delivery.

The five receivers at a glance

The kube-prometheus-stack chart in the platform-monitoring namespace renders one Slack receiver per severity level. Each receiver posts to its own, pre-existing Slack channel.

The channel IDs are the part to copy without retyping. A reader pasting one into a Slack search or an API call lands somewhere else if a character is dropped, and the two high* channel names differ by a single word (#alerts-highest-severity-level vs #alerts-high-severity-level), which is the easiest thing here to get wrong by eye.

No channel was created or renamed to serve this routing. All five already existed in the workspace.

One webhook, five destinations

All five receivers read a single Slack incoming-webhook URL from api_url_file: /etc/alertmanager/secrets/slack-webhook-url. The webhook was created against #alerts-highest-severity-level as the highest-band default, but Alertmanager overrides the destination per receiver via chat.postMessage, so one webhook serves all five channels. The file is mounted from the Kubernetes Secret alertmanager-slack-webhook, which an ExternalSecret keeps projected from AWS Secrets Manager. Do not commit the URL; it is a credential that lives only in Secrets Manager.

The severity scale, and which severities are actually in use

The scale is five levels: highest, high, medium, low, lowest.

critical, warning and info are mapped onto the top three levels of that scale:

• critical → highest band → #alerts-highest-severity-level
• warning → high band → #alerts-high-severity-level
• info → medium band → #alerts-medium-severity-level

This mapping is decided and ratified, not an open question. The reason it exists is that critical, warning and info are the severities this stack actually emits: every rule in resources/prometheus-rules.yaml and every rule in the chart’s own defaultRules uses one of those three, and none uses highest/high/medium/low/lowest. Without those three routes the five-level table would match nothing and every real alert would fall through to the catch-all, which is the highest-severity channel.

low and lowest are wired and waiting for a rule to use them. They are reachable by a test alert carrying that label, but no production rule currently does.

Two routing behaviours to know before changing anything

The catch-all is severity-highest, not null. An alert whose severity matches no child route is still a real alert, and the loudest available failure is one that reaches a human. Pointing the catch-all at null would re-create the silent-drop class of defect this routing exists to end.

The heartbeat drops sit above every delivery route. Alertmanager evaluates child routes in order and stops at the first match, so moving a delivery route above them would deliver Watchdog and every severity=none alert to a severity channel as though they were real alerts.

What is deliberately dropped

A sixth receiver, null, catches three noise sources and posts them nowhere:

The Watchdog heartbeat fires on its own repeat_interval, so if #alerts-highest-severity-level ever reports a Watchdog alert, the null routes have been reordered or removed.

How to check that a channel works

This is an operator procedure, not something a document can assert on your behalf. To check a channel, an operator should:

1. Port-forward the Alertmanager service and fire a smoke alert per severity with amtool alert add, the same way the source runbook’s step 5 does.
2. Open each target channel in Slack and confirm the message is there. Confirming the channel, not just the delivery, is the whole point: a single webhook posting every severity into one channel is precisely the failure this routing fixes, and only a person looking at Slack can see which channel a message actually landed in.
3. Confirm the drops still drop — no Watchdog in #alerts-highest-severity-level.
4. Remove the smoke alerts afterwards with amtool alert remove --alertmanager.url="$AM_URL" '*'.

If nothing arrives, the source runbook’s step 5 has the triage order: read the Alertmanager logs before assuming routing. kubectl -n platform-monitoring logs alertmanager-kube-prometheus-stack-alertmanager-0 -c alertmanager reports channel_not_found (wrong or renamed channel) separately from a delivery failure (egress or webhook rejected).

What a repository check cannot prove

Every cluster-side health signal stays green whether or not a message arrives: a Synced/Healthy Argo CD Application, a Ready=True ExternalSecret, a populated webhook file, and the CI gates that pin this mapping all stay green against a receiver that delivers nothing. That gap hid a silent-drop defect for a month.

The CI gates pin the mapping by asserting it pairwise (severity → receiver → channel), never as a presence check. That distinction matters: “five receivers exist and each names a real channel” still passes with the levels shuffled, and a lowest posting to the highest-severity channel is exactly the defect that would slip through. The two gates are scripts/ci/test-gitops-manifests.sh against the manifests and scripts/ci/test-alertmanager-severity-render.sh against the rendered chart output.

None of them observe Slack. Only the operator procedure above closes that gap.

The retired receiver name

Before this routing there was a single receiver named platform-alerts-slack, posting every alert to #platform-alerts. That channel never existed in the Slack workspace at all — Slack answers channel_not_found for it — so the old receiver could not have delivered even with a populated webhook URL. Both the receiver name and the channel name are gone from main.

If you find platform-alerts-slack or #platform-alerts in an old config, a stale commit, or a wiki page, you are reading pre-#173 state.

Not the same thing as the Grafana severity labels

Severity labels also appear in Grafana Alert Rules Reference. Those are Grafana-managed alert severities — Normal / High / Highest on the opensearch-hh-prod CPU and memory rules, Warning / Critical on the Sidekiq latency rules — and they are a different scheme from the Alertmanager bands documented here. Neither set corresponds to the other and nothing maps one onto the other: a Grafana High rule does not imply an Alertmanager high route, and a Grafana Highest rule does not imply anything lands in #alerts-highest-severity-level.

When this page is stale

The mapping lives in platform-infra, not here. Re-read the runbook on main if:

• a severity arrives in a channel whose name does not match the table above;
• a receiver is added, removed, renamed, or pointed at a different channel;
• a new rule starts using low or lowest, or a new severity label appears in resources/prometheus-rules.yaml;
• the platform team’s on-call rotation changes what “source of record” means.

Fix this page in the same PR that changes the runbook, or delete the table and leave only the pointer. A guide that silently forks the mapping goes stale exactly the way the code did.

Related

• Grafana Alert Rules Reference — Grafana-managed alert rules and their separate severity labels
• Infrastructure Repository Guide: hungryhub-terraform vs platform-infra
• Incident Drill Checklist — the drill that exercises on-call readiness for a different service
• hungryhub-terraform — Architecture and Operations