Grafana Alert Rules Reference
Source: Grafana Alerting Last updated: 2026-07-20 Total: 42 alert rules (all normal)
k8s-pod-dashboard
Group: k8s-pod-dashboard
Rules: 4
Folder: Grafana-managed
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| Current hh-server Replicas in end-user | Normal | hungryhub-server pod is currently {{ $values.B.Value }} replica |
| Current hh-server Replicas in syn | Normal | hungryhub-server pod is currently {{ $values.B.Value }} replica |
| Current hh-server Replicas in vendor | Normal | hungryhub-server pod is currently {{ $values.B.Value }} replica |
| Current hh-server Replicas in cosmos | Normal | hungryhub-server pod is currently {{ $values.B.Value }} replica |
Monitors current replica count for hh-server across all 4 namespaces: end-user, syn, vendor, cosmos.
Kubernetes Cluster
Group: Kubernetes Cluster
Rules: 2
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| Cluster CPU usage | Normal | Kubernetes cluster CPU usage currently is {{ $values.B.Value }} |
| Cluster Pod Restart | Paused | Pod {{ $labels.pod }} has restarted {{ $values.A.Value }} times in the last 10 minutes |
Monitors overall cluster health — CPU utilization and pod restart counts.
opensearch-hh-prod — CPU Utilization
Group: opensearch-hh-prod
Rules: 12 (4 severity levels × 3 metric types)
Evaluate every: 1m
4 CPU metric types tracked:
- System — CPU system utilization
- IOwait — CPU I/O wait time
- IRQ — CPU interrupt request utilization
- User — CPU user-space utilization
3 severity levels per metric:
| Alert | State | Severity | Summary |
|---|---|---|---|
| CPU System/Normal/High/Highest Utilization * | Normal | Normal / High / Highest | [Severity] CPU {type} usage has reached {{ printf "%.2f" $values.B.Value }}% on host {{ $labels.host }} |
Applied to the Aiven OpenSearch opensearch-hh-prod nodes.
opensearch-hh-prod — Memory Usage
Group: opensearch-hh-prod
Rules: 3
Evaluate every: 1m
| Alert | State | Severity | Summary |
|---|---|---|---|
| Memory Usage Normal | Normal | Normal | [Normal Alert] Memory usage has reached {{ printf "%.2f" $values.B.Value }}% on host {{ $labels.host }} |
| Memory Usage High | Normal | High | [High Alert] Memory usage has reached {{ printf "%.2f" $values.B.Value }}% on host {{ $labels.host }} |
| Memory Usage Highest | Normal | Highest | [Highest Alert] Memory usage has reached {{ printf "%.2f" $values.B.Value }}% on host {{ $labels.host }} |
Services Metric — health-check-status
Group: Services Metric
Rules: 1
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| Public Services | Paused | 🚨 PUBLIC ENDPOINT DOWN — Service: {{ .Labels.name }}. Impact: Endpoint is unreachable. Action: Immediate investigation required. Status Page: https://internal-status.hungryhub.com/ |
Note: Currently paused. Monitors public-facing service endpoints for availability.
Services Metric — hh-menu
Group: Services Metric
Rules: 2
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| hh-menu High CPU Usage Alert | Paused | hh-menu deployment has reached its scaling limit (5 replicas), but CPU usage (>500m) remains high |
| hh-menu Backlog Alert | Paused | Puma request backlog detected on {{ $labels.namespace }} / {{ $labels.pod }} |
Note: Both currently paused. First alert triggers when CPU exceeds 500m at max replicas (5). Second monitors Puma backlog.
Services Metric — imgproxy
Group: Services Metric
Rules: 1
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| imgproxy Timeout Errors | Paused | imgproxy is returning timeout errors |
Note: Currently paused. Alerts when imgproxy image-processing service returns timeout errors.
Services Metric — Kafka
Group: Services Metric
Rules: 3
Evaluate every: 1m
| Alert | State | Summary |
|---|---|---|
| Kafka Lag Alert | Paused | High Kafka Consumer Lag detected on topic {{ $labels.topic }}. Current lag is {{ $values.B.Value }} messages, above threshold of 1000 |
| URP Alert | Paused | One or more Kafka partitions are under-replicated. Replicas not in sync with leader — risk of data loss if broker fails. Current value: {{ $value }} |
| Offline Partitions Alert | Paused | One or more Kafka partitions have no active leader. Producers and consumers cannot read/write. Current value: {{ $value }} |
Note: All paused. Covers consumer lag, under-replicated partitions (URP), and offline partitions.
Sidekiq Monitor — Latency
Group: Sidekiq Monitor
Rules: 14 (7 queues × 2 severity levels)
Evaluate every: 1m
7 Sidekiq queues monitored:
| Queue | Severity | Alert Name |
|---|---|---|
| Kafka Producer | Warning | Kafka Producer - Warning |
| Kafka Producer | Critical | Kafka Producer - Critical |
| Kafka Inventory Producer | Warning | Kafka Inventory Producer - Warning |
| Kafka Inventory Producer | Critical | Kafka Inventory Producer - Critical |
| Long Process | Warning | Long Process - Warning |
| Long Process | Critical | Long Process - Critical |
| Critical | Warning | Critical - Warning |
| Critical | Critical | Critical - Critical |
| Default | Warning | Default - Warning |
| Default | Critical | Default - Critical |
| Loyalty Program | Warning | Loyalty Program - Warning |
| Loyalty Program | Critical | Loyalty Program - Critical |
| Inventory | Warning | Inventory - Warning |
| Inventory | Critical | Inventory - Critical |
Summary: 🔥 Current Sidekiq latency is **{{ $values.A.Value }}s** – above threshold!
Note: All currently paused. Warning/Critical thresholds vary per queue. Default threshold applies when no queue-specific threshold is configured.
Legend
- Normal — Alert rule is healthy, no firing condition met
- Paused — Rule evaluation suspended (manual or scheduled pause)
- Firing — Condition met, alert actively firing (not present at time of capture)
- Pending — Condition met, waiting for duration/frequency before firing