Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Canary Dashboard Spec

This document defines the minimum dashboard panels and PromQL queries required to operate canary rollout safely for /v2/homepage and /v2/homepage/refresh.

Provisioned template:

  • k8s/grafana/recsys-canary-dashboard.json

Dashboard Scope

  • Time range defaults:
    • Primary: last 30 minutes
    • Incident view: last 6 hours
  • Refresh interval:
    • 30s during active canary
    • 60s outside rollout windows

Required Panels

1) Homepage P95 Latency

  • Query:
histogram_quantile(
  0.95,
  sum(rate(hh_lion_request_latency_seconds_bucket{endpoint="homepage"}[5m])) by (le)
)
  • Guardrail: <= 1.0s (auto-rollback threshold).
  • Warning threshold: > 0.75s sustained for 10m.

2) Homepage Fallback Ratio

  • Query:
sum(increase(hh_lion_homepage_decision_events_total{fallback="true"}[10m]))
/
clamp_min(sum(increase(hh_lion_homepage_decision_events_total[10m])), 1)
  • Guardrail: <= 0.10 (auto-rollback threshold).
  • Warning threshold: > 0.05 sustained for 10m.

3) Refresh Timeout-Terminalized Ratio

  • Query:
sum(increase(hh_lion_refresh_outcome_events_total{outcome="timeout_terminalized"}[10m]))
/
clamp_min(sum(increase(hh_lion_refresh_outcome_events_total[10m])), 1)
  • Guardrail: <= 0.02 (auto-rollback threshold).

4) Member Feature Availability (Min Over 10m)

  • Query:
min_over_time(hh_lion_feature_availability_ratio{feature_type="member"}[10m])
  • Guardrail: >= 0.85 (auto-rollback threshold).
  • Warning threshold: < 0.90 sustained for 10m.

5) Ranker Decision Failure Ratio

  • Query:
sum(increase(hh_lion_ranker_decision_events_total{outcome=~"primary_failed|fallback_failed"}[10m]))
/
clamp_min(sum(increase(hh_lion_ranker_decision_events_total[10m])), 1)
  • Warning threshold: > 0.05 sustained for 10m.

6) Personalization Queue Depth (Max Over 10m)

  • Query:
max_over_time(hh_lion_personalization_queue_depth[10m])
  • Warning threshold: > 100.

7) Pending Refresh Age (Max Over 10m)

  • Query:
max_over_time(hh_lion_personalization_pending_age_seconds[10m])
  • Warning threshold: > 45s.

8) Worker Failure Ratio

  • Query:
sum(rate(hh_lion_personalization_worker_events_total{status="failed"}[10m]))
/
clamp_min(sum(rate(hh_lion_personalization_worker_events_total{status="started"}[10m])), 0.001)
  • Critical threshold: > 0.10 sustained for 10m.

9) ANN Index Age P95

  • Query:
histogram_quantile(0.95, sum(rate(hh_lion_faiss_index_age_hours_bucket[15m])) by (le))
  • Warning threshold: > 24h sustained for 15m.

10) Homepage Decision P95 Latency

  • Query:
histogram_quantile(
  0.95,
  sum(rate(hh_lion_request_latency_seconds_bucket{endpoint="homepage_decision"}[5m])) by (le)
)
  • Guardrail: <= 0.25s (auto-rollback threshold).

11) Homepage Decision Fallback Ratio

  • Query:
sum(increase(hh_lion_homepage_decision_endpoint_events_total{fallback="true"}[10m]))
/
clamp_min(sum(increase(hh_lion_homepage_decision_endpoint_events_total[10m])), 1)
  • Guardrail: <= 0.02 (auto-rollback threshold).

12) Homepage Decision Outcomes

  • Query:
sum by (strategy, action, fallback) (increase(hh_lion_homepage_decision_endpoint_events_total[10m]))
  • Diagnostic panel only; no auto-rollback threshold.

13) Homepage Decision Section P95 Latency by Strategy/Action

  • Query:
histogram_quantile(
  0.95,
  sum by (le, strategy, action, fallback) (
    rate(hh_lion_homepage_decision_section_latency_seconds_bucket[5m])
  )
)
  • Diagnostic panel only; use it to identify whether the slow tail comes from specific strategy/action paths.

14) Homepage Decision P95 Latency by Pod

  • Query:
histogram_quantile(
  0.95,
  sum by (le, pod) (
    rate(hh_lion_request_latency_seconds_bucket{endpoint="homepage_decision"}[5m])
  )
)
  • Diagnostic panel only; use it to identify pod-local latency drift after deploys or restarts.

15) Homepage Decision RPS by Pod

  • Query:
sum by (pod) (
  rate(hh_lion_request_latency_seconds_bucket{
    endpoint="homepage_decision",
    le="+Inf"
  }[5m])
)
  • Diagnostic panel only; use it to identify load imbalance before comparing pod p95 latency.

16) API Pod Memory Working Set

  • Query:
sum by (pod) (
  container_memory_working_set_bytes{
    namespace="hh-lion",
    pod=~"hh-lion-api-.*",
    container="lion-api"
  }
)
  • Diagnostic panel only; use it to identify pod-local RSS or working-set drift.

17) API Process CPU by Pod

  • Query:
sum by (pod) (
  rate(process_cpu_seconds_total{
    namespace="hh-lion",
    pod=~"hh-lion-api-.*"
  }[5m])
)
  • Diagnostic panel only; use it to identify pod-local CPU drift as a latency-tail contributor.

18) Homepage Decision Candidate Pool by Source (10m)

  • Query:
sum by (section_id, source) (
  increase(hh_lion_homepage_decision_candidate_pool_total[10m])
)
  • Diagnostic panel only; use it to identify candidate-source mix changes.

19) Homepage Decision Result Semantics (10m)

  • Query:
sum by (strategy, action, fallback_reason, introduced_new_item_ids, same_set) (
  increase(hh_lion_homepage_decision_result_events_total[10m])
)
  • Diagnostic panel only; use it to confirm quality-impacting decision outcomes, such as repeated same-set results or fallback reasons.

20) Cohort Candidate Online Store Lookups

  • Query:
sum(rate(hh_lion_cohort_candidate_online_lookups_total[5m])) by (outcome)
  • Guardrail: lookup errors must remain zero during canary.
  • Diagnostic: hit traffic should be nonzero when selected-items replacement smoke traffic is running.

21) Cohort Candidate Online Store Rows

  • Query:
sum(rate(hh_lion_cohort_candidate_online_rows_total[5m])) by (outcome)
  • Diagnostic panel only; use it to confirm online-store hits return candidate rows.

22) Selected-Items Replacement Success (30m)

  • Query:
sum(increase(hh_lion_homepage_decision_policy_events_total{personalization_policy="recommendation_selected_items",strategy="selected_items_feature_rerank",action="replace_items",fallback="false"}[30m]))
  • Guardrail: must be greater than zero after the selected-items smoke test.

Rollout Stage Checklist

At each rollout step (5%, 25%, 50%, 100%), verify:

  1. No active critical auto-rollback alerts.
  2. Homepage p95 latency and fallback ratio remain within guardrails.
  3. Homepage decision p95 latency and fallback ratio remain within guardrails.
  4. Refresh timeout-terminalized ratio remains below 2%.
  5. Feature availability is stable and above 85%.
  6. Pod-level homepage decision latency, RPS, memory, and process CPU do not show pod-local drift.
  7. Queue lag and worker failure metrics are stable.
  8. Candidate-source mix and result semantics do not show unexpected fallback or same-set regressions.
  9. Cohort candidate online-store lookup errors are zero and selected-items replacement smoke traffic is visible.

Alert-to-Runbook Mapping

  • Model/serving degradation: model_rollback_runbook.md
  • Queue/pending degradation: personalization_queue_lag_runbook.md
  • ANN staleness: ann_index_staleness_runbook.md