Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Canonical Ingestion Canary Dashboard Spec

This document defines the minimum dashboard panels and PromQL queries required to operate the Pegasus -> Tiger -> hh-lion canonical interaction-event canary safely in engineering and low-percentage production rollout.

Dashboard Scope

  • Time range defaults:
    • Primary: last 30 minutes
    • Incident view: last 6 hours
  • Refresh interval:
    • 30s during active canary
    • 60s outside rollout windows
  • Expected producer headers for this rollout:
    • producer_app=tiger
    • producer_client=pegasus
    • producer_version=canonical-v1

Required Panels

1) Accepted Events By Type

  • Query:
sum(increase(recsys_ingestion_events_total{status="accepted"}[10m])) by (event_type)
  • Guardrail: all expected funnel event types show non-zero counts during engineering validation windows.

2) Rejected Events By Type

  • Query:
sum(increase(recsys_ingestion_events_total{status="rejected"}[10m])) by (event_type)
  • Guardrail: should remain near zero after upstream payload fixes.

3) Validation Errors By Reason

  • Query:
sum(increase(recsys_ingestion_validation_errors_total[10m])) by (category, event_type)
  • Guardrail: no sustained growth for a single validation category during canary.

4) DLQ Volume

  • Query:
sum(increase(recsys_ingestion_dlq_events_total[10m])) by (error_type)
  • Guardrail: zero sustained DLQ growth for canonical producer traffic.

5) Missing Anonymous ID

  • Query:
sum(increase(recsys_ingestion_missing_fields_total{field_name="anonymous_id"}[10m])) by (event_type)
  • Guardrail: zero for all canary traffic.

6) Missing Request ID

  • Query:
sum(increase(recsys_ingestion_missing_fields_total{field_name="request_id"}[10m])) by (event_type)
  • Guardrail: zero for search, impression, click, and view.

7) Missing Package ID

  • Query:
sum(increase(recsys_ingestion_missing_fields_total{field_name="package_id"}[10m])) by (event_type)
  • Guardrail: zero for package_selected, begin_checkout, and booking_confirmed.

8) Producer Header Coverage

  • Query:
sum(increase(recsys_ingestion_producer_events_total[10m])) by (producer_app, producer_client, producer_version, event_type)
  • Guardrail: canonical canary events appear under tiger / pegasus / canonical-v1.

9) Event Lag P95

  • Query:
histogram_quantile(
  0.95,
  sum(rate(recsys_ingestion_event_lag_seconds_bucket[10m])) by (le, event_type)
)
  • Warning threshold: investigate sustained p95 lag above 60s for any active canary event type.

Engineering Validation Checklist

Before production canary, confirm all of the following in engineering:

  1. search, impression, click, view, favorite_added, package_selected, begin_checkout, and booking_confirmed all appear in accepted-event panels.
  2. Rejected-event and DLQ panels remain at or near zero during representative guest and member flows.
  3. Missing-field panels stay at zero for anonymous_id, request_id, and package_id.
  4. Producer-header panel shows tiger / pegasus / canonical-v1.
  5. Event lag remains operationally acceptable for the expected engineering load.

Production Canary Checklist

At each rollout step (for example 1%, 5%, 25%, 50%, 100%), verify:

  1. Accepted event counts scale with traffic for all active funnel steps.
  2. Rejection reasons stay flat or trend to zero after upstream fixes.
  3. DLQ volume does not rise materially.
  4. Missing-field counts remain zero.
  5. Producer headers remain stable and attributable to the Tiger canonical path.