Operations & SRE Runbook

Goal of this chapter: A technical guide for SREs, DevOps engineers, and on-call developers to operate, troubleshoot, and maintain the Restaurant Prioritization pipeline and Streamlit service.

1. Operational Cadence & Schedule

The pipeline runs on two automated schedules:

2. Infrastructure & Compute Environment

The batch pipeline runs via GitHub Actions and Kubernetes:

  ┌────────────────────────────────────────────────────────┐
  │ GitHub Actions Workflow (.github/workflows/pipeline.yml)│
  └───────────────────────────┬────────────────────────────┘
                              │
                              ▼
  ┌────────────────────────────────────────────────────────┐
  │ Dedicated Self-Hosted Arc Runner:                      │
  │ arc-runner-set-restaurant-prioritization-cpu2          │
  │ • CPU: 2 vCPU                                          │
  │ • Memory: 4 GB RAM                                     │
  │ • Network: WireGuard VPN tunnel to ClickHouse Cluster  │
  └───────────────────────────┬────────────────────────────┘
                              │
                              ▼
  ┌────────────────────────────────────────────────────────┐
  │ Kubernetes Production Cluster:                         │
  │ • ghcr.io/hungryhub-team/restaurant-prioritization:app │
  │ • CronJob Deadline: activeDeadlineSeconds = 720 (12m) │
  │ • Resource Limits: CPU 2000m, Memory 4096Mi            │
  └────────────────────────────────────────────────────────┘

Important

Dedicated Runner Isolation:
The pipeline must never run on public GitHub-hosted runners because it requires a secure WireGuard VPN connection to reach the private ClickHouse cluster (10.x.x.x). Always use arc-runner-set-restaurant-prioritization-cpu2.

3. The 10 Automated Pipeline Health Checks

Before swapping fresh rankings into the production serving table (hh_power_bi.combined_rankings), the automated pipeline executes 10 mandatory sanity checks:

1. Row Count Check: Total rows in combined_rankings_new must be $\ge 7,000$ (guards against partial ingestion).
2. Bookable Restaurant Floor: Must contain $\ge 2,500$ active bookable restaurants.
3. No Null IDs: restaurant_id column must have 0% null values.
4. Duplicate Primary Key: restaurant_id must be 100% unique per execution week.
5. Priority Score Range: All computed priority scores must lie between $0.0$ and $1.0$.
6. Realistic GMV Values: Average predicted weekly GMV must fall between ฿200,000 and ฿800,000 (catches the legacy 10x inflation bug).
7. Segment Completeness: Every restaurant must be assigned to one of the 4 quadrants (no unclassified or null segments).
8. Date Alignment: Ingested bookings must match created_at timestamps up to the prior Sunday 23:59:59.
9. Zero Negative Spend: All ad spend and KOL costs must be $\ge 0$.
10. Memory Ceiling: Pipeline execution must stay within 3.2 GB RAM (below the 4 GB pod limit).

If any of these 10 checks fail, the pipeline halts immediately, sends a high-priority alert to #alerts-data-team on Slack, and does NOT execute the table swap. Production users continue seeing the previous week’s clean data.

4. Troubleshooting Common Failures

Issue A: Pipeline Exceeds 12-Minute Timeout (DeadlineExceeded)

• Symptoms: GitHub Actions job fails with activeDeadlineSeconds exceeded.
• Probable Causes: ClickHouse slow query due to heavy concurrent ETL, or network latency over the WireGuard VPN tunnel.
• Remedy:
  1. Check ClickHouse query log:

     SELECT query_id, user, elapsed, query 
     FROM system.processes 
     WHERE current_database = 'raw_data_staging';
     

  2. If an ETL query is blocked by table locks, kill the hung query or re-trigger the pipeline manually via GitHub Actions (workflow_dispatch).

Issue B: VPN Tunnel Disconnected (ConnectionRefused: ClickHouse)

• Symptoms: Pipeline fails at Step 1 (Connecting to ClickHouse).
• Probable Causes: The WireGuard interface on the Arc runner pod dropped its handshake.
• Remedy:
  1. Restart the runner deployment:

     kubectl rollout restart deployment arc-runner-set-restaurant-prioritization-cpu2 -n runners
     

  2. Verify handshake with wg show.

5. Emergency Rollback Procedure

If flawed data somehow bypasses checks and reaches the dashboard:

# 1. SSH / connect to ClickHouse client
clickhouse-client --user=<USER> --password=<PASSWORD>

# 2. Swap back to the backup table (atomic 1ms swap)
EXCHANGE TABLES hh_power_bi.combined_rankings AND hh_power_bi.combined_rankings_backup;

# 3. Clear the Streamlit server cache
curl -X POST http://restaurant-prioritization.internal.hungryhub.com/api/cache/clear

6. Maintenance & Model Re-calibration

• Quarterly Drift Audit: Compare predicted GMV vs actual realized GMV. If Mean Absolute Percentage Error (MAPE) drifts higher than 25%, schedule a feature review.
• Feature Additions: When adding new columns, update both src/features/ in the code repository and document them in 04. Machine Learning Pipeline.