Canary Deployment and Verification

Overview

hh-server production builds use canary deployment pattern: deploy to a low-traffic namespace first, verify health + Passenger metrics, then roll out to all remaining namespaces. This limits blast radius — a broken image only affects hh-vendor-private before it reaches high-traffic namespaces.

Two scripts enforce safety at each stage:

Buildspec Flow (buildspec-prod.yaml)

pre_build
  └─ ECR login, pull :latest

build
  └─ Docker build + push artifacts to S3

post_build
  └─ ECR push (tagged + :latest)
  └─ DB migration + Karafka topic setup
  └─ [CANARY]   hh-vendor-private
  │              safe-rollout.sh (300s)
  │              verify-canary.sh || exit 1   ← HARD GATE
  └─ [MAIN]     hungryhub
  │              safe-rollout.sh × 5 deployments
  │              verify-canary.sh || true     ← SOFT GATE
  └─ [REMAINING] 7 namespaces, each:
                   kustomize + apply
                   safe-rollout.sh
                   verify-canary.sh || exit 1 ← HARD GATE

Namespace Deployment Order

Why hungryhub is a soft gate

hungryhub is the legacy namespace. Verification uses || true because:

• It is being phased out (see Infrastructure and Manifest Management Documentation)
• Blocking on it would delay all other namespace deployments
• It still runs sidekiq workers that need image updates

safe-rollout.sh

Waits for Kubernetes rollout with automatic rollback.

./manifest/scripts/safe-rollout.sh -d <deployment> -n <namespace> -t <timeout>

Behavior:

1. kubectl rollout status deployment/<name> -n <ns> --timeout=<timeout>
2. On success → exit 0
3. On timeout/failure → kubectl rollout undo deployment/<name> -n <ns> → wait 30s → exit 1

What it catches: Image pull errors, CrashLoopBackOff, liveness probe failures, resource exhaustion.

verify-canary.sh

After safe-rollout.sh confirms rollout, verify-canary.sh runs 5 health checks against live traffic.

./manifest/scripts/verify-canary.sh -n <namespace> -w 60 -p http://prometheus-server.monitoring.svc.cluster.local:9090

Step 0: Traffic Warmup Wait

Waits -w seconds (default 60) before checking metrics. This gives the load balancer time to route traffic to the new pods so Prometheus metrics have data.

Step 1: Pod Readiness Check

Polls pods every 5s for up to 300s.

Steps 2–5: Prometheus Passenger Metrics

Scope: Steps 2–5 only run for namespaces ending in -public. Private namespaces and the hungryhub namespace skip Prometheus checks (passenger-go-exporter is not deployed there).

Prerequisites: PROMETHEUS_URL must be set and jq must be installed.

Step 2: Request Queue Size

max(passenger_go_wait_list_size{namespace="<namespace>"})

Step 3: Passenger Process Count

min(passenger_go_process_count{namespace="<namespace>"})

Step 4: Worker Memory Usage

max(passenger_go_process_real_memory_bytes{namespace="<namespace>"})

Step 5: Request Throughput

sum(rate(passenger_go_process_processed{namespace="<namespace>"}[2m]))

Throughput has a retry loop: if below threshold, waits 10s and rechecks up to 3 times total. This accounts for cold-start latency where traffic hasn’t reached the new pods yet.

Prometheus Connectivity Fallback

If direct HTTP query returns 000 (connection refused/timeout):

1. Parse hostname from PROMETHEUS_URL (e.g. prometheus-server.monitoring.svc.cluster.local)
2. kubectl port-forward svc/prometheus-server -n monitoring 19090:9090
3. Retry query via http://127.0.0.1:19090
4. If port-forward also fails → skip metric checks with WARN (does not fail the build)

Port-forward process is cleaned up on script exit via trap cleanup EXIT.

Summary Output

=== Canary verification PASSED for namespace 'hh-vendor-private' — proceeding with full rollout ===

or

=== Canary verification FAILED for namespace 'hh-vendor-private' ===
    To rollback: kubectl rollout undo deployment/<deployment-name> -n hh-vendor-private
    List deployments: kubectl get deployments -n hh-vendor-private

Environment Variables

All configurable via environment or CLI flags. CLI flags (-n, -w, -p) take precedence.

Running Manually

Useful for debugging a failed canary or dry-running checks without the full build pipeline.

# 1. Deploy to canary namespace
kubectl kustomize manifest/overlays/prod/hh-vendor-private > /tmp/canary.yaml
kubectl apply -f /tmp/canary.yaml -n hh-vendor-private

# 2. Wait for rollout
./manifest/scripts/safe-rollout.sh -d sidekiq-google-reserve-lp -n hh-vendor-private -t 300s

# 3. Run canary verification
./manifest/scripts/verify-canary.sh \
  -n hh-vendor-private \
  -w 60 \
  -p http://prometheus-server.monitoring.svc.cluster.local:9090

# 4. If verification fails — rollback
kubectl rollout undo deployment/sidekiq-google-reserve-lp -n hh-vendor-private

Dry-run Prometheus checks locally

# Port-forward to Prometheus first
kubectl port-forward svc/prometheus-server -n monitoring 19090:9090 &

# Query Passenger metrics directly
curl -s 'http://127.0.0.1:19090/api/v1/query' \
  --data-urlencode 'query=max(passenger_go_wait_list_size{namespace="hh-syn-public"})' | jq '.data.result'

curl -s 'http://127.0.0.1:19090/api/v1/query' \
  --data-urlencode 'query=sum(rate(passenger_go_process_processed{namespace="hh-syn-public"}[2m]))' | jq '.data.result'

Troubleshooting

Canary verification FAILS on step 1 (pod readiness)

# Check pod status and events
kubectl get pods -n hh-vendor-private
kubectl describe pods -n hh-vendor-private -l app=<deployment>
kubectl logs -n hh-vendor-private -l app=<deployment> --tail=50

# Common causes
# - Missing env var / secret: check AWS SSM parameter path
# - Image pull error: verify ECR repo and image tag
# - CrashLoopBackOff: check application logs for startup errors

Canary verification FAILS on step 2 (queue size)

# Check what's actually in the queue
curl -s 'http://127.0.0.1:19090/api/v1/query' \
  --data-urlencode 'query=passenger_go_wait_list_size{namespace="hh-syn-public"}' | jq

# Likely causes
# - Traffic spike during deploy: increase CANARY_WAIT to let traffic settle
# - Slow queries in new code: check APM traces

Canary verification FAILS on step 5 (throughput)

# Check if traffic is reaching the pods at all
curl -s 'http://127.0.0.1:19090/api/v1/query' \
  --data-urlencode 'query=sum(rate(passenger_go_process_processed{namespace="hh-syn-public"}[2m]))' | jq

# Check if endpoints are registered with the service
kubectl get endpoints -n hh-syn-public

# Likely causes
# - No ingress/LB pointing to new pods: verify Service and Ingress resources
# - Namespace scaled to zero (queue-based scaling): expected for -private namespaces
# - Cold start: wait and retry (throughput check has built-in retry)

Prometheus metrics show WARN (skipped) instead of PASS/FAIL

# Check if namespace ends in -public
# Only -public namespaces get Prometheus checks

# Check if jq is installed
which jq || apt-get install -y jq

# Check if PROMETHEUS_URL is set
echo $PROMETHEUS_URL

safe-rollout.sh FAILS (auto-rollback triggered)

# Check what happened to the deployment
kubectl rollout history deployment/<deployment> -n <namespace>

# Re-deploy if needed
kubectl apply -f <manifest> -n <namespace>