Serving API V2 Performance Runbook

Purpose

This runbook defines the repeatable functional, latency, and bounded-load validation for the v2 recommendation serving API. Run it after a staging deployment and before promoting an overhaul that changes recommendation serving, feature retrieval, ranking, caching, or model-loading behavior.

The benchmark is intentionally safe for shared staging by default: it runs functional smoke checks and sequential latency samples. Bounded concurrency only runs when --load-requests is explicitly provided.

Release Gates

The checked-in baseline is recsys/serving/baselines/api_v2.json.

Initial serving API gates:

• Health endpoint: p95 <= 100 ms, p99 <= 250 ms.
• Readiness endpoint: p95 <= 150 ms, p99 <= 300 ms.
• Guest homepage: p95 <= 300 ms, p99 <= 600 ms.
• Member homepage: p95 <= 500 ms, p99 <= 1000 ms.
• Tiger homepage decision: p95 <= 500 ms, p99 <= 1000 ms.
• Cohort feature API: p95 <= 800 ms, p99 <= 1200 ms.
• 20-item feature batch: p95 <= 250 ms, p99 <= 500 ms.
• Direct recommendation API: p95 <= 500 ms, p99 <= 1000 ms.
• Safe mixed load: p95 <= 2500 ms, p99 <= 4000 ms.

Tighten these gates only after collecting stable production-like baselines. Do not loosen them without a documented capacity review.

Required Staging Command

Run from a network location that can reach the staging service. Prefer in-cluster execution or a dedicated load-test runner over a local port-forward for latency evidence.

uv run python scripts/benchmark_serving_api_v2.py \
  --base-url http://<staging-hh-lion-service> \
  --member-user-id <known-featured-member-id> \
  --iterations 20 \
  --warmup-iterations 5 \
  --load-requests 50 \
  --load-concurrency 5 \
  --require-selected-items-replace \
  --fail-on-regression \
  --output text

Use local port-forward only for functional reachability checks:

kubectl --context do-sgp1-staging -n hh-lion port-forward svc/hh-lion 18080:80
uv run python scripts/benchmark_serving_api_v2.py \
  --base-url http://127.0.0.1:18080 \
  --iterations 10 \
  --warmup-iterations 2 \
  --output text

What The Benchmark Validates

• /health, /health/live, and /health/ready reachability.
• /v2/model/info reachability.
• Cohort feature lookup with production-like cohort parameters.
• Item batch lookup using restaurants returned by the cohort feature response.
• Single item feature lookup.
• Optional member feature lookup when --member-user-id is provided.
• Guest homepage response shape and latency.
• Optional member homepage response shape and latency.
• Homepage refresh polling for the guest request.
• Homepage decision endpoint using cohort-derived candidate IDs, including latency and bounded-load coverage.
• Selected-items homepage decision replacement from the Redis cohort candidate online store when --require-selected-items-replace is enabled.
• Direct recommendation endpoint.
• Optional bounded mixed load across homepage and cohort requests.

Operating Rules

• Do not run stress tests against shared staging without an agreed request rate, duration, rollback owner, and dashboard coverage.
• Treat 5xx responses and HTTP 429 responses as release-blocking during the benchmark.
• Treat missing cohort candidates as release-blocking because item feature and decision validation depend on production-like feature data.
• Capture the benchmark output, deployment commit SHA, feature-store validation job result, and Grafana or Prometheus snapshots in the release record.
• If local port-forward latency fails but in-cluster latency passes, use the in-cluster result as the release signal and keep the port-forward result as diagnostic evidence only.

Required Production Metrics

Use these queries after staging deploys and production canaries to verify the selected-items path is using the Redis online candidate store and staying within latency guardrails:

sum(rate(hh_lion_cohort_candidate_online_lookups_total[5m])) by (outcome)

sum(rate(hh_lion_cohort_candidate_online_rows_total[5m])) by (outcome)

sum(increase(hh_lion_homepage_decision_policy_events_total{personalization_policy="recommendation_selected_items",strategy="selected_items_feature_rerank",action="replace_items",fallback="false"}[30m]))

histogram_quantile(0.95, sum(rate(hh_lion_request_latency_seconds_bucket{endpoint="homepage_decision"}[5m])) by (le))

Failure Triage

1. Confirm Kubernetes readiness for hh-lion-api, hh-lion-personalization-worker, Redis, ClickHouse, and Feast registry access.
2. Check the latest hh-lion-feature-store-v2-validation job before debugging API behavior.
3. If feature endpoints fail, inspect Feature Store v2 freshness, online publication, and Redis connectivity first.
4. If selected-items replacement fails, inspect cohort candidate online-store lookup outcomes, Redis key population, and /v2/homepage/decide diagnostics before checking slower offline stores.
5. If homepage endpoints fail, inspect model/runtime initialization, feature adapter errors, GrowthBook status, queue health, and cache writes.
6. If only bounded load fails, inspect API replica count, worker count, Redis connections, ClickHouse connection pooling, CPU throttling, and pod memory pressure before changing code.