Feature Store V2 Performance Runbook

Purpose

This runbook defines the required performance validation for the v2 Feast feature store before engineering promotion or production cutover.

Release Gates

• Online member lookup: p95 <= 20 ms, p99 <= 50 ms.
• Online cohort lookup: p95 <= 20 ms, p99 <= 50 ms.
• Online 100-item lookup: p95 <= 75 ms, p99 <= 150 ms.
• Homepage feature assembly: p95 <= 100 ms, p99 <= 250 ms.
• Historical retrieval: at least 50,000 entity rows per minute.
• Online publication: at least 10,000 rows per minute per worker.
• Redis evictions must remain zero during load tests.
• Redis CPU, memory, and connection count must remain below 70%, 75%, and 70% of configured limits.
• ClickHouse query plans must prune on feature_timestamp partitions for historical retrieval.

Required Commands

Run the benchmark smoke test locally:

uv run pytest tests/performance -m performance -q
uv run python scripts/benchmark_feature_store_v2.py --fail-on-regression --output text

Run capacity planning with engineering or production-like measured inputs:

uv run python scripts/capacity_plan_feature_store_v2.py \
  --active-members <members> \
  --active-items <items> \
  --active-cohorts <cohorts> \
  --member-payload-bytes <bytes> \
  --item-payload-bytes <bytes> \
  --cohort-payload-bytes <bytes> \
  --daily-member-rows <rows> \
  --daily-item-popularity-rows <rows> \
  --daily-item-metadata-rows <rows> \
  --daily-cohort-rows <rows> \
  --retention-days 730 \
  --output text

Engineering Validation

1. Run v2 integration tests with live Feast, ClickHouse, and Redis.
2. Run online lookup benchmarks for member, cohort, and 100-item item lookups.
3. Run historical retrieval against at least 50,000 entity rows.
4. Run publication throughput against each v2 feature set.
5. Capture Redis memory, CPU, connection count, and evictions during the test window.
6. Capture ClickHouse EXPLAIN indexes=1 output for historical retrieval queries and verify feature_timestamp partition pruning.
7. Compare p95, p99, throughput, payload size, and memory against recsys/feature_store/baselines/feature_store_v2.json.

Do not cut over v2 serving if any release gate fails. Update the checked-in baseline only after a documented engineering benchmark run with production-like data volume.