Feature Store V2 Operations Runbook

Purpose

This runbook defines the production operating model for the v2 Feast feature store. The v2 serving path must not receive traffic until database migrations, Feast registry application, historical feature validation, online publication, performance gates, capacity checks, recommendation-quality gates, and Kubernetes readiness all pass.

Deployment Order

Production and engineering deployments must follow this order:

1. Apply ClickHouse migrations, including 020_create_feature_store_v2.sql.
2. Apply Feast registry definitions from /app/feature_repo.
3. Validate v2 historical feature tables with scripts.validate_feature_store_v2.
4. Validate performance and capacity budgets with scripts/benchmark_feature_store_v2.py and scripts/capacity_plan_feature_store_v2.py.
5. Validate recommendation quality with scripts/evaluate_recommendation_quality_v2.py.
6. Apply serving manifests.
7. Wait for Kubernetes readiness on hh-lion-api and hh-lion-personalization-worker.

All deployments enforce the v2 gate stack. Do not deploy until the v2 backfill and publication gates are already passing in the target environment.

Required Deployment Variables

For every deployment environment, configure:

• FEATURE_STORE_V2_ACTIVE_MEMBERS
• FEATURE_STORE_V2_ACTIVE_ITEMS
• FEATURE_STORE_V2_ACTIVE_COHORTS
• FEATURE_STORE_V2_MEMBER_PAYLOAD_BYTES
• FEATURE_STORE_V2_ITEM_PAYLOAD_BYTES
• FEATURE_STORE_V2_COHORT_PAYLOAD_BYTES
• FEATURE_STORE_V2_DAILY_MEMBER_ROWS
• FEATURE_STORE_V2_DAILY_ITEM_POPULARITY_ROWS
• FEATURE_STORE_V2_DAILY_ITEM_METADATA_ROWS
• FEATURE_STORE_V2_DAILY_COHORT_ROWS
• FEATURE_STORE_V2_RETENTION_DAYS, defaulting to 730 when omitted

The recommendation-quality artifact must exist at:

artifacts/recommendation_quality/time/latest_vs_production.json

That artifact must compare the candidate model or policy against the current production baseline on a time-based holdout.

Feature Validation

The in-cluster validation job is:

k8s/shared/feature-store-v2-validation-job.yaml

It runs:

python -m scripts.validate_feature_store_v2 --lookback-days 3 --output text

The job reads lion-config and lion-secrets, so it uses the same ClickHouse and Feast registry configuration as serving. It fails the rollout when required fields are null, duplicate feature rows exist, feature timestamps drift, source watermarks lag, or freshness SLAs are breached.

When cohort candidate online serving is enabled, the validation job must also probe representative Redis candidate-set keys:

python -m scripts.validate_feature_store_v2 \
  --feature-set cohort_popularity_features_v2 \
  --validate-candidate-online-store \
  --candidate-online-probe-key <known-cohort-key> \
  --output text

The probe fails closed when no representative online candidate sets are available. Do not replace this gate with a request-time ClickHouse fallback.

Online Publication

Engineering bootstrap feature jobs publish both historical rows and online Redis values:

• scripts.compute_member_features --mode full --publish-online
• scripts.compute_item_features --mode full --publish-online --with-opensearch-metadata
• scripts.compute_cohort_features --mode full --publish-online

Online publication must succeed before validation. Failed publication is treated as a failed bootstrap and must not be masked by direct ClickHouse fallback.

The cohort feature job also publishes hh-lion cohort candidate sets to Redis for selected-items decision augmentation. Monitor publication and lookup health with:

sum(rate(hh_lion_cohort_candidate_online_lookups_total[5m])) by (outcome)

sum(rate(hh_lion_cohort_candidate_online_rows_total[5m])) by (outcome)

Serving Safety

Serving manifests set:

PRODUCTION_SAFE_MODE=true

In engineering and production, readiness fails closed when required v2 Feature Store components are unhealthy. Serving feature lookups must use the Feast v2 online path; request-time direct ClickHouse feature fallback is not part of normal serving.

Failure Handling

Registry errors:

• Check FEAST_REGISTRY_PATH, FEAST_REGISTRY_READ_PATH, and registry database reachability.
• Re-run the Feast registry job only after the database migration job has succeeded.
• Do not roll serving pods when the registry job fails.

Stale features:

• Inspect the feature_compute_runs table for the failed feature set.
• Re-run the relevant feature job with online publication enabled.
• Re-run feature-store-v2-validation-job.yaml.

Publication failures:

• Treat Redis publication failures as release-blocking.
• Confirm Redis connectivity, Feast online store configuration, and push-source names.
• Confirm cohort candidate online-store publication and representative probe keys before serving selected-items replacement traffic.
• Do not mask Redis or Feast online publication failures with request-time direct ClickHouse feature reads.

Data quality failures:

• Do not patch validation thresholds to pass a rollout.
• Identify the upstream compute or source-data issue and rerun the deterministic feature job.
• Re-run validation and keep the failed report attached to the incident or release record.

Recommendation-quality failures:

• Block rollout when recall@100, ndcg@10, map@10, or mrr@10 regresses versus production.
• Block rollout when duplicate, invalid, unavailable, or untraceable recommendation exposure is greater than zero.
• Review segment-level regressions before ramping any traffic.