Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Feature Store V2 Backfill Runbook

Purpose

The v2 feature store uses ClickHouse historical tables as the source of truth for offline retrieval and Redis as the latest online serving copy. Backfills must create point-in-time correct historical rows and publish the validated latest rows online.

Scope

Backfill the most recent 730 days, or all available source history when less than 730 days exists. Keep the retention window aligned with ClickHouse table TTLs and model-training lookback windows.

Preconditions

  • ClickHouse migration 020_create_feature_store_v2.sql has succeeded.
  • Feast registry definitions from /app/feature_repo have been applied.
  • FEAST_REGISTRY_PATH and FEAST_REGISTRY_READ_PATH are configured.
  • Redis and ClickHouse are reachable from bootstrap jobs.
  • Source events have completed ingestion for the target backfill window.

Engineering Backfill

Run the engineering bootstrap workflow with:

  • recompute_features=true
  • cleanup_existing_data=false unless intentionally rebuilding the environment
  • run_training=false for a feature-only validation pass
  • register_model=false
  • update_engineering_run_id=false

The workflow runs member, item, and cohort feature jobs with --mode full --publish-online, then runs the v2 validation job. For a replayable backfill, patch the job command to include an explicit UTC --as-of and matching --source-watermark.

Production Backfill

For production, run feature refresh jobs independently of serving rollout until validation passes:

kubectl apply -n hh-lion -f k8s/shared/bootstrap-feast-repo-job.yaml
kubectl apply -n hh-lion -f k8s/shared/bootstrap-member-features-job.yaml
kubectl apply -n hh-lion -f k8s/shared/bootstrap-item-features-job.yaml
kubectl apply -n hh-lion -f k8s/shared/bootstrap-cohort-features-job.yaml
kubectl apply -n hh-lion -f k8s/shared/feature-store-v2-validation-job.yaml

Patch each job image to the exact release image before applying it. Do not reuse a stale latest tag for production backfills.

Validation Gates

Backfill is not complete until all gates pass:

  • Historical feature validation: scripts.validate_feature_store_v2 --lookback-days 3.
  • Online publication audit: attempted rows equal published rows and failed rows are zero.
  • Serving readiness: feature_store_registry, feature_store_online_store, and feature_store_freshness are ready.
  • Performance benchmark: p95/p99 latency and throughput do not regress beyond the committed baseline.
  • Capacity plan: Redis memory and ClickHouse partitions have approved headroom.
  • Recommendation quality: candidate metrics do not regress and result-validity exposure rates are zero.

Rollout

Only after backfill validation passes:

  1. Confirm the recommendation-quality artifact is present at artifacts/recommendation_quality/time/latest_vs_production.json.
  2. Run the deploy workflow.
  3. Confirm the deploy workflow executes the v2 feature-store, performance, capacity, and recommendation-quality gates before applying serving manifests.
  4. Watch readiness and feature-store metrics for the full rollout window.

Rollback

If validation fails before serving rollout, fix the failed feature set before deploying. If serving has already rolled out and readiness fails, roll back the deployment image or disable traffic through the approved release process. Do not change validation thresholds during rollback.