Recommendation Quality V2 Runbook

Purpose

Recommendation quality is a production contract. A model or policy is not eligible for rollout unless it is measurable, traceable, valid for the request context, and at least as good as the production baseline on the required offline gates.

This runbook covers the v2 quality gates introduced for the Feast recommender overhaul:

• Offline ranking quality on a time-based holdout.
• Candidate-generation recall, coverage, source contribution, and duplicate checks.
• Served result validity checks with zero tolerance for duplicate, invalid, unavailable, or untraceable items.
• Per-segment quality summaries for personalization regressions.
• Online experiment gates for sample-ratio mismatch, primary metric regression, guardrail breaches, and segment regressions.

Required Recommendation Log Fields

Every served recommendation item must be traceable to the request, model, feature contract, and candidate source.

Required context fields:

• request_id
• experiment_id
• model_version
• feature_service_version
• timestamp

Required item fields:

• item_id
• rank
• score
• trace.candidate_source

Required item display and eligibility fields:

• metadata.name
• metadata.image_url
• is_available
• city_id when city eligibility applies
• metadata.eligible_locales when locale eligibility applies

Offline Quality Gate

Prepare a time-based holdout artifact:

{
  "candidate_model": "latest",
  "baseline": "production",
  "split": "time",
  "catalog_items": ["1", "2"],
  "examples": [
    {
      "request_id": "req-1",
      "relevant_items": ["1"],
      "candidate_recommendations": [
        {"item_id": "1", "score": 0.9, "source": "two_tower"}
      ],
      "baseline_recommendations": [
        {"item_id": "2", "score": 0.7, "source": "popular_city"}
      ],
      "segments": {
        "member_type": "established_member",
        "city_id": "bkk",
        "language": "en",
        "traffic_source": "direct",
        "device_class": "ios",
        "request_hour_bucket": "lunch"
      }
    }
  ]
}

Run:

uv run python scripts/evaluate_recommendation_quality_v2.py \
  --input artifacts/recommendation_quality/time/latest_vs_production.json \
  --candidate-model latest \
  --baseline production \
  --split time \
  --k 10 \
  --output json

The gate fails when any required metric regresses below the production baseline:

• recall@100
• ndcg@10
• map@10
• mrr@10

By default, allowed regression is 0.0. Do not loosen this for production rollout without an approved experiment design and explicit risk acceptance.

Served Result Validation Gate

Recommendation result logs may be supplied as JSON or JSONL. Each record must contain context and items.

Run:

uv run python scripts/validate_recommendation_results_v2.py \
  --input artifacts/recommendation_quality/served_results.jsonl \
  --output json

This gate fails when any of these rates is greater than zero:

• Duplicate item exposure.
• Invalid item exposure.
• Unavailable item exposure.
• Untraceable result exposure.

Empty result sets and fallback recommendations must include an explicit fallback reason. The reason should be specific enough for operations to identify the failed dependency or policy path.

Segment Coverage

Quality artifacts should include these segment fields wherever available:

• member_type
• cold_start_cohort
• city_id
• language
• traffic_source
• device_class
• request_hour_bucket

Segment analysis is launch-blocking when a sufficiently sampled segment regresses beyond the approved threshold, even if the global metric passes.

Online Experiment Gates

Before ramping traffic, validate:

• Assignment ratios match the experiment design.
• Primary business metric does not regress beyond the approved bound.
• Guardrail metrics stay within configured thresholds.
• Segments with enough samples do not regress.

The pure gate primitives live in recsys.evaluation.experiment_gates and should be wired into the experiment-analysis job that reads production experiment telemetry.

Validation Commands

uv run pytest tests/unit/test_recommendation_metrics_v2.py \
  tests/unit/test_recommendation_result_quality_v2.py \
  tests/unit/test_experiment_gates_v2.py -q

uv run pytest tests/integration/test_recommendation_quality_pipeline_v2.py -q

uv run ruff check recsys/evaluation/recommendation_metrics.py \
  recsys/evaluation/result_quality.py \
  recsys/evaluation/segment_analysis.py \
  recsys/evaluation/experiment_gates.py \
  scripts/evaluate_recommendation_quality_v2.py \
  scripts/validate_recommendation_results_v2.py

uv run mypy recsys/evaluation/recommendation_metrics.py \
  recsys/evaluation/result_quality.py \
  recsys/evaluation/segment_analysis.py \
  recsys/evaluation/experiment_gates.py \
  scripts/evaluate_recommendation_quality_v2.py \
  scripts/validate_recommendation_results_v2.py

Production Readiness Checklist

• Holdout data is time-based and built without leakage from post-cutoff interactions.
• Candidate and baseline outputs are generated from the same request set.
• Every result row includes model, experiment, FeatureService, candidate source, item, score, rank, and timestamp.
• Duplicate, invalid, unavailable, and untraceable exposure rates are exactly zero.
• recall@100, ndcg@10, map@10, and mrr@10 do not regress versus production.
• Segment reports are reviewed for established members, sparse-history members, cold-start cohorts, city, language, traffic source, device class, and request-hour buckets.
• Experiment-analysis jobs enforce SRM, primary metric, guardrail, and segment gates before traffic ramps.