HH Lion Feast Recommender Platform Overhaul Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the current latest-snapshot Feast integration with a production-grade, point-in-time correct feature platform for hh-lion recommender training and serving.

Architecture: ClickHouse becomes the append-only historical offline feature store, Redis remains the Feast online store, and Feast becomes the explicit train/serve feature contract rather than a best-effort cache wrapper. Serving reads typed online feature contracts, training reads Feast historical features, and batch jobs are idempotent, observable, and fail closed when feature consistency cannot be guaranteed.

Tech Stack: Python 3.12, uv, FastAPI, Feast, ClickHouse, Redis, MLflow, Kafka, pytest, Ruff, mypy, Kubernetes, GitHub Actions.

1. Executive Summary

The current implementation must be overhauled rather than patched. It has a reasonable starting shape: Feast definitions, Redis online serving, ClickHouse storage, deployment jobs, and tests. The core contract is still wrong for production recommender ML because the offline feature tables store latest snapshots, while training code and documentation describe point-in-time retrieval. That violates Feast’s intended historical retrieval model and creates material leakage and training-serving skew risk.

The target design is a simpler and stricter feature platform:

Append-only historical feature tables in ClickHouse, keyed by entity and feature_timestamp.
Feast offline sources reading historical rows with timestamp_field and created_timestamp_column.
Feast online Redis containing latest production feature values pushed or materialized from validated feature runs.
Versioned model-specific Feast FeatureServices.
Typed online/offline clients with no hidden config mutation and no silent fallback masking.
Feature computation jobs that are deterministic for a supplied --as-of timestamp.
Readiness, metrics, and alerts that fail production when required feature paths are broken.

Backward compatibility is intentionally not preserved. Existing v1 latest-snapshot feature tables and broad adapters should be replaced after v2 validation.

2. Current-State Assessment

What is worth keeping

The project already separates Feast feature definitions under features/.
Redis online store and ClickHouse offline store are appropriate technologies for this workload.
Deployment workflows apply Feast registry definitions before serving rollout.
Unit and contract tests exist for adapter behavior and feature store schema shape.
The codebase already recognizes Feast as the preferred path for member, item, and cohort features.

What is not production-grade

Current ClickHouse feature tables are latest-snapshot tables, not historical feature stores.
Feast historical retrieval is documented and used as point-in-time retrieval, but the underlying source data cannot support that contract.
Online and offline writes are not atomic and push failures can be swallowed.
Serving health checks can pass while Feast is unavailable.
Feature freshness is inconsistently represented and sometimes fabricated.
The adapter layer is too broad, mixes config rendering, online reads, offline reads, push serialization, fallback behavior, and health checks.
FeatureServices are not versioned by model contract.
Candidate sets and serving payloads are mixed into feature storage.
Recommendation result quality is not expressed as an explicit production contract with offline, online, and per-segment gates.
Tests mostly validate mocked code paths, not real Feast plus ClickHouse plus Redis behavior.

3. Major Architectural and Implementation Problems

P0: Offline features are not point-in-time correct

Current tables use ReplacingMergeTree(version) ordered by entity only, such as ORDER BY (user_id), ORDER BY (item_id), and ORDER BY (cohort_key). Feast source queries read FINAL, so older feature snapshots are removed from the query result. Feast can only perform correct point-in-time joins if historical feature rows exist with event timestamps.

Impact:

Offline training can use future aggregate values.
Backtests and offline metrics can be inflated.
Ranker features can mismatch online serving values.
Data leakage is likely whenever event timestamps predate the latest feature snapshot.

Production requirement:

Store feature history explicitly.
Validate point-in-time joins with integration tests containing multiple snapshots per entity.

P0: Feature computation is not deterministic for arbitrary historical timestamps

Current jobs compute features from now() and write a latest row. For production backfills and reproducible training, each feature run must be anchored to a supplied feature_timestamp or schedule boundary.

Impact:

Historical backfills cannot recreate the feature state available at a past time.
Training datasets cannot be reproduced exactly.
Model lineage is incomplete.

Production requirement:

Every compute job accepts --as-of.
Every output row stores feature_timestamp, created_timestamp, compute_run_id, and feature_row_version.

P1: Online and offline consistency is not enforced

Current jobs can write ClickHouse successfully and then silently fail to push online features. Serving then falls back to ClickHouse or cohort defaults, hiding the inconsistency.

Impact:

Redis online store can drift from offline history.
Degraded recommendations can ship without failing readiness.
Production debugging becomes guesswork.

Production requirement:

Feature jobs fail when required online publication fails.
Push/materialization results are audited by row count, coverage, and freshness.

P1: Serving dependency health is too weak

Serving readiness checks Redis and model state, but not the Feast registry contract, online feature availability, or feature freshness. The runtime can construct a feature service even when the Feast adapter is unavailable.

Impact:

Broken feature store deployments can receive traffic.
Fallback behavior can mask production regressions.

Production requirement:

Production readiness requires Feast registry validation, online store health, and freshness within SLA.

P1: Feature contracts are not versioned

FeatureServices such as member_activity and item_details are broad service names, not model-version contracts. Serving code also falls back to direct feature references.

Impact:

Schema changes can silently alter model inputs.
MLflow model versions cannot reliably declare their feature dependencies.

Production requirement:

FeatureServices are versioned per model family and major feature schema version.
MLflow artifacts record the Feast project, registry version, FeatureService name, and feature list.

P2: The adapter layer is too complex

recsys/features/feast.py has too many responsibilities: environment mutation, runtime repo generation, registry validation, historical reads, online reads, push serialization, fallback key logic, and health checks.

Impact:

Small changes are risky.
Tests need broad mocking.
Production behavior is harder to reason about.

Production requirement:

Split configuration, registry validation, online reads, offline reads, writing, serialization, and health into focused modules.

P2: Candidate sets are stored as features

Cohort top_restaurants, popularity maps, and candidate lists are retrieval artifacts. Feast should hold model input features, not arbitrary recommendation outputs.

Impact:

Feature store becomes a mixed feature/retrieval cache.
Schema contracts become hard to maintain.
Large arrays or maps increase serialization and online-store risk.

Production requirement:

Store candidate sets in a dedicated candidate table/cache.
Store only stable model features in Feast.

P1: Recommendation quality is not a release gate

The current plan protects feature correctness and serving reliability, but it must also define what a relevant, reliable recommendation result means. A recommender can be technically correct and fast while still producing duplicate, unavailable, poorly ranked, low-diversity, or unmeasurable results.

Impact:

Offline model improvements can fail to translate into better production outcomes.
Candidate generation can silently lose relevant restaurants while ranking metrics look stable.
Cold-start, guest, city, language, and sparse-history segments can regress without a global metric moving.
Product-quality regressions can ship if canary decisions rely only on infrastructure health.

Production requirement:

Treat recommendation quality as a production contract with offline ranking metrics, result-validity checks, segment-level dashboards, online experiment guardrails, and launch-blocking gates.

4. Proposed Target Architecture

Data flow

flowchart LR
    A["Raw events and source snapshots"] --> B["Deterministic feature compute jobs"]
    B --> C["ClickHouse historical feature tables"]
    C --> D["Feast offline store"]
    C --> E["Latest feature validation view"]
    E --> F["Feast online Redis"]
    D --> G["Training and export jobs"]
    F --> H["FastAPI serving"]
    B --> I["Feature compute run audit"]
    F --> J["Online feature freshness metrics"]

Core principles

ClickHouse history is authoritative for offline feature values.
Redis is the low-latency serving copy, never the source of historical truth.
Feast is the contract boundary for model features.
Direct ClickHouse serving fallback is disabled by default in production.
Any fallback is explicit, observable, and isolated from normal personalized serving.
Feature computation is deterministic and idempotent.
Feature schema changes are versioned and treated as model-contract changes.

Package layout

Create a new focused package and migrate callers into it:

recsys/feature_store/
  __init__.py
  config.py
  contracts.py
  health.py
  offline_client.py
  online_client.py
  publisher.py
  registry.py
  serialization.py

recsys/evaluation/
  __init__.py
  experiment_gates.py
  recommendation_metrics.py
  result_quality.py
  segment_analysis.py

recsys/features/
  feature_jobs/
    __init__.py
    base.py
    member.py
    item_popularity.py
    item_metadata.py
    cohort.py

features/
  entities.py
  sources.py
  member_v2.py
  item_v2.py
  cohort_v2.py
  services_v2.py
  feature_store.yaml

5. Components to Delete, Replace, or Refactor

Delete after v2 cutover

features/data/feast_registry.db: generated local registry state must not be used as a shared deployment artifact.
v1 Feast definitions in features/member_features.py, features/item_features.py, features/cohort_features.py, and features/feature_services.py.
v1 serving fallback paths that read legacy entity-only feature snapshots as if they were equivalent to Feast.
The as_of query parameter on online feature serving routes unless a separate authenticated historical API is built.

Replace

recsys/features/feast.py with focused clients under recsys/feature_store/.
recsys/features/service.py with a smaller serving feature provider that only orchestrates typed clients and explicit fallback policy.
scripts/compute_member_features.py, scripts/compute_item_features.py, and scripts/compute_cohort_features.py with deterministic v2 jobs under recsys/features/feature_jobs/ plus thin CLI wrappers.
Current feature migrations with v2 historical tables and latest validation views.

Refactor

scripts/extract_ranker_training_data.py: keep Feast historical retrieval, but require v2 FeatureServices and fail if point-in-time coverage is incomplete.
recsys/features/exporter.py: remove direct feature-table joins for training features; use Feast offline retrieval only.
recsys/serving/runtime_builder.py: require production feature dependencies during readiness.
recsys/serving/routes/health.py: expose separate registry, online store, freshness, and feature coverage statuses.
GitHub workflows and Kubernetes jobs: apply v2 Feast definitions, run v2 compute jobs, validate, then publish online features.

6. Database and Table Redesign Recommendations

Historical feature table rules

Every Feast-backed feature table must include:

Entity key columns.
feature_timestamp DateTime64(3, 'UTC'): the timestamp Feast uses for point-in-time joins.
created_timestamp DateTime64(3, 'UTC'): insertion/production time for deduplication of late writes.
compute_run_id UUID: lineage for the exact job run.
feature_row_version UInt64: monotonically increasing version for corrections to the same entity and feature timestamp.
source_watermark DateTime64(3, 'UTC'): newest source event included in the computation.
feature_schema_version UInt16: explicit schema version.

Use ReplacingMergeTree(feature_row_version) ordered by entity and feature_timestamp, not entity only. This preserves historical snapshots while allowing deterministic correction of the same snapshot.

Required v2 tables

Create scripts/migrations/020_create_feature_store_v2.sql with:

Do not rewrite already-applied v1 migrations. Add v2 objects in a new migration, then remove v1 objects through an explicit drop migration after cutover.

CREATE TABLE IF NOT EXISTS feature_compute_runs
(
    compute_run_id UUID,
    feature_set LowCardinality(String),
    feature_schema_version UInt16,
    feature_timestamp DateTime64(3, 'UTC'),
    source_watermark DateTime64(3, 'UTC'),
    started_at DateTime64(3, 'UTC'),
    finished_at Nullable(DateTime64(3, 'UTC')),
    updated_at DateTime64(3, 'UTC'),
    status LowCardinality(String),
    row_count UInt64,
    failed_reason Nullable(String)
)
ENGINE = ReplacingMergeTree(updated_at)
PARTITION BY toYYYYMM(started_at)
ORDER BY (feature_set, feature_timestamp, compute_run_id);

CREATE TABLE IF NOT EXISTS member_feature_history_v2
(
    user_id String,
    feature_timestamp DateTime64(3, 'UTC'),
    booking_count UInt32,
    meaningful_interaction_count UInt32,
    cuisine_affinity Map(String, Float32),
    preference_strength Float32,
    avg_spend_tier LowCardinality(String),
    preferred_price_tier LowCardinality(String),
    booking_frequency_bucket LowCardinality(String),
    last_booking_timestamp Nullable(DateTime64(3, 'UTC')),
    meets_history_threshold UInt8,
    created_timestamp DateTime64(3, 'UTC'),
    source_watermark DateTime64(3, 'UTC'),
    compute_run_id UUID,
    feature_schema_version UInt16,
    feature_row_version UInt64
)
ENGINE = ReplacingMergeTree(feature_row_version)
PARTITION BY toYYYYMM(feature_timestamp)
ORDER BY (user_id, feature_timestamp)
TTL toDateTime(feature_timestamp) + INTERVAL 730 DAY;

CREATE TABLE IF NOT EXISTS item_popularity_feature_history_v2
(
    item_id String,
    feature_timestamp DateTime64(3, 'UTC'),
    popularity_score Float32,
    impression_count_7d UInt32,
    click_count_7d UInt32,
    booking_count_7d UInt32,
    ctr_7d Float32,
    cvr_7d Float32,
    availability_ratio_7d Nullable(Float32),
    next_available_at Nullable(DateTime64(3, 'UTC')),
    is_available UInt8,
    created_timestamp DateTime64(3, 'UTC'),
    source_watermark DateTime64(3, 'UTC'),
    compute_run_id UUID,
    feature_schema_version UInt16,
    feature_row_version UInt64
)
ENGINE = ReplacingMergeTree(feature_row_version)
PARTITION BY toYYYYMM(feature_timestamp)
ORDER BY (item_id, feature_timestamp)
TTL toDateTime(feature_timestamp) + INTERVAL 730 DAY;

CREATE TABLE IF NOT EXISTS item_metadata_feature_history_v2
(
    item_id String,
    feature_timestamp DateTime64(3, 'UTC'),
    cuisine_categories Array(String),
    primary_cuisine Nullable(String),
    price_tier LowCardinality(String),
    pricing_type_primary Nullable(String),
    package_type_primary Nullable(String),
    location_cluster Nullable(String),
    city_id Nullable(String),
    geo_lat Nullable(Float64),
    geo_lng Nullable(Float64),
    price_min Nullable(Decimal64(2)),
    price_max Nullable(Decimal64(2)),
    price_currency Nullable(String),
    reviews_score Nullable(Float32),
    reviews_count Nullable(UInt32),
    google_review_score Nullable(Float32),
    favorites_count Nullable(UInt32),
    accept_voucher UInt8,
    hashtags Array(String),
    facilities Array(String),
    dining_styles Array(String),
    created_timestamp DateTime64(3, 'UTC'),
    source_watermark DateTime64(3, 'UTC'),
    compute_run_id UUID,
    feature_schema_version UInt16,
    feature_row_version UInt64
)
ENGINE = ReplacingMergeTree(feature_row_version)
PARTITION BY toYYYYMM(feature_timestamp)
ORDER BY (item_id, feature_timestamp)
TTL toDateTime(feature_timestamp) + INTERVAL 730 DAY;

CREATE TABLE IF NOT EXISTS cohort_popularity_feature_history_v2
(
    cohort_key String,
    feature_timestamp DateTime64(3, 'UTC'),
    origin_country Nullable(String),
    country String,
    city_id Nullable(String),
    language String,
    referrer_category String,
    top_cuisines Array(String),
    sample_size UInt32,
    created_timestamp DateTime64(3, 'UTC'),
    source_watermark DateTime64(3, 'UTC'),
    compute_run_id UUID,
    feature_schema_version UInt16,
    feature_row_version UInt64
)
ENGINE = ReplacingMergeTree(feature_row_version)
PARTITION BY toYYYYMM(feature_timestamp)
ORDER BY (cohort_key, feature_timestamp)
TTL toDateTime(feature_timestamp) + INTERVAL 730 DAY;

CREATE TABLE IF NOT EXISTS cohort_candidate_sets_v2
(
    cohort_key String,
    computed_at DateTime64(3, 'UTC'),
    expires_at DateTime64(3, 'UTC'),
    item_ids Array(String),
    scores Array(Float32),
    source_watermark DateTime64(3, 'UTC'),
    compute_run_id UUID,
    feature_row_version UInt64
)
ENGINE = ReplacingMergeTree(feature_row_version)
PARTITION BY toYYYYMM(computed_at)
ORDER BY (cohort_key, computed_at)
TTL toDateTime(computed_at) + INTERVAL 90 DAY;

CREATE TABLE IF NOT EXISTS feature_publication_audit
(
    compute_run_id UUID,
    feature_set LowCardinality(String),
    feature_timestamp DateTime64(3, 'UTC'),
    target LowCardinality(String),
    attempted_rows UInt64,
    published_rows UInt64,
    failed_rows UInt64,
    started_at DateTime64(3, 'UTC'),
    finished_at DateTime64(3, 'UTC'),
    status LowCardinality(String),
    failed_reason Nullable(String)
)
ENGINE = ReplacingMergeTree(finished_at)
PARTITION BY toYYYYMM(started_at)
ORDER BY (feature_set, target, feature_timestamp, compute_run_id);

Latest views

Create latest views for validation and operational inspection only:

CREATE VIEW IF NOT EXISTS member_feature_latest_v2 AS
SELECT h.*
FROM member_feature_history_v2 AS h
INNER JOIN
(
    SELECT
        user_id,
        max(tuple(feature_timestamp, feature_row_version)) AS latest_key
    FROM member_feature_history_v2
    GROUP BY user_id
) AS latest
ON h.user_id = latest.user_id
AND tuple(h.feature_timestamp, h.feature_row_version) = latest.latest_key;

Repeat the same pattern for item popularity, item metadata, and cohort popularity. Do not point Feast historical sources at latest views.

7. Performance Engineering and Production Workload Optimization

Production readiness requires explicit performance budgets, capacity planning, query optimization, load testing, and regression gates. Correctness is the first requirement, but the v2 design must also prove it can serve production traffic and refresh features within operational windows.

Performance SLOs

Use these initial SLOs as hard release gates. They can be tightened after production baselines exist, but they must not be weakened without a documented capacity review.

Online single-entity feature lookup: p95 <= 20 ms and p99 <= 50 ms inside the feature provider, excluding upstream model inference.
Online item batch lookup for 100 items: p95 <= 75 ms and p99 <= 150 ms inside the feature provider.
Homepage request feature assembly: p95 <= 100 ms and p99 <= 250 ms for all feature retrieval and normalization work.
Feast online Redis timeout budget: <= 50 ms per request, with bounded retries disabled on the synchronous serving path.
ClickHouse historical training retrieval: >= 50,000 entity rows per minute for point-in-time training feature extraction in engineering using production-like data volume.
Online publication throughput: >= 10,000 feature rows per minute per worker with zero silent row loss.
Incremental feature jobs: complete within 30 minutes for the production daily workload.
Full 730-day backfill: complete successfully in engineering before production cutover, run without competing with serving workloads, and publish no online data until validation passes.

ClickHouse optimization requirements

Historical tables must be partitioned by toYYYYMM(feature_timestamp) to keep point-in-time scans bounded.
Primary sort keys must start with the Feast join key followed by feature_timestamp.
Feast historical source queries must not use latest views.
Avoid FINAL on hot training queries. If corrected duplicate rows require deduplication, validate whether the cost is acceptable with EXPLAIN and engineering benchmarks before release.
Add minmax or bloom-filter indexes only after a query profile demonstrates a measurable benefit.
Keep large candidate arrays out of Feast history tables.
Use low-cardinality string types for bounded enums and dimensions.
Validate storage growth with a 730-day retention model before production backfill.

Redis and Feast online optimization requirements

Keep online feature payloads small: scalar features and compact lists only; large candidate lists stay in the candidate store.
Use connection pooling and explicit connection limits in the online client.
Use bounded batch sizes for item lookups. Default batch size is 100 item entities per Feast call unless benchmarks prove a better value.
Do not perform synchronous ClickHouse fallback on the normal production request path.
Registry cache TTL must be configured for serving processes; registry refresh should not occur on every request.
Online write paths must publish in bounded chunks and record attempted, published, and failed rows.
Redis memory sizing must account for key cardinality, serialized payload size, TTL, and peak duplicate writes during publication.

Workload and capacity model

Before v2 cutover, create a capacity model with:

Peak homepage request rate: maximum of 100 RPS or 2.5x the observed production p99 homepage request rate over the previous 30 days.
Peak item batch size: 100 item IDs per request unless serving code enforces a lower maximum.
Feature key cardinality: current active members, active items, active cohorts, and projected 12-month growth.
Online memory estimate: key count multiplied by measured serialized payload size plus 35% allocator and replication overhead.
Batch compute volume: rows scanned, rows produced, and rows published for each feature set.
Backfill volume: number of historical monthly partitions and estimated rows per partition.

Performance validation gates

The release is blocked unless all gates pass in engineering with production-like data volume:

Online feature provider p95 and p99 latency meet the SLOs above for single-member, single-cohort, and 100-item batch lookups.
Homepage feature assembly latency meets the SLOs above at the defined peak request rate for 30 minutes.
Redis CPU remains below 70%, memory remains below 75% of the configured max memory, connection count remains below 70% of the configured limit, and evictions stay at zero during load tests.
ClickHouse training retrieval meets the throughput SLO without memory-limit failures or excessive entity-table growth.
ClickHouse EXPLAIN and query logs show partition pruning on feature_timestamp for historical retrieval.
Feature publication throughput meets the SLO and the audit table reports attempted rows equal published rows.
Incremental feature jobs complete inside the 30-minute budget for three consecutive engineering runs.
Backfill dry run produces the expected partition and row-count profile before any production backfill.
Performance regression tests fail when p95 latency, p99 latency, memory, or query duration regresses by more than 10% from the checked-in baseline.

8. Personalization and Recommendation Quality

The feature platform exists to produce relevant, reliable, and measurable recommendations. Recommendation quality must be validated independently from feature-store correctness, infrastructure health, and request latency.

Quality contract

Every recommendation response must satisfy these hard rules:

No duplicate item IDs in the returned list.
No item that is unavailable, inactive, hidden from the serving surface, or missing required display metadata.
No item outside the request’s eligible city, locale, availability, and product constraints.
No result with an untraceable source: each item must include candidate source, ranker/model version, score, feature contract version, and experiment assignment in logs.
No silent empty result: empty or low-confidence results must return an explicit degraded reason and use a documented non-personalized fallback.
Stable ordering for identical inputs, model version, feature values, and candidate set.

Offline recommendation metrics

Evaluate every model and ranking-policy change on time-based holdout data using the same candidate-generation and ranking pipeline shape used in serving. At minimum, compute:

Candidate generation: Recall@50, Recall@100, candidate coverage, candidate source contribution, and candidate deduplication rate.
Ranking: NDCG@5, NDCG@10, MAP@10, MRR@10, Precision@10, and Recall@10.
Result quality: valid-result rate, duplicate rate, unavailable-item rate, empty-result rate, diversity by cuisine/category, geographic relevance, price-tier relevance, and freshness of available inventory.
Segment quality: member vs guest, established vs sparse-history member, cold-start cohort, city, language, traffic source, device class, and request hour bucket.
Calibration and reliability: score distribution drift, rank-position click/booking curves, and predicted-score monotonicity against observed outcomes.

Online measurement and experimentation

Every production model or recommendation-policy change must run through controlled measurement:

Log impression, click, booking, dismissal, fallback, and empty-result events with request ID, user/session key, item ID, rank, score, candidate source, model version, FeatureService version, experiment ID, and timestamp.
Use a stable randomization unit for experiments and enforce sample-ratio-mismatch checks before reading results.
Ramp through 1%, 5%, 25%, 50%, and 100% exposure only when guardrails pass at each stage.
Primary online metrics: click-through rate, booking conversion, revenue per session, and successful recommendation interaction rate.
Guardrail metrics: latency, error rate, empty-result rate, fallback rate, unavailable-item exposure rate, duplicate rate, cancellation or reversal proxy metrics, and segment-level regressions.
Keep a long-lived holdout or champion baseline so gains are measured against the production system, not only against offline baselines.

Recommendation-quality validation gates

The release is blocked unless all gates pass:

Candidate Recall@100 is not lower than the production baseline on the time-based holdout.
NDCG@10, MAP@10, and MRR@10 are not lower than the production baseline on the global holdout.
No critical segment has more than a 3% relative regression in NDCG@10, Recall@10, or valid-result rate.
Duplicate item rate is exactly 0 in offline replay and engineering shadow traffic.
Unavailable or invalid item exposure rate is exactly 0 in offline replay and engineering shadow traffic.
Empty-result rate is below 0.1% for eligible requests in engineering shadow traffic.
Fallback rate is below 1% during healthy engineering operation.
Online canary shows no statistically significant regression in primary metrics and no guardrail breach before each ramp step.
Any model claiming personalization improvement must beat the non-personalized baseline for member, guest, and cold-start slices separately.

9. Step-by-Step Implementation Plan

Task 1: Establish v2 feature contracts

Files:

Create: recsys/feature_store/contracts.py
Create: tests/unit/test_feature_store_contracts_v2.py
Define Pydantic or dataclass contracts for MemberFeaturesV2, ItemPopularityFeaturesV2, ItemMetadataFeaturesV2, and CohortPopularityFeaturesV2.
Require feature_timestamp, created_timestamp, source_watermark, compute_run_id, feature_schema_version, and feature_row_version in every contract.
Represent preference_strength as float, not a string.
Keep maps and arrays only where the model contract actually consumes them.
Add tests that invalid timestamps, missing lineage fields, and invalid enum values fail validation.
Run: uv run pytest tests/unit/test_feature_store_contracts_v2.py -q.

Task 2: Add v2 ClickHouse schema

Files:

Create: scripts/migrations/020_create_feature_store_v2.sql
Create: tests/contract/test_clickhouse_feature_store_v2_schema.py
Add the v2 tables listed in section 6.
Add schema tests that assert each history table is ordered by entity plus feature_timestamp.
Add schema tests that reject entity-only ORDER BY for Feast historical tables.
Add schema tests that all Feast-backed tables expose feature_timestamp and created_timestamp.
Run: uv run pytest tests/contract/test_clickhouse_feature_store_v2_schema.py -q.

Task 3: Rebuild Feast definitions around historical sources

Files:

Create: features/sources.py
Create: features/member_v2.py
Create: features/item_v2.py
Create: features/cohort_v2.py
Create: features/services_v2.py
Modify: features/entities.py
Modify: features/__init__.py
Create: tests/contract/test_feast_repo_v2.py
Define ClickHouse sources against *_feature_history_v2 tables, not latest views.
Set timestamp_field="feature_timestamp" and created_timestamp_column="created_timestamp" for every Feast source.
Use FINAL only on v2 historical ReplacingMergeTree sources when needed to deduplicate corrected rows for the same (entity, feature_timestamp); never point Feast at entity-only latest-snapshot sources.
Define TTLs per feature view: member 12h, item popularity 24h, item metadata 7d, cohort 24h.
Define model-specific FeatureServices such as homepage_ranker_v2, member_two_tower_context_v2, and guest_cold_start_v2.
Remove candidate sets from Feast FeatureViews.
Add contract tests that every FeatureView uses a historical v2 source and every production FeatureService name ends in _v2.
Run: uv run pytest tests/contract/test_feast_repo_v2.py -q.

Task 4: Replace the monolithic Feast adapter

Files:

Create: recsys/feature_store/config.py
Create: recsys/feature_store/registry.py
Create: recsys/feature_store/offline_client.py
Create: recsys/feature_store/online_client.py
Create: recsys/feature_store/publisher.py
Create: recsys/feature_store/serialization.py
Create: recsys/feature_store/health.py
Modify: recsys/features/__init__.py
Delete after migration: recsys/features/feast.py
Create: tests/unit/test_feature_store_config.py
Create: tests/unit/test_feature_store_clients.py
Create: tests/unit/test_feature_store_publisher.py
Move config parsing into config.py; production/engineering must fail if required Feast registry and store settings are absent.
Remove runtime mutation from application startup. Config rendering belongs in explicit bootstrap/deployment commands.
Implement OfflineFeatureClient.get_historical_features(entity_df, feature_service_name).
Implement OnlineFeatureClient.get_online_features(entity_rows, feature_service_name).
Implement FeaturePublisher.publish_online(frame, push_source_name, compute_run_id) that raises on any failed push.
Implement serialization as pure functions with tests for maps, arrays, timestamps, nullable values, and decimals.
Implement FeatureStoreHealth.check() returning separate registry, online store, offline store, freshness, and publication states.
Run: uv run pytest tests/unit/test_feature_store_config.py tests/unit/test_feature_store_clients.py tests/unit/test_feature_store_publisher.py -q.

Task 5: Replace feature computation jobs

Files:

Create: recsys/features/feature_jobs/base.py
Create: recsys/features/feature_jobs/member.py
Create: recsys/features/feature_jobs/item_popularity.py
Create: recsys/features/feature_jobs/item_metadata.py
Create: recsys/features/feature_jobs/cohort.py
Replace: scripts/compute_member_features.py
Replace: scripts/compute_item_features.py
Replace: scripts/compute_cohort_features.py
Create: tests/unit/test_feature_jobs_v2.py
Create: tests/integration/test_feature_job_idempotency_v2.py
Every job accepts --as-of, --source-watermark, --mode full|incremental|backfill, and --publish-online.
Every job writes feature_compute_runs with running, succeeded, or failed.
Every job writes historical rows before publishing online features.
Every job fails if --publish-online is set and online publication fails.
Every job is idempotent for the same (feature_set, feature_timestamp, compute_run_id).
Member features must use windows anchored to --as-of, not wall-clock now().
Item popularity features must use windows anchored to --as-of.
Item metadata features must snapshot source metadata at --as-of or record the source snapshot watermark explicitly.
Cohort candidate sets must write to cohort_candidate_sets_v2, not Feast.
Run: uv run pytest tests/unit/test_feature_jobs_v2.py tests/integration/test_feature_job_idempotency_v2.py -q.

Task 6: Make training and export Feast-first

Files:

Modify: scripts/extract_ranker_training_data.py
Modify: recsys/features/exporter.py
Modify: scripts/train.py
Modify: scripts/train_catboost.py
Create: tests/integration/test_feast_point_in_time_v2.py
Create: tests/unit/test_training_feature_contract_v2.py
Create: tests/unit/test_train_catboost.py
Require v2 FeatureService names for training datasets.
Remove direct joins from v1 feature tables for model input features.
Build entity dataframes with event timestamps in UTC and retrieve features through OfflineFeatureClient.
Fail training export when Feast returns missing required features above an explicit threshold.
Record FeatureService name, feature schema version, registry version, and feature coverage in MLflow metadata.
Keep direct interaction-event loading only for models whose contract explicitly has no Feast features.
Run: uv run pytest tests/integration/test_feast_point_in_time_v2.py tests/unit/test_training_feature_contract_v2.py -q.

Task 7: Simplify serving feature access and fallback policy

Files:

Create: recsys/serving/feature_gateway.py
Create: recsys/serving/features/provider.py
Create: recsys/serving/features/fallback_policy.py
Modify: recsys/serving/runtime_builder.py
Modify: recsys/serving/feature_adapter.py
Modify: recsys/serving/routes/features.py
Delete after migration: recsys/features/service.py
Delete after migration: recsys/serving/feature_service.py
Create: tests/unit/test_serving_feature_provider_v2.py
Create: tests/unit/test_serving_feature_fallback_policy_v2.py
Rename the serving layer away from FeatureService to avoid confusion with Feast FeatureService.
Online serving must call OnlineFeatureClient with versioned FeatureService names.
Direct ClickHouse fallback must be disabled in production unless FEATURE_FALLBACK_MODE=explicit_degraded is set.
Missing required online features must return a typed unavailable result, not fabricated values.
Remove or reject the online as_of parameter.
Candidate retrieval should use cohort_candidate_sets_v2 through a separate retriever, not Feast feature lookups.
Run: uv run pytest tests/unit/test_serving_feature_provider_v2.py tests/unit/test_serving_feature_fallback_policy_v2.py -q.

Task 8: Add production health, observability, and data quality gates

Files:

Modify: recsys/serving/routes/health.py
Modify: recsys/serving/utils/observability.py
Create: recsys/feature_store/quality.py
Create: scripts/validate_feature_store_v2.py
Create: tests/unit/test_feature_store_health_v2.py
Create: tests/unit/test_feature_quality_checks_v2.py
Readiness requires valid Feast registry, online store health, and freshness within SLA for required FeatureServices.
Liveness remains process-level and must not depend on downstream stores.
Emit metrics for online hit rate, missing required feature rate, feature age, push success, push row count, historical coverage, and registry version.
Implement validation checks for duplicate feature rows, null required fields, timestamp drift, source watermark lag, and freshness SLA.
Make deployment jobs fail when validation fails.
Run: uv run pytest tests/unit/test_feature_store_health_v2.py tests/unit/test_feature_quality_checks_v2.py -q.

Task 9: Build real integration tests

Files:

Create: tests/integration/test_feast_clickhouse_redis_v2.py
Create: tests/integration/test_feature_publication_v2.py
Modify: pyproject.toml
Add an integration test marker for Feast plus ClickHouse plus Redis.
Insert two historical snapshots for the same entity and assert Feast returns the older row for an older event timestamp and the newer row for a newer event timestamp.
Publish latest online features and assert online retrieval returns the latest production row.
Simulate a publication failure and assert the compute job exits non-zero.
Run narrow test command: uv run pytest -m "integration and feature_store_v2" -q.
Run compose-backed release-safety gate: USE_EXISTING_CLICKHOUSE=1 USE_EXISTING_REDIS=1 uv run pytest -m "integration and feature_store_v2" -rs -q.
Run full validation before cutover: uv run pytest && uv run ruff check . && uv run ruff format --check . && uv run mypy recsys.

Task 10: Add performance benchmarks and workload gates

Files:

Create: recsys/feature_store/performance.py
Create: scripts/benchmark_feature_store_v2.py
Create: scripts/capacity_plan_feature_store_v2.py
Create: tests/performance/test_feature_store_online_v2.py
Create: tests/performance/test_feature_store_offline_v2.py
Create: tests/performance/baselines/feature_store_v2.json
Modify: pyproject.toml
Create: docs/runbooks/feature_store_v2_performance.md
Add a performance pytest marker and keep performance tests separate from default unit tests.
Implement online benchmarks for member lookup, cohort lookup, and 100-item batch lookup.
Implement offline benchmarks for Feast historical retrieval over 50,000 entity rows.
Implement publication benchmarks that measure attempted rows, published rows, failed rows, throughput, and payload size.
Implement a capacity planning script that reports Redis key cardinality, serialized payload size, estimated memory, ClickHouse partition count, and backfill row volume.
Add ClickHouse query-plan checks that require partition pruning on feature_timestamp for historical retrieval queries.
Replace the initial SLO-budget baseline with a documented measured engineering baseline after live v2 integration tests pass.
Fail performance tests when p95 latency, p99 latency, memory, or query duration regresses by more than 10% from the baseline.
Run: uv run pytest tests/performance -m performance -q.
Run the capacity planner with production-like local validation inputs:

uv run python scripts/capacity_plan_feature_store_v2.py \
  --active-members <members> \
  --active-items <items> \
  --active-cohorts <cohorts> \
  --member-payload-bytes <bytes> \
  --item-payload-bytes <bytes> \
  --cohort-payload-bytes <bytes> \
  --daily-member-rows <rows> \
  --daily-item-popularity-rows <rows> \
  --daily-item-metadata-rows <rows> \
  --daily-cohort-rows <rows> \
  --retention-days 730 \
  --output text

Local validation used representative production-like inputs only. Production cutover still requires measured engineering inputs and a checked-in documented engineering baseline.

Task 11: Add personalization and recommendation-quality gates

Files:

Create: recsys/evaluation/recommendation_metrics.py
Create: recsys/evaluation/result_quality.py
Create: recsys/evaluation/segment_analysis.py
Create: recsys/evaluation/experiment_gates.py
Create: scripts/evaluate_recommendation_quality_v2.py
Create: scripts/validate_recommendation_results_v2.py
Create: tests/unit/test_recommendation_metrics_v2.py
Create: tests/unit/test_recommendation_result_quality_v2.py
Create: tests/unit/test_experiment_gates_v2.py
Create: tests/integration/test_recommendation_quality_pipeline_v2.py
Create: docs/runbooks/recommendation_quality_v2.md
Implement precision_at_k, recall_at_k, ndcg_at_k, map_at_k, and mrr_at_k with deterministic tie handling.
Implement candidate-generation metrics for Recall@50, Recall@100, source contribution, coverage, and deduplication.
Implement result-quality validation for duplicates, unavailable items, invalid metadata, locale/city eligibility, empty results, fallback reason, and stable ordering.
Implement segment analysis for member, guest, established member, sparse-history member, cold-start cohort, city, language, traffic source, device class, and request hour bucket.
Implement a quality evaluator that compares a candidate model or policy against the current production baseline on a time-based holdout.
Implement online experiment gates for sample-ratio mismatch, primary metric regression, guardrail breaches, and per-segment regressions.
Require recommendation logs to include request ID, experiment ID, model version, FeatureService version, candidate source, score, rank, item ID, and timestamp.
Fail validation when duplicate rate, invalid item exposure, unavailable item exposure, or untraceable result rate is greater than 0.
Fail validation when candidate Recall@100, NDCG@10, MAP@10, or MRR@10 regresses below the production baseline.
Run: uv run pytest tests/unit/test_recommendation_metrics_v2.py tests/unit/test_recommendation_result_quality_v2.py tests/unit/test_experiment_gates_v2.py -q.
Run: uv run pytest tests/integration/test_recommendation_quality_pipeline_v2.py -q.
Add the release gate command for the production-baseline quality artifact:

uv run python scripts/evaluate_recommendation_quality_v2.py \
  --input artifacts/recommendation_quality/time/latest_vs_production.json \
  --candidate-model latest \
  --baseline production \
  --split time \
  --k 10

Task 12: Update deployment and operations

Files:

Modify: .github/workflows/deploy.yml
Modify: .github/workflows/bootstrap-engineering.yml
Modify: k8s/shared/bootstrap-feast-repo-job.yaml
Modify: k8s/shared/bootstrap-member-features-job.yaml
Modify: k8s/shared/bootstrap-item-features-job.yaml
Modify: k8s/shared/bootstrap-cohort-features-job.yaml
Modify: k8s/engineering/serving.yaml
Modify: k8s/production/serving.yaml
Create: docs/runbooks/feature_store_v2_operations.md
Create: docs/runbooks/feature_store_v2_backfill.md
Apply database migration before applying v2 Feast definitions.
Apply Feast definitions from the v2 repo.
Run v2 backfill for the most recent 730 days, or all available source history when less than 730 days exists.
Validate historical coverage and online publication coverage.
Validate online latency, offline retrieval throughput, Redis memory headroom, and ClickHouse query plans before enabling v2 serving.
Validate recommendation-quality gates before enabling v2 serving.
Publish online features only after validation succeeds.
Make serving rollout wait for feature readiness.
Document failure handling for registry errors, stale features, publication failures, and data quality failures.

Task 13: Decommission v1 feature store paths

Files:

Delete: features/member_features.py
Delete: features/item_features.py
Delete: features/cohort_features.py
Delete: features/feature_services.py
Delete: recsys/features/feast.py
Delete: recsys/features/service.py
Modify: docs/deployment_setup.md
Modify: docs/README.md
Create: scripts/migrations/021_drop_feature_store_v1.sql
Remove v1 code imports and compatibility branches.
Drop v1 tables only after 14 consecutive production days with v2 serving enabled, no P0/P1 feature-store incidents, and passing daily feature validation.
Remove docs that describe legacy snapshot Feast retrieval as an acceptable training contract.
Update docs to state that training features must be point-in-time correct or excluded from the model contract.
Run a targeted search for legacy v1 Feast imports, legacy feature definition imports, entity-only FINAL feature-table reads, and obsolete snapshot-retrieval descriptions across recsys, scripts, features, docs, and tests.
The search must return no active v1 production paths.

10. Testing and Validation Strategy

Required test layers

Unit tests for contract validation, serialization, config parsing, and error handling.
Contract tests for Feast definitions, ClickHouse schemas, FeatureService names, and required fields.
Integration tests with real Feast, ClickHouse, Redis, and SQL registry.
Data quality checks for coverage, nulls, duplicates, timestamp monotonicity, and freshness.
Training export tests proving point-in-time joins.
Serving tests proving missing features are not fabricated.
Deployment guardrail tests proving production cannot start with local/file registry settings.
Performance tests proving online latency, offline throughput, publication throughput, and capacity headroom.
Recommendation-quality tests proving ranking metrics, candidate recall, result validity, segment quality, and experiment gates.

Required validation commands

Use narrow commands during implementation:

uv run pytest tests/unit/test_feature_store_contracts_v2.py -q
uv run pytest tests/contract/test_clickhouse_feature_store_v2_schema.py -q
uv run pytest tests/contract/test_feast_repo_v2.py -q
uv run pytest tests/integration/test_feast_point_in_time_v2.py -q
uv run pytest tests/integration/test_feast_clickhouse_redis_v2.py -q
uv run pytest tests/performance -m performance -q
uv run python scripts/capacity_plan_feature_store_v2.py \
  --active-members <members> \
  --active-items <items> \
  --active-cohorts <cohorts> \
  --member-payload-bytes <bytes> \
  --item-payload-bytes <bytes> \
  --cohort-payload-bytes <bytes> \
  --daily-member-rows <rows> \
  --daily-item-popularity-rows <rows> \
  --daily-item-metadata-rows <rows> \
  --daily-cohort-rows <rows> \
  --retention-days 730 \
  --output text
uv run pytest tests/unit/test_recommendation_metrics_v2.py tests/unit/test_recommendation_result_quality_v2.py tests/unit/test_experiment_gates_v2.py -q
uv run pytest tests/integration/test_recommendation_quality_pipeline_v2.py -q
uv run python scripts/evaluate_recommendation_quality_v2.py \
  --input artifacts/recommendation_quality/time/latest_vs_production.json \
  --candidate-model latest \
  --baseline production \
  --split time \
  --k 10

Use full validation before rollout:

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy recsys
uv run pytest tests/performance -m performance -q
uv run python scripts/capacity_plan_feature_store_v2.py \
  --active-members <members> \
  --active-items <items> \
  --active-cohorts <cohorts> \
  --member-payload-bytes <bytes> \
  --item-payload-bytes <bytes> \
  --cohort-payload-bytes <bytes> \
  --daily-member-rows <rows> \
  --daily-item-popularity-rows <rows> \
  --daily-item-metadata-rows <rows> \
  --daily-cohort-rows <rows> \
  --retention-days 730 \
  --output text
uv run pytest tests/integration/test_recommendation_quality_pipeline_v2.py -q
uv run python scripts/evaluate_recommendation_quality_v2.py \
  --input artifacts/recommendation_quality/time/latest_vs_production.json \
  --candidate-model latest \
  --baseline production \
  --split time \
  --k 10

The capacity planner intentionally requires measured inputs. The recommendation-quality CLI intentionally requires the time-based production-baseline artifact; a clean local checkout without that artifact must fail the release gate.

Non-negotiable correctness tests

For one member with snapshots at T1 and T2, a training event at T1 + 1 minute must retrieve the T1 feature row.
For the same member, a training event at T2 + 1 minute must retrieve the T2 feature row.
A training event before the first feature snapshot must return null or a documented default, and the export must account for coverage.
Online retrieval after publication must return the latest validated row.
A failed Redis/Feast publication must fail the compute job.
Missing feature_timestamp or created_timestamp must fail validation before data reaches Feast.
Online member, cohort, and 100-item batch lookups must meet the p95 and p99 latency SLOs in section 7.
Historical retrieval must sustain at least 50,000 entity rows per minute in engineering.
Publication must sustain at least 10,000 rows per minute per worker with attempted rows equal to published rows.
Recommendation lists must have zero duplicates, zero invalid item exposures, and zero unavailable item exposures in offline replay and engineering shadow traffic.
Candidate Recall@100, NDCG@10, MAP@10, and MRR@10 must not regress below the production baseline.
No critical segment may regress by more than 3% relative in NDCG@10, Recall@10, or valid-result rate.
Engineering shadow traffic must keep empty-result rate below 0.1% and fallback rate below 1% during healthy operation.

11. Migration and Rollout Plan

Backward compatibility is not required, but production safety still requires a controlled cutover.

Phase 1: Build v2 side by side

Create v2 tables and Feast definitions.
Keep v1 serving untouched while v2 data is generated.
Backfill v2 history for the most recent 730 days, or all available source history when less than 730 days exists.
Validate point-in-time correctness and coverage.
Build the first engineering capacity model and record baseline online latency, offline retrieval throughput, and publication throughput.
Build the first production-baseline recommendation-quality report using the current serving policy and a time-based holdout.

Phase 2: Shadow v2

Run v2 feature jobs on the production schedule.
Publish v2 online features to the production online store under v2 FeatureServices.
Add shadow reads from serving or a background verifier that compares v1 availability with v2 availability without influencing responses.
Record coverage and freshness metrics.
Run 30-minute engineering load tests at the peak request rate defined in section 7.
Run recommendation-quality replay on shadow outputs and enforce zero invalid, duplicate, or unavailable results.

Phase 3: Train and validate models on v2

Rebuild ranker/training datasets using v2 FeatureServices.
Record feature contract metadata in MLflow.
Compare offline metrics against leakage-resistant baselines.
Run canary serving with v2 feature provider.
Verify that training extraction meets the historical retrieval throughput SLO.
Require candidate recall, ranking metrics, and segment metrics to meet the section 8 gates before canary.

Phase 4: Cut over

Switch serving to v2 FeatureServices.
Make readiness depend on v2 feature health.
Disable v1 fallback paths.
Monitor online hit rate, feature age, missing feature rate, recommendation latency, model business metrics, and error rate.
Block cutover if online latency, Redis headroom, ClickHouse query duration, or publication throughput fails the section 7 gates.
Block cutover if recommendation-quality, result-validity, experiment, or segment guardrails fail the section 8 gates.

Phase 5: Remove v1

Delete v1 feature code paths.
Drop v1 tables through a migration after 14 consecutive production days with v2 serving enabled, no P0/P1 feature-store incidents, and passing daily feature validation.
Remove obsolete documentation and tests.

12. Production-Readiness Checklist

All Feast-backed offline tables preserve historical rows by entity and feature_timestamp.
Feast FeatureViews use historical v2 sources with event timestamp and created timestamp columns.
FeatureServices are versioned and model-specific.
Training/export jobs use Feast historical retrieval for all model input features.
Serving uses Feast online retrieval through typed v2 clients.
Production readiness fails when Feast registry, online store, or freshness checks fail.
Feature publication failures fail jobs and alert operators.
Feature jobs are deterministic for supplied --as-of timestamps.
Feature compute runs and publication attempts are audited.
MLflow model metadata records the feature contract.
Integration tests prove point-in-time correctness with multiple snapshots per entity.
Integration tests prove online/offline consistency.
Direct ClickHouse feature fallback is disabled by default in production.
Candidate sets are stored outside Feast.
Local registry state is not used for shared deployments.
Online feature provider p95 and p99 latency meet the section 7 SLOs.
Historical Feast retrieval sustains at least 50,000 entity rows per minute in engineering.
Online publication sustains at least 10,000 rows per minute per worker with no row loss.
Redis memory sizing includes measured payload size, projected key cardinality, replication, and 35% allocator overhead.
ClickHouse query plans show partition pruning for historical feature retrieval.
Incremental feature jobs complete within 30 minutes for three consecutive engineering runs.
Performance regression tests and capacity planning scripts are part of release validation.
Candidate Recall@100, NDCG@10, MAP@10, and MRR@10 meet or exceed the production baseline.
Duplicate item rate is 0 in offline replay and engineering shadow traffic.
Invalid or unavailable item exposure rate is 0 in offline replay and engineering shadow traffic.
Empty-result rate is below 0.1% and fallback rate is below 1% during healthy engineering operation.
No critical segment regresses by more than 3% relative in NDCG@10, Recall@10, or valid-result rate.
Recommendation logs include request ID, experiment ID, model version, FeatureService version, candidate source, score, rank, item ID, and timestamp.
Canary or A/B ramp gates include sample-ratio mismatch checks, primary online metrics, guardrail metrics, and segment regressions.
Full validation passes: tests, Ruff, format check, and mypy.

13. Risks and Trade-Offs

More storage in ClickHouse

Historical feature rows increase storage. This is the correct trade-off for leakage-free training. Use TTLs, partitions, and explicit retention windows rather than collapsing history.

More up-front migration work

Replacing the v1 feature pipeline is larger than patching FINAL queries. The current design cannot support true point-in-time joins, so smaller patches would preserve the most important correctness bug.

Stricter readiness may reduce availability during feature-store incidents

Failing readiness when required features are unavailable is intentional. Serving degraded personalized responses silently is worse than explicit degraded mode. A separate non-personalized fallback can remain available if product requirements demand it.

Feast type limitations may require feature reshaping

Some complex map and array payloads may be awkward in Feast online retrieval. Model input features should be scalars or small stable lists where possible. Large candidate payloads should move to a candidate store.

Backfill can expose source data quality issues

Historical backfills may reveal missing event fields, timestamp drift, or inconsistent IDs. Treat these as data quality defects, not reasons to weaken the feature contract.

Performance gates may delay rollout

Latency, throughput, and capacity gates can delay v2 cutover even after correctness tests pass. This is intentional. A feature platform that is correct but cannot meet production load is not production-ready.

Offline recommendation metrics can mislead

Offline ranking metrics are necessary but not sufficient. Time-based holdouts, online experiments, segment dashboards, and result-validity checks are all required because offline relevance labels can be biased by prior exposure and historical ranking policy.

Quality constraints can reduce short-term metric gains

Filtering unavailable items, enforcing diversity, protecting cold-start slices, and blocking segment regressions can reduce a single global metric. This is the correct trade-off when the product goal is reliable recommendation quality rather than narrow offline score improvement.

14. Final Recommendations

Treat this as a feature-platform v2 replacement, not a cleanup of the existing adapter.
Implement historical ClickHouse feature tables first; every other production guarantee depends on them.
Make Feast the only supported path for model input features in training and serving.
Split the monolithic adapter into focused, typed clients.
Remove silent fallback and fabricated freshness values.
Move candidate sets out of Feast.
Require real integration tests with Feast, ClickHouse, Redis, and multiple historical snapshots before cutover.
Treat performance budgets as release gates, not advisory metrics.
Treat personalization quality, result validity, candidate recall, ranking metrics, online guardrails, and segment outcomes as release gates.
Do not deploy v2 to production until point-in-time correctness, online publication reliability, freshness, readiness checks, online latency, offline throughput, capacity headroom, and recommendation-quality gates are proven by automated tests and engineering runs.

Authoritative References

Feast Feature Views: https://docs.feast.dev/getting-started/concepts/feature-view
Feast point-in-time joins: https://docs.feast.dev/getting-started/concepts/point-in-time-joins
Feast production guidance: https://docs.feast.dev/how-to-guides/running-feast-in-production
Feast local registry reference: https://docs.feast.dev/reference/registries/local
Feast push source reference: https://docs.feast.dev/reference/data-sources/push
ClickHouse query optimization: https://clickhouse.com/docs/optimize/query-optimization
ClickHouse primary indexes: https://clickhouse.com/docs/primary-indexes
ClickHouse partitions: https://clickhouse.com/docs/partitions
ClickHouse EXPLAIN statement: https://clickhouse.com/docs/sql-reference/statements/explain
Redis memory optimization: https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/
Redis key eviction: https://redis.io/docs/latest/develop/reference/eviction/
Microsoft Recommenders evaluation metrics: https://microsoft-recommenders.readthedocs.io/en/latest/evaluation.html
Spark MLlib ranking metrics: https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#ranking-systems
Google Rules of Machine Learning: https://developers.google.com/machine-learning/guides/rules-of-ml/

Keyboard shortcuts

HungryHub Knowledge Base