RFC: Refactor Homepage Serving and Evaluation for Production-Grade Personalization

Status: Draft Owner: Recommendation Platform / HH-Lion Last Updated: 2026-04-24

1. Executive Summary

HH-Lion’s current recommendation stack has a solid retrieval and model-ops foundation, but the homepage serving decision layer is preventing improved retrieval models from translating into better homepage outcomes.

Confirmed current behavior

Across multiple official production evaluations:

retrieval metrics improved versus baseline
homepage metrics regressed versus baseline
top homepage misses remained dominated by source=cohort
the repeated cohort-heavy failures persisted even after several serving-side fixes

Latest confirmed example:

candidate run: ed69431670334926b5a366a23b81cbf6
retrieval improved
homepage still regressed
top homepage misses remained repeated cohort items

Core conclusion

The current issue is not primarily retriever quality.

The most likely root cause is a combination of:

serving-path personalization eligibility / fallback behavior causing many warm-user requests to resolve into cohort-heavy output before personalized signal can dominate
evaluation-path mismatch, where the official homepage evaluation exercises a constrained foreground path with sparse, uniform context rather than the full production experience we actually care about
over-entanglement between eligibility, candidate generation, fusion, fallback, reranking, and section assembly, which makes the system hard to reason about and hard to debug

Recommendation

Do not replace the current model-training and retrieval foundation.

Instead, perform a major refactor of the homepage serving and evaluation layer:

separate eligibility, candidate generation, blending, and section assembly into distinct stages
introduce explicit stage-level diagnostics and promotion gating
split evaluation into foreground-fast-path and full-personalization modes
make warm-user homepage sections explicitly model-seeded instead of allowing open-ended cohort competition

2. Scope

This RFC covers:

homepage serving architecture
homepage evaluation architecture
request-stage diagnostics and observability
fallback behavior for warm users
source blending / section assembly rules

This RFC does not propose replacing:

ClickHouse as canonical interaction store
MLflow model lifecycle
FAISS ANN retrieval foundation
GitHub Actions / OIDC production workflow split
current production training / evaluation / promotion / rollback workflow structure

3. Evidence Base

This RFC is based on the repository code and official documentation.

Repository code reviewed

docs/project_overview.md
recsys/serving/homepage_pipeline.py
recsys/serving/homepage_orchestrator.py
recsys/serving/homepage_pyfunc.py
recsys/serving/runtime_builder.py
recsys/serving/models.py
recsys/retrieval/model_retriever.py
recsys/retrieval/merger.py
recsys/retrieval/cohort_retriever.py
recsys/features/service.py
recsys/serving/feature_adapter.py
recsys/ranking/strategies.py
recsys/ranking/catboost_ranker.py
scripts/evaluate.py
scripts/train.py
recsys/contracts/model_signatures.py

Official documentation reviewed

MLflow PyFunc documentation: https://mlflow.org/docs/latest/api_reference/python_api/mlflow.pyfunc.html
Feast documentation: https://docs.feast.dev/
Faiss documentation: https://faiss.ai/

Evidence gaps

The following are not conclusively proven from code alone and should be treated as hypotheses until instrumented:

exact percentage of warm-user requests that fail personalization eligibility in live production
exact percentage of requests that reach model retrieval but later fall back to cohort-dominant section output
precise contribution of GrowthBook config to the observed production behavior
exact relationship between live /v2/homepage + /refresh usage and the current official evaluation path

4. Current Architecture Summary

The current system is a hybrid multi-stage recommender.

4.1 Retrieval layer

Personalized retrieval

Two-Tower embeddings
FAISS ANN search
ModelRetriever maps serving user IDs to model user indices
results are canonicalized into serving-safe public restaurant IDs

Code:

recsys/retrieval/model_retriever.py

This is aligned with Faiss’s documented role as an ANN library for dense vector similarity search.

Additional candidate sources

session-based retrieval
cohort popularity retrieval
new-items retrieval
search-intent retrieval

Code:

recsys/serving/homepage_pipeline.py
recsys/retrieval/cohort_retriever.py

4.2 Candidate merge layer

Multiple source candidate lists are merged using Reciprocal Rank Fusion (RRF), with optional source weighting.

Code:

recsys/retrieval/merger.py

4.3 Reranking layer

After merge, the system may rerank via:

CatBoost ranker when available
fallback heuristic reranking otherwise

Code:

recsys/ranking/catboost_ranker.py
recsys/ranking/strategies.py

4.4 Serving orchestration layer

The request orchestration determines whether a user receives:

personalized
cohort_popularity
fallback behavior

based on:

member feature lookup success
feature freshness
member threshold status
whether runtime artifacts can personalize the user
latency budget

Code:

recsys/serving/homepage_orchestrator.py

4.5 Packaging / model serving layer

The homepage stack is packaged as a custom MLflow PyFunc model.

Code:

recsys/serving/homepage_pyfunc.py

This is supported by official MLflow PyFunc documentation. However, MLflow now recommends “models from code” as a simpler long-term direction for custom models.

5. Confirmed Failure Pattern

The repeated production evaluation pattern is:

retrieval metrics improve
homepage metrics regress
homepage debug output remains cohort-dominated

Important confirmed examples

Candidate `9eb19eec69e045abbd0a2b4091f77069`

retrieval improved
homepage regressed
not promoted

Candidate `10b549e5fa9d4d028854ee2a24ef11ed`

retrieval improved materially
homepage still regressed
top homepage misses remained cohort-dominated
not promoted

Candidate `ea17e982af34422280ed62696e373697`

retrieval improved materially
homepage got worse
top homepage misses remained cohort-dominated
not promoted

Candidate `2a040f647c614394a3dc11d701c1eabd`

retrieval improved materially
homepage still regressed
top homepage misses remained cohort-dominated
not promoted

Candidate `ed69431670334926b5a366a23b81cbf6`

retrieval improved materially
homepage still regressed
top homepage misses remained cohort-dominated
not promoted

Conclusion from the pattern

The retriever repeatedly produces candidates that beat baseline on retrieval metrics, but the homepage-serving stack does not preserve or surface that quality.

6. Root Cause Analysis

6.1 Root cause A: homepage evaluation is not production-faithful

The official homepage evaluation path in scripts/evaluate.py explicitly sets:

ENVIRONMENT=development
ENABLE_SESSION_CANDIDATES=false
ENABLE_NEW_ITEMS_CANDIDATES=false
PRODUCTION_SAFE_MODE=false

It also constructs a uniform sparse request context for all users:

geo = TH-1
language = en
device_type = mobile
referrer_category = direct
search_query = None
background_personalization = False

This means the current promotion gate does not evaluate the full production homepage experience. It evaluates a constrained foreground path under simplified request assumptions.

Why this matters

If cohort retrieval is keyed by geo/language/referrer, then many requests sharing the same context will naturally produce the same cohort results whenever personalized routing fails or weakens.

That matches the observed repeated cohort items in homepage debug examples.

Root-cause statement

The current official homepage gate is testing a particular serving mode, not necessarily the most important production mode.

6.2 Root cause B: warm-user personalization eligibility is brittle and binary

Current serving orchestration effectively requires all of the following for the request to remain personalized:

member features available
member features not stale
member profile meets threshold
runtime model artifacts can personalize the user

If any of those checks fail, the system moves toward COHORT_POPULARITY.

Architectural problem

This makes serving eligibility significantly stricter than offline holdout eligibility.

Offline evaluation users are chosen by historical interaction criteria, but online serving eligibility depends on additional runtime and freshness conditions. Those two notions of “eligible user” are not explicitly unified.

Result

A user can be eligible in offline evaluation logic but still fail to get durable personalized serving treatment online.

6.3 Root cause C: source weighting and top-window rebalance were treating a symptom, not the failure point

Several focused serving fixes were already attempted:

stronger warm-user source weights
top-window preservation of model-backed items
broader promotion of model candidates from deeper in the ranked list
skipping cohort retrieval on the warm fast path when model candidates exist

Yet the homepage debug output still showed repeated cohort-dominated misses.

What this proves

The primary issue is probably not just merge weighting.

More likely failure points are:

user never truly reaches personalized mode
model candidate availability is low or absent at serve time
another fallback / assembly stage still surfaces cohort-heavy output
current official evaluation does not exercise the mode where personalized refresh is strongest

6.4 Root cause D: too much control-flow coupling in the homepage-serving path

The current flow entangles:

personalization eligibility
feature freshness decisions
candidate generation
optional-stage skipping due to latency budget
source merge
reranking
warm-user rebalance
fallback section generation
section assembly

This makes it hard to answer simple production questions such as:

Was the user eligible for personalization?
Did model retrieval execute?
How many model candidates were produced?
Did reranking run?
Did the request fall back because of freshness, timeout, model mapping, or candidate emptiness?
Why did Recommended For You end up cohort-heavy?

This is an observability and maintainability problem, not just a tuning problem.

7. Best-Practice Gap Analysis

7.1 What is aligned with best practice

Hybrid multi-stage recommendation architecture

The repo already follows a reasonable industry pattern:

dense personalized retrieval
heuristic fallback retrieval
candidate fusion
reranking
business constraints
graceful degradation

This is a valid production architecture.

FAISS for dense retrieval

Using FAISS for ANN over learned item vectors is appropriate and aligned with official FAISS documentation.

Feature store usage

Using Feast and a serving feature layer to bridge offline and online features is aligned with official Feast guidance around consistent training/serving access.

MLflow PyFunc packaging

Packaging a custom homepage model via MLflow PyFunc is valid and supported by official docs.

7.2 Where the implementation deviates from best practice

Gap 1: eligibility and fallback are too binary for warm users

Current behavior is effectively:

fully personalized, or
cohort fallback

For a production homepage, this is too coarse.

Best practice is a staged degrade ladder, for example:

full personalized
personalized-lite
session-boosted personalized
cohort backfill
cohort primary
global fallback

Gap 2: source blending is too heuristic and weakly calibrated

RRF is a good bootstrap technique, but it is not a sufficient long-term strategy for warm-user homepage quality when multiple source families compete heavily.

For warm users, Recommended For You should not be left to open-ended competition between model and cohort sources.

Gap 3: the final homepage objective is not first-class in the architecture

Training improves retriever quality, but promotion is blocked by the final homepage experience after serving logic, fallback logic, and section assembly.

That means the promoted artifact is not being judged on a fully aligned end-to-end target.

Gap 4: evaluation parity is poor

The official gate does not mirror the main production behavior we likely care about. This creates architectural ambiguity around what “good enough to promote” actually means.

Gap 5: insufficient stage-level observability

Current diagnostics improved but still do not expose enough control-flow truth to isolate the dominant failure point quickly.

8. Target Architecture

Refactor the homepage-serving system into explicit stages with explicit contracts.

8.1 Stage 1: Personalization eligibility engine

Introduce an explicit eligibility result object:

EligibilityDecision {
  mode: FULL_PERSONALIZED | PERSONALIZED_LITE | SESSION_PERSONALIZED | COHORT | GLOBAL_FALLBACK
  reason: <enum>
  member_features_source: feast | clickhouse | cohort | missing
  member_features_stale: bool
  model_mapping_available: bool
  request_budget_seconds: float
}

Design goals

separate routing decisions from candidate generation
make every downgrade reason explicit
unify online eligibility semantics with evaluation reporting

8.2 Stage 2: Candidate generation layer

Return distinct candidate pools rather than a single blended list.

CandidatePools {
  model_candidates
  session_candidates
  cohort_candidates
  new_items_candidates
  intent_candidates
}

Design goals

preserve stage-level transparency
make candidate availability measurable
enable quota- or policy-based section construction

8.3 Stage 3: Candidate scoring / blending layer

For warm personalized requests, replace open-ended source competition with explicit policy:

Recommended For You must be model-seeded
cohort can backfill only when model candidate supply is insufficient
session can boost but should not erase the model-backed seed set
blending behavior should be explicit and measurable

Short-term implementation option

Keep RRF as a fallback blend mechanism, but introduce deterministic section policies for warm users.

Long-term implementation option

Train a final-stage scorer / blender that uses:

retriever score
source type
source count
user eligibility mode
request context
item popularity
availability / quality signals
session-intent features
freshness signals

8.4 Stage 4: Section assembly layer

Section assembly should consume already-scored candidates and explicit section policies.

For example:

Recommended For You: model-seeded personalized section
Trending Near You: cohort / popularity section
New Arrivals: new-items section
Based on Your Taste: session / cuisine / model blend section

Do not let section identity emerge accidentally from whichever source wins a heuristic merge.

9. Evaluation Refactor

Split homepage evaluation into two official modes.

9.1 Foreground Fast-Path Evaluation

Purpose:

measure the low-latency synchronous homepage path
validate minimal acceptable quality under tight serving budgets

Properties:

foreground request budget
optional stage skipping allowed
no assumption of background refresh completion

9.2 Full Personalization Evaluation

Purpose:

measure the full production-quality homepage experience for warm users
use the serving path that best reflects what product intends users to see

Properties:

allow richer feature access
exercise background / refresh-compatible personalized behavior where applicable
use more representative request contexts instead of one uniform context for every user

9.3 Promotion policy

Promotion should use the evaluation mode that reflects the intended production UX.

If live product behavior depends materially on post-response personalization or richer serving context, that behavior must be represented in the official promotion gate.

10. Required New Diagnostics

Before changing more ranking logic, add first-class request-stage metrics and debug payloads.

Per-request / per-eval-example fields

eligibility_mode
eligibility_reason
member_features_source
member_features_stale
model_mapping_available
model_candidates_count
session_candidates_count
cohort_candidates_count
new_items_candidates_count
intent_candidates_count
merge_strategy_used
reranker_used
reranker_fallback_used
for_you_top_10_source_counts
for_you_top_10_model_share
for_you_top_10_cohort_share
section_strategies
fallback_triggered
fallback_reason
timed_out

Aggregated evaluation summary fields

% eval users in FULL_PERSONALIZED
% eval users in PERSONALIZED_LITE
% eval users downgraded due to stale features
% eval users downgraded due to user not in model mapping
% eval users with zero model candidates
% eval users where reranking ran
% eval users where Recommended For You top-10 is >= 50% cohort

These metrics should be included in:

evaluation workflow summaries
production model-state reporting where practical
optionally promotion workflow summaries

11. Refactor Plan

Phase 0: Freeze unsafe hypothesis churn

Stop repeated source-weight tweaks until stage-level diagnostics are available.

Deliverables:

no more merge-weight-only PRs without stage-level evidence

Phase 1: Observability / truth extraction

Deliverables:

add explicit EligibilityDecision
add candidate-pool counts
add per-section source-mix reporting
expose fallback reasons in evaluation summaries

Success criteria:

we can explain exactly why a given warm-user example became cohort-heavy

Phase 2: Evaluation parity split

Deliverables:

new official Evaluate Production Candidate (Fast Path) workflow
new official Evaluate Production Candidate (Full Personalization) workflow
docs clarifying which evaluation controls promotion

Success criteria:

no ambiguity about what experience is being evaluated

Phase 3: Serving architecture extraction

Deliverables:

separate modules for:
- eligibility
- candidate generation
- blending
- section assembly
shrink homepage_orchestrator.py and homepage_pipeline.py responsibilities

Success criteria:

each stage has an explicit interface and testable contract

Phase 4: Warm-user section contract

Deliverables:

make Recommended For You explicitly model-seeded
define cohort-backfill thresholds
add tests for minimum model-backed presence in warm-user top-N

Success criteria:

cohort cannot dominate warm-user Recommended For You unless model supply is provably insufficient

Phase 5: Final-stage scorer / calibrated blending

Deliverables:

either:
- explicit calibrated blending rules, or
- learned final-stage scorer using source-aware features

Success criteria:

homepage metrics improve without sacrificing serving stability

Phase 6: Packaging cleanup (optional follow-on)

Deliverables:

assess migration from current PyFunc packaging toward MLflow “models from code”

Success criteria:

simpler artifact inspection and lower packaging ambiguity

12. Testing Strategy

Unit tests

Add tests covering:

eligibility downgrade reasons
candidate-pool counts by source
warm-user model-seeded section rules
cohort-backfill-only behavior when model supply is insufficient
explicit fallback reason propagation

Integration tests

Add serving tests covering:

foreground personalized request with full eligibility
foreground request with stale member features
request with model mapping miss
request with zero model candidates but valid member history
full-personalization evaluation path

Workflow guardrails

Add tests to enforce:

official evaluation workflow separation
expected diagnostics in step summaries
no accidental drift back to opaque evaluation behavior

Implement EligibilityDecision and candidate-pool diagnostics
Add evaluation summaries for:
- eligibility mode distribution
- candidate-pool counts
- fallback reasons
Split official homepage evaluation into:
- fast path
- full personalization
Refactor Recommended For You to be model-seeded for warm eligible users
Re-run the official train/evaluate loop only after the new diagnostics are in place

15. Recommendation to Stakeholders

Do not throw away the current retrieval and model-ops foundation.

Do not continue shipping repeated warm-user source-weight tweaks without deeper diagnostics.

Approve a major refactor of the homepage serving and evaluation layer with the following principle:

For warm users, the production homepage must be explicitly and observably personalized, not merely allowed to become personalized if heuristic source competition happens to work.

This is the shortest path to a system that is:

scalable
debuggable
promotion-safe
aligned with actual homepage quality

Keyboard shortcuts

HungryHub Knowledge Base