Personalization Queue Lag Runbook

Use this runbook when refresh jobs are not completing in time or queue pressure is rising.

When To Use

• Pending personalization requests stay unresolved.
• Queue depth grows faster than workers can drain.
• Worker failures or enqueue failures spike.

Signals

• RecSysPersonalizationQueueLagHigh
• RecSysPendingRefreshAgeHigh
• RecSysWorkerFailureRateHigh
• RecSysQueueEnqueueFailuresHigh
• RecSysRefreshTerminalFailureRateHigh

Immediate Actions (First 10 Minutes)

1. Confirm queue and worker health:
   • Queue depth metric: hh_lion_personalization_queue_depth
   • Worker outcomes: hh_lion_personalization_worker_events_total
2. Check Redis connectivity and saturation.
3. Pause canary traffic expansion while lag is unresolved.

Triage Steps

1. Validate worker process state:

uv run rq info --url redis://localhost:16379/0

2. Check worker logs for repeated terminal failures (artifact load, cache write, runtime errors).
3. Confirm serving startup and health endpoints:

curl -s http://<serving-host>/health/ready | jq

4. Inspect refresh outcomes:
   • timeout_terminalized
   • cache_error
   • corrupted

Recovery Actions

1. Scale worker replicas/concurrency.
2. Restart unhealthy workers.
3. If failures are model-related, switch to last known good model (see model rollback runbook).
4. If Redis is degraded, fail refresh requests cleanly and avoid unbounded pending growth.

Validation Checklist

1. Queue depth trend is flat or decreasing.
2. Pending age drops below threshold.
3. Worker failure ratio returns to baseline.
4. Refresh timeout terminalizations drop below alert threshold.

Related Runbooks

• Model Rollback Runbook
• ANN Index Staleness Runbook