Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

PRD — Availability Cache-Warming (“Processing”) Indicator

Status: Drafted from grilling; ready for review Branch: continues perf/hhserver-vendor-availability-major Tracking: EPIC: Availability cache-warming (PROCESSING) indicator #8317 - tracked on PR #8274 Audience: HungryHub Engineering + CS-Ops Relationship to prior work: Direct continuation of the shipped bookability diagnosis feature (docs/PRD-bookability-diagnosis.md) — adds a third verdict state to its cache-vs-config attribution.

1. Problem

When someone edits availability-affecting config (agenda days, package active/visible/dates, pricing, reservation duration), the availability cache is recomputed asynchronously. For a short window the diagnosis page (and the customer surface) can show availability that does not yet reflect the edit.

Today the bookability diagnosis page classifies a not-bookable result as either a CACHE BUG (fresh DB says bookable, cache disagrees) or a CONFIG MISTAKE (fresh DB itself says not bookable). It has no concept of “still recomputing.” So a page viewed mid-warm — where the cache legitimately hasn’t caught up yet — is falsely attributed as a CACHE BUG, generating exactly the kind of false ticket the diagnosis feature was built to eliminate.

We want to surface a third, honest state: “⏳ Recomputing availability — this is expected, not a bug. Started Ns ago.”

2. Grounded reality (what the codebase already does)

This materially de-risks the feature versus a naive “build async warming” plan:

  • Invalidation is already asynchronous via Kafka. On an availability- affecting save, update_inv_checker_restaurant_package_level_cache (app/models/hh_package/restaurant_package.rb:594, also agenda.rb:173, package_attr.rb:225, pricing.rb:148) checks a monitored field set — auto_extend, start_date, end_date, deleted_at, active, active_for_third_party_only, is_visible, is_visible_for_staff, reservation_duration — and, when one changed, fires Karafka.producer.produce_async to EVENTS::INVENTORY::CACHE::TOPIC.
  • A consumer already re-warms. Mirage::CacheConsumer (karafka.rb:79, consumer group inventory_service, 2 partitions) consumes that topic and re-warms the cache.
  • Warming primitives exist. VendorAvailability::BulkPrecomputer.warm_dates / warm_range; ReadService also warms inline on read-miss as a fallback; find_available_date (legacy cache) vs calc_find_available_date (fresh DB truth) are the two sides the diagnosis already compares.

Conclusion: the async “processing window” (edit → Kafka event → Mirage::CacheConsumer re-warm) already exists. This feature surfaces that window; it does not build async warming from scratch. The only new backend mechanism is a lightweight warm-status record so the UI can tell whether a re-warm is in flight.

3. Goal

Make the bookability diagnosis page (admin/CS) correctly show a PROCESSING state while a config-triggered re-warm is in flight, and flip to up to date when it completes — so a mid-warm page is never mis-attributed as a cache bug.

4. Primary user & surface

  • Primary: CS-Ops / admin on the bookability diagnosis page, who currently risk reading a mid-warm page as a cache bug.
  • v1 surface: admin diagnosis page only. Customer-facing “recomputing” UI is a non-goal for v1 (flagged as future).

5. Solution

5.1 Warm-status record (the one new backend mechanism)

A small status entry, written when a cache-invalidation event is produced and updated when the consumer finishes:

  • Key shape: availability_warm_status:{restaurant_id}:{slug|all} (Redis hash recommended — cheap, TTL-able, no migration). Fields: state (pending|processing|done|failed), window (date range or “default”), enqueued_at, started_at, finished_at, source (which model/field triggered it), kafka_offset|job_id (for debugging).
  • Lifecycle:
    1. Producer side (update_inv_checker_restaurant_package_level_cache): on producing the Kafka event, write state: pending, enqueued_at: now.
    2. Consumer side (Mirage::CacheConsumer): on pickup set processing, started_at; on successful warm set done, finished_at; on error failed.
  • TTL / stuck semantics: status key TTL ~15 min. If state is pending|processing but enqueued_at is older than a stuck threshold (e.g. 2 min), the UI shows “taking longer than expected” rather than spinning forever. failed surfaces a distinct “warm failed — refresh or escalate.”
  • Idempotency: writing status must not break the existing Kafka path if it errors (wrap in the same safe_call/APM-report pattern used in the presenter).

Note: if Redis status proves insufficient (e.g. need history/audit), fall back to a DB row; v1 uses Redis for zero-migration speed.

5.2 Third diagnosis state: PROCESSING

Extend Admin::BookabilityPresenter#attribution:

  • Before classifying CACHE BUG vs CONFIG MISTAKE, check the warm-status record.
  • If state ∈ {pending, processing} (and within the stuck threshold) and the cache disagrees with fresh → return PROCESSING, not CACHE BUG.
  • If fresh and cache agree → normal verdict (warming irrelevant).
  • If failed or stuck → a distinct “warm stalled” advisory (still not a silent cache-bug claim).

5.3 UI (reuse PR #7960’s pattern only)

  • A banner on the diagnosis page: “⏳ Recomputing availability for this restaurant (started Ns ago). Values below may not reflect your latest edit yet.”
  • A polling JS controller (modeled on #7960’s inventory_override_controller.js, built fresh & minimal) that hits a small JSON endpoint returning the warm-status for (restaurant, slug?) and flips the banner to “✓ Up to date” when done.
  • Endpoint: GET /admin/restaurants/:restaurant_id/availability_warm_status (admin-namespaced, read-only JSON).

5.4 Synchronous fallback (safety)

ReadService already warms inline on read-miss. We do not remove that. The indicator is informational; if the async warm is slow, a read still self-warms, so the customer path is never slower or staler than today.

6. Acceptance criteria

  1. After editing a monitored field (e.g. closing Mondays on an agenda — the tested flow), the diagnosis page for an affected date shows PROCESSING within X seconds (target ≤ the produce→consume latency, ~a few seconds).
  2. When Mirage::CacheConsumer finishes the re-warm, the page (via polling) flips to up to date without a manual reload.
  3. A mid-warm not-bookable page is never attributed as CACHE BUG.
  4. A stuck/failed warm shows a distinct “taking longer / failed” state, not an infinite spinner and not a false CACHE BUG.
  5. Sync fallback intact: disabling the indicator must not change cache freshness or read latency vs today.
  6. No regression to the existing Kafka producer / Mirage::CacheConsumer warm path — status writes are additive and failure-isolated.
  7. Status writes/reads add no measurable latency to the edit-save path or the diagnosis page load (Redis hash get/set only).

7. Non-goals (v1)

  • Not changing the availability algorithm or the Kafka/Mirage warm pipeline.
  • Not the write-side inventory-override batch progress (that is PR #7960’s scope — see §9).
  • No customer-facing “recomputing” UI (future).
  • No new Sidekiq worker unless §8 shows the Kafka consumer can’t host the status write (see decision in §8).

8. Open decision — Sidekiq vs Kafka-consumer for status

The grilling assumed Sidekiq. The codebase shows the warm already runs in the Karafka consumer, not Sidekiq. Recommendation: write the status from the existing producer + Mirage::CacheConsumer (no new Sidekiq worker) — fewer moving parts, and the status naturally tracks the actual warm. Only introduce a Sidekiq worker if status must be written from a context where Karafka isn’t available. This must be confirmed during implementation (engineer to verify Mirage::CacheConsumer can write the status cheaply and idempotently).

9. Relationship to PR #7960

PR #7960 (feat/inv-progress, 46 files, +1411/−1364) is a write-side inventory-override batch-progress feature (InventoryBulkOverrideWorker, override grid, batch columns on updated_inventories). It is a different feature from this read-side freshness indicator. The user is not confident in it.

Recommendation: Close #7960 and build this focused feature. Reuse only the UI pattern (status record + polling controller + banner). If write-side override progress is still wanted, scope it separately later, smaller.

10. Risks

  • Status/warm divergence: if the consumer crashes after marking processing but before done, status could be stuck — mitigated by TTL + stuck threshold.
  • Partition ordering: 2 Kafka partitions mean events for one restaurant could interleave; key status per (restaurant, slug) and treat the latest enqueued_at as authoritative.
  • Over-signaling: showing PROCESSING too eagerly (for warms that finish in ms) is noise; only show it when the cache actually disagrees with fresh AND a warm is in flight.

11. Source-of-truth file map

  • Producer/trigger: app/models/hh_package/restaurant_package.rb:594, agenda.rb:173, package_attr.rb:225, pricing.rb:148
  • Kafka topic: EVENTS::INVENTORY::CACHE::TOPIC; consumer Mirage::CacheConsumer (karafka.rb:79)
  • Warm primitives: app/services/vendor_availability/bulk_precomputer.rb, read_service.rb
  • Diagnosis attribution to extend: app/presenters/admin/bookability_presenter.rb (attribution)
  • UI pattern reference (do not merge): PR #7960 app/javascript/controllers/admin/inventory_override_controller.js