Batched Warm-Up Failure Modes Design

Refactor (completed in #8351): Move VendorAvailability::BulkPrecomputer into Inventory::InvCheckerHungryHubService::FindAvailableDates. The migration landed in commit 3f0b015f65; this doc is retained as design rationale for the failure-mode rescue strategy that ships in bulk_fetch_find_available_dates.

Author: Staff Engineer
Date: 2026-07-06
Status: Implemented (see #8351 for the final code; this doc is design rationale, kept for the failure-mode inventory and rescue strategy that survived into the implementation).

1. Failure Mode Inventory

The batched warm-up path (FindAvailableDates#bulk_fetch_find_available_dates) differs from the per-date path (find_available_date) in a critical way: one SQL query fetches inventory for N dates, whereas the per-date path executes N separate queries. If the batched query fails, all N dates fail at once — whereas per-date failures are isolated.

1.1 Database Layer Failures

Failure Mode	Trigger	Per-Date Behavior	Batched Behavior	Blast Radius
DB connection drop mid-query	Network partition, MySQL restart, Pod crash	After date X fails, dates X+1…N retry on next request (fresh connection)	Entire query aborts; no dates written to Redis	N dates (entire batch)
Query timeout	`ActiveRecord::StatementTimeout` — large date range + slow DB	Date X times out, dates X+1…N continue on retry	Query aborted; partial results (if any) discarded	N dates
SQL syntax error	Malformed query (unlikely in production, possible in code changes)	N/A (each query is tested)	N/A	N dates if query never runs
Deadlock	Concurrent batched queries on same table	N/A (single-row locks per date)	Both queries fail; no dates warmed	N dates × concurrent callers

1.2 Redis Layer Failures

Failure Mode	Trigger	Per-Date Behavior	Batched Behavior	Blast Radius
Redis connection drop	Network partition, Redis failover	Date X fails → `fetch_mapped_hmget` falls back to default, returns `NOT_BOOKABLE_ATTRIBUTE`	Pipeline fails mid-way; partial writes may occur	N dates (partial or full failure)
Redis write timeout	Slow Redis (memory pressure, large payload)	Date X fails → `mapped_hmset` fails silently; caller gets default	Same — `mapped_hmset` may raise or return partial	N dates (partial)
MessagePack serialization failure	Corrupt data in computed payload	N/A (only reads, not writes)	If `calc_find_available_date_payload` produces invalid object → `dump_data` raises	N dates if all dates fail

1.3 Application Logic Failures

Failure Mode	Trigger	Per-Date Behavior	Batched Behavior	Blast Radius
`restaurant_packages` empty	Package deleted mid-warm-up	Returns `NOT_BOOKABLE_ATTRIBUTE` for that date	Same — computed payload reflects empty packages	1 date
`seat_lefts` helper fails	Downstream dependency error	Returns default 0 seat_left → `NOT_BOOKABLE_ATTRIBUTE`	Same — computation uses fallback	1 date per date
`res_durations_is_enough?` returns unexpected type	Code bug	Returns `false` equivalent → date unavailable	Same — consistent failure	1 date per date
Empty result set (no inventory for date range)	Valid state — restaurant has no inventory	Returns `NOT_BOOKABLE_ATTRIBUTE` for each date	Same — batched SQL returns empty rows → all dates get not_bookable	N dates (expected state)

1.4 N+1 and Collisions (Not Applicable)

The batched path eliminates the N+1 problem by design. No collision risk exists because:

Each date writes to a distinct Redis hash field (HMGET/HMSET per date)
The batched mapped_hmset call at data_store.rb:61-86 writes atomically per invocation

Rescue: rescue ActiveRecord::StatementInvalid, ActiveRecord::StatementTimeout, Mysql2::Error::ConnectionError
Fallback: Log error at Rails.logger.error, re-raise as Inventory::WarmUpFailed (new custom error class)
Per-date fallback: Let the caller (the V1 restaurants controller’s find_available_dates_without_start_time path and the bookability presenter) handle the warm-up result normally; the read path iterates the returned hash as usual. Note: the final implementation chose silent degradation over the WarmUpFailed signal described here — see §5.2.4 for the decision.
Location: In bulk_fetch_find_available_dates (the warm-up method)

Redis Connection Drop / Timeout / BaseError

Rescue: rescue Redis::BaseError, Redis::TimeoutError, Errno::ECONNREFUSED
Fallback: Log error via Rails.logger.error, swallow silently — the read path will fall back to per-date computation
Per-date fallback: No retry needed; fetch_mapped_hmget in the read path handles Redis failures gracefully (see helper.rb:121-126 pattern: rescue StandardError → return default)
Location: In the second rescue clause of bulk_fetch_find_available_dates (sibling to the DB rescue)

MessagePack Serialization Failure

Rescue: rescue MessagePack::PackError, TypeError
Fallback: Log error, skip that date’s write, continue with remaining dates
Per-date fallback: Skip failed dates; let read path compute them individually
Location: Inside the per-date loop in bulk_fetch_find_available_dates

Application Logic Exceptions (TypeError, NoMethodError)

Rescue: rescue TypeError, NoMethodError, StandardError
Fallback: Log via APMErrorHandler.report (consistent with helper.rb:123-126), return default payload
Per-date fallback: Skip failed dates; read path falls back naturally
Location: Wrap the computation block

2.3 New Custom Error Class

# app/services/inventory/errors.rb
module Inventory
  class WarmUpFailed < StandardError
    attr_reader :dates, :cause

    def initialize(dates, cause)
      @dates = dates
      @cause = cause
      super("Warm-up failed for #{dates.count} dates: #{cause.message}")
    end
  end
end

2.4 Implementation Location

Method	Responsibility
`bulk_fetch_find_available_dates` (entry)	Catches `WarmUpFailed`, logs, returns empty result
`_inventories_batch`	Catches DB errors, wraps in `WarmUpFailed`
`data_store.mapped_hmset` (inside loop)	Catches Redis errors, continues with remaining dates

3. Test Cases

3.1 Database Layer

Test Name	Setup	Expected Behavior
`#bulk_fetch_find_available_dates raises on DB connection drop`	Stub `_inventories_batch` to raise `Mysql2::Error::ConnectionError`	Outer rescue logs via `APMErrorHandler`, returns a hash with `NOT_BOOKABLE_ATTRIBUTE` for every date in the window
`#bulk_fetch_find_available_dates raises on query timeout`	Stub to raise `ActiveRecord::StatementTimeout`	Outer rescue logs, returns NOT_BOOKABLE-shaped hash (same as above)
`#bulk_fetch_find_available_dates raises on StatementInvalid`	Stub to raise `ActiveRecord::StatementInvalid`	Outer rescue logs, returns NOT_BOOKABLE-shaped hash (same as above)
`#bulk_fetch_find_available_dates handles partial inventory (some dates empty)`	Return inventory only for dates[0], empty for dates[1..N]	Writes not_bookable for empty dates, computed for others

3.2 Redis Layer

Test Name	Setup	Expected Behavior
`#bulk_fetch_find_available_dates continues on Redis connection failure mid-write`	Stub `data_store.mapped_hmset` to raise `Redis::BaseError` after first date	Logs error, continues, returns N-1 dates warmed
`#bulk_fetch_find_available_dates handles MessagePack serialization failure`	Stub `dump_data` to raise `MessagePack::PackError`	Skips that date, continues with remaining
`#bulk_fetch_find_available_dates handles Redis timeout`	Stub to raise `Redis::TimeoutError`	Logs error, returns 0 or partial count

3.3 Application Logic

Test Name	Setup	Expected Behavior
`#bulk_fetch_find_available_dates handles empty restaurant_packages`	Stub `restaurant_packages_for_calc` to return []	Writes not_bookable for all dates
`#bulk_fetch_find_available_dates handles seat_lefts returning nil`	Stub `seat_lefts` to return nil	Uses fallback 0, date is not_bookable
`#bulk_fetch_find_available_dates handles res_durations_is_enough? raising`	Stub to raise `NoMethodError`	Reports to APM, skips that date

3.4 Integration / Read Path

Test Name	Setup	Expected Behavior
`V1 restaurants controller returns 200 with NOT_BOOKABLE-shaped data when batched warm-up fails`	Stub `bulk_fetch_find_available_dates` to raise `Mysql2::Error::ConnectionError`; hit `GET /api/vendor/v1/restaurants/.../available_dates`	Controller returns 200; response body has the same shape as a no-inventory restaurant (every date `not_bookable`); no 5xx
`Bookability presenter degrades gracefully when batched warm-up fails`	Stub `checker.find_available_dates_without_start_time` to return all-`not_bookable` results	Presenter renders successfully; no exception bubbles up to the admin action

4. Observability Recommendations

4.1 Logging

Follow the pattern in helper.rb:123-126 and redis_deduplicator.rb:32-38:

Rails.logger.error('Batched warm-up failed', {
  restaurant_id: restaurant_id,
  date_range: dates.first..dates.last,
  date_count: dates.size,
  error_class: e.class.name,
  error_message: e.message,
})

4.2 APM Spans

Follow the pattern in inv_checker_hungry_hub_service.rb:115-117:

ElasticAPM.with_span('Batched warm-up', 'app.cache', subtype: 'inventory', action: 'warm_up') do |span|
  span.set_label(:restaurant_id, restaurant_id)
  span.set_label(:date_count, dates.size)
  span.set_label(:failure, true) if failed
end

4.3 Metrics

Metric	Type	Labels	Purpose
`inventory.warm_up.batch.success`	Counter	restaurant_id, slug	Track successful warm-ups
`inventory.warm_up.batch.failure`	Counter	restaurant_id, error_class	Track failure rate
`inventory.warm_up.batch.duration`	Histogram	restaurant_id	Track latency
`inventory.warm_up.batch.dates_warmed`	Gauge	restaurant_id	Track dates written

4.4 Existing Patterns to Follow

Span naming: action: 'warm_up', subtype: 'inventory' — consistent with update_cache_value at inv_checker_hungry_hub_service.rb:115
Error reporting: APMErrorHandler.report with context — same pattern as helper.rb:124
Logging: Rails.logger.error with structured hash — same as redis_deduplicator.rb:33-36

5. Open Questions for Team Lead

5.1 Product Decisions

Silent failure vs. alert: Should a batched warm-up failure trigger a PagerDuty alert, or is silent degradation acceptable since per-date fallback handles it?
- Recommendation: Silent degradation (current path) — warm-up is optimization, not correctness-critical.
Partial write behavior: If 15/20 dates succeed before Redis fails, should we:
- (a) Return 15, accept partial success?
- (b) Roll back and return 0, forcing full retry?
- Recommendation: (a) — partial success is better than full failure; the read path handles the missing 5 dates individually.
Failure impact on user response: If warm-up fails, does the user see different availability than if it succeeded?
- Answer from code: No (modulo the custom error class decision). The current implementation does NOT re-raise — bulk_fetch_find_available_dates catches DB/Redis errors and returns a hash with NOT_BOOKABLE_ATTRIBUTE for every date. The caller (vendor V1 controller’s find_available_dates_without_start_time path) iterates this hash as usual; the user sees availability data shaped exactly like the “no inventory” case, with no error surfaced. On the next request the cache TTL will have expired and the warm-up retries.

5.2 Technical Decisions

Custom error vs. re-raise original: Should we wrap DB/Redis errors in WarmUpFailed, or let them propagate?
- Recommendation: Wrap in WarmUpFailed — gives the caller a clear signal to fall back, without coupling to specific DB/Redis error classes.
- Decision (option B): NOT shipping Inventory::WarmUpFailed. The batched warm-up degrades silently to NOT_BOOKABLE_ATTRIBUTE for the whole window. Tradeoff: callers can’t distinguish “warm-up failed” from “no inventory”, but the read path stays available without any caller-side retry logic. Revisit if production telemetry shows silent degradation is hiding real issues.
Retry logic: Should we retry failed dates individually after a batch failure, or skip retry entirely?
- Recommendation: Skip explicit retry — the per-date fallback in find_available_date naturally retries on each user request. Explicit retry adds complexity with marginal benefit.
Testing scope: Where should failure-mode tests live alongside the existing find_available_dates_spec.rb?
- Recommendation: New spec: spec/services/inventory/inv_checker_hungry_hub_service/find_available_dates_batched_spec.rb — tests the refactored method in its new home.

6. References

app/services/inventory/inv_checker_hungry_hub_service/find_available_dates.rb — bulk_fetch_find_available_dates at line 92
app/services/inventory/inv_checker_hungry_hub_service/data_store.rb — mapped_hmset at line 61
app/services/inventory/inv_checker_hungry_hub_service/helper.rb — error handling patterns at lines 121-126
app/my_lib/redis_deduplicator.rb — Redis error handling at lines 32-38
app/controllers/api/vendor/v1/restaurants_controller.rb — direct caller (V1 controller)
app/presenters/admin/bookability_presenter.rb — direct caller (admin bookability)

HungryHub Knowledge Base