Inventory Cache Consistency: Architecture & Solutions

Last Updated: July 30, 2026
Status: Implementation Guide

Table of Contents

1. Executive Summary
2. Business Requirements
3. Architecture Overview
4. Root Cause Analysis
5. Solutions Implemented
6. Performance Considerations
7. Troubleshooting Guide

Executive Summary

This document describes the inventory cache consistency architecture for the HH booking system, addressing the critical issue of stale seat_left cache data that can lead to overbooking or false “sold out” errors.

Key Decisions

• DB Updates: Synchronous (within transaction)
• Cache Updates: Asynchronous via Kafka (performance over 100% consistency)
• Authoritative Source: Database row locks in inv_lvl1_available?
• Cache Strategy: Eventual consistency with async refresh

Design Principles

1. Performance First: Cache updates are async to avoid blocking DB transactions
2. DB is Truth: Row-level locks prevent overbooking regardless of cache state
3. Eventual Consistency: Cache staleness acceptable for 100ms-2s window
4. Single Responsibility: UpdateMirageCache concern handles ALL cache invalidation

Business Requirements

With Locking System (End User Flow)

User booking journey with inventory checks:

1. User Checks Availability
   └── Cache Level Check (fast, might be stale)
       └── app/services/inventory/inv_checker_hungry_hub_service.rb

2. Create Temporary Booking
   └── app/services/reservation_service/init/hungry_hub.rb
       ├── inv_lvl2_available? → Cache check (skip_seat_left_check: true)
       └── inv_lvl1_available? → DB check with row lock (AUTHORITATIVE)

3. Payment Flow
   ├── With Payment: trigger payment → waiting → webhook → confirmed
   └── Without Payment: temporary → confirmed immediately

Without Locking System (Admin/Staff Flow)

Internal booking without availability checks:

Admin/Staff Create Booking
   └── app/controllers/api/dashboard/reservations_controller.rb#create
       └── app/services/reservation_service/create.rb (no availability checks)

Architecture Overview

Two-Layer Data System

Data Flow Pattern

┌─────────────────────────────────────────────────────────────────┐
│                         BOOKING FLOW                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. Cache Check (inv_lvl2_available?)                           │
│     └── Fast pre-validation (skip seat_left check)              │
│                                                                  │
│  2. DB Lock (inv_lvl1_available?) ← AUTHORITATIVE               │
│     └── Row-level lock → Update total_booked_seat → Save        │
│                                                                  │
│  3. Transaction Commits                                          │
│     └── after_save: update_inv_relations                        │
│         └── Recalculate inventory.total_booked_seat             │
│                                                                  │
│  4. Cache Invalidation (after_commit)                           │
│     └── UpdateMirageCache.clear_mirage_cache                    │
│         └── Kafka Event → CacheConsumer → UpdateCacheWorker     │
│             └── Async Redis update (eventual consistency)       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Event-Driven Architecture

Kafka Integration:

• Topic: EVENTS::INVENTORY::CACHE::TOPIC
• Event Type: EVENTS::INVENTORY::CACHE::TYPES::RESET_CACHE
• Consumer: Mirage::CacheConsumer (future microservice)
• Worker: Inventory::UpdateCacheWorker
• Queue: :inventory

Root Cause Analysis

Problem Statement

Admin dashboard showed DB seat_left = 0 but cache seat_left = 1 for hours after bookings.

8 Identified Root Causes

1. Kafka Message Loss/Delay

• Network issues, consumer lag, broker failures
• Producer produce_async may fail silently
• Worker queue congestion or Sidekiq failures

2. Callback Condition Bypass

inv_attributes_changed? only monitors specific attributes:

ATTRIBUTES_TO_MONITOR = %w[active ack adult date start_time end_time
                           for_locking_system is_temporary is_valid_reservation]

Missed scenarios: Direct DB updates, update_columns, bulk operations

3. Race Conditions

Multiple concurrent bookings updating same inventory:

User A books → Worker A reads DB → User B books → Worker A caches stale value

4. Callback Execution Order

Both after_commit callbacks run, but order matters:

• update_inv_relations updates DB
• clear_mirage_cache sends Kafka event
• If read before write completes = stale data

5. State Transition Edge Cases

for_locking_system transitions (temporary → confirmed → expired) may not trigger cache updates properly.

6. Time-Spanning Calculation Errors

include_datetimes_around_to_clear calculates affected time slots:

duration = end_time - start_time
affected_times = (duration / 15.minutes).to_i + 1

Incorrect if package’s duration changed.

7. Direct Database Updates

calc_total_booked_seat bypassed if:

• SQL updates via admin tools
• Migrations
• Manual console operations

8. Long Cache TTL

expire_at = (date.to_date + 1.day).end_of_day
ttl_seconds = (expire_at - Time.current).to_i

For future dates (9+ days away), stale data persists.

9. find_available_date Batched vs Legacy Key Divergence (fixed in #8520)

Layer note: Root causes 1–8 above concern the reservation-level seat_left cache. This one is a separate layer — the per-date bookability payload (find_available_date), which caches availability / seat_left / min_seat / max_seat / booked_seat for a whole date, not a single time slot.

There are two independent reads of the same date-bookability data, and they use two different Redis keys:

Key builders diverge on purpose (find_available_date_cache_key vs find_available_date_cache_namespace in app/services/inventory/inv_checker_hungry_hub_service/find_available_dates.rb) — the batched path packs many dates into one hash for a single hmget.

The bug: the post-booking / inventory-change refresh (set_by_restaurant_level, set_by_reservation_level → set_find_available_date) wrote only the legacy per-date key. The batched shared-namespace key was orphaned from every invalidation path — it self-warms only on a cache miss, so a stale hit persisted until its midnight+1day TTL.

Symptom (prod, restaurant 5647, 2026-07-30): the bookability diagnosis page (/admin/restaurants/:id/inventories?advanced=1) showed the “New cache (ReadService)” payload as NOT_BOOKABLE_ATTRIBUTE (availability:false, seat_left:0, min_seat:0, max_seat:0) while both the fresh DB recompute and the legacy cache said availability:true, seat_left:10. Result: customers could not book a genuinely-available date. A stale NOT_BOOKABLE_ATTRIBUTE is recognisable because a genuinely sold-out date keeps min_seat:1 / max_seat:1000000 — the all-zeros min/max shape is the default constant, not a real computation.

Solutions Implemented

Architecture Decision: Async-Only Cache Updates

Design: Cache updates are handled EXCLUSIVELY by the UpdateMirageCache concern via Kafka events in after_commit callbacks.

Rationale:

1. Performance: Cache updates can take 50-200ms; blocking DB transaction is unacceptable
2. DB Authoritative: inv_lvl1_available? row locks prevent overbooking regardless of cache state
3. Eventual Consistency: 100ms-2s cache staleness is acceptable business tradeoff
4. Single Source of Truth: One mechanism reduces complexity and bugs

Implementation Details

1. DB Updates (Synchronous - within after_commit)

Location: lib/model_ext/reservations/callbacks.rb

# IMPORTANT: Runs in after_commit (AFTER transaction commits)
# This is required because MySQL InnoDB FK constraint checks only see committed data.
# When Reservation.save! is called inside an outer transaction (e.g., create_temporary),
# the reservation row is not visible for FK validation until the outer transaction commits.
after_commit(:update_inv_relations)

def update_inv_relations
  # Guard: Verify reservation still exists (handles race conditions)
  return unless self.class.exists?(id)
  
  # Update inventory_reservations table
  # Recalculate inventory.total_booked_seat with row locks
  # FK constraint now succeeds because reservation is committed and visible
end

Why after_commit (not after_save)?

• MySQL InnoDB checks FK constraints against committed data only
• When save! is called inside an outer transaction, the parent row isn’t visible to FK checks
• after_commit ensures the reservation is fully committed before child records are inserted

2. Cache Invalidation (Asynchronous)

Location: app/models/concerns/reservations/update_mirage_cache.rb

# Runs AFTER transaction commits successfully
after_commit :clear_mirage_cache

def clear_mirage_cache
  return if @datetimes_to_clear.blank?

  # Fire Kafka event - consumed by Mirage::CacheConsumer
  Karafka.producer.produce_async(
    topic: EVENTS::INVENTORY::CACHE::TOPIC,
    payload: {
      event_type: EVENTS::INVENTORY::CACHE::TYPES::RESET_CACHE,
      restaurant_id: restaurant.id,
      reservation_level: true,
      datetimes: @datetimes_to_clear,
      # ... additional context
    }.to_json
  )
end

3. Cache Update Worker

Location: engines/mirage/app/consumers/mirage/cache_consumer.rb → app/workers/inventory/update_cache_worker.rb

# Processes Kafka events
def consume
  messages.each do |message|
    case message.payload['event_type']
    when EVENTS::INVENTORY::CACHE::TYPES::RESET_CACHE
      ::Inventory::UpdateCacheWorker.perform_async(
        payload['restaurant_id'], payload, debug_payload
      )
    end
  end
end

4. Stale Cache Detection & Auto-Refresh

Location: app/services/reservation_service/init/hungry_hub.rb

When inv_lvl1_available? (DB check) fails after inv_lvl2_available? (cache check) passed, it indicates potential cache staleness. The system fires a rate-limited Kafka event to trigger cache refresh.

def fire_stale_cache_refresh_event(start_times)
  return if reservation.restaurant.use_third_party_inventory?

  # Rate limit: one event per restaurant/date/service_type every 5 seconds
  rate_limit_key = "stale_cache_refresh:#{restaurant_id}:#{date}:#{service_type}"

  # Use Redis SETNX for distributed rate limiting
  redis = Redis.current
  rate_limited = !redis.set(rate_limit_key, '1', nx: true, ex: 5)
  return if rate_limited

  # Fire Kafka event to trigger cache refresh
  Karafka.producer.produce_async(
    topic: EVENTS::INVENTORY::CACHE::TOPIC,
    payload: {
      event_type: EVENTS::INVENTORY::CACHE::TYPES::RESET_CACHE,
      restaurant_id: reservation.restaurant_id,
      reservation_level: true,
      datetimes: datetimes,
      trigger_source: 'stale_cache_detected',
      # ... additional context
    }.to_json
  )
end

Rate Limiting Strategy:

• Key: stale_cache_refresh:{restaurant_id}:{date}:{service_type}
• TTL: 5 seconds
• Method: Redis SET NX (distributed lock)
• Purpose: Prevent Kafka flooding when multiple users hit same stale cache

Trigger Flow:

User passes inv_lvl2 (cache) → Fails inv_lvl1 (DB) → Stale detected
    ↓
Check rate limit (Redis SETNX)
    ↓
If not rate-limited → Fire Kafka RESET_CACHE event
    ↓
Mirage::CacheConsumer → Inventory::UpdateCacheWorker → Cache refreshed

5. Utility Worker (Manual Recovery)

Location: app/workers/inventory/refresh_seat_lefts_worker.rb

# For manual cache refresh or ops/debugging
class Inventory::RefreshSeatLeftsWorker < ApplicationWorker
  sidekiq_options queue: :inventory, retry: 3

  def perform(params)
    # Updates restaurant-level AND all package-level caches
    Inventory::InvCheckerHungryHubService.set_by_restaurant_level(...)
    restaurant_packages.each do |package|
      Inventory::InvCheckerHungryHubService.set_by_restaurant_package_level(...)
    end
  end
end

Usage:

# Manual cache refresh for specific time slots
Inventory::RefreshSeatLeftsWorker.perform_async({
  restaurant_id: 123,
  date: '2025-12-21',
  start_times: ['17:00', '17:15', '17:30'],
  service_type: 'dine_in'
})

6. Dual-Key Write for find_available_date (#8520)

Location: app/services/inventory/inv_checker_hungry_hub_service/find_available_dates.rb (+ DataStore#write_mapped_field in .../data_store.rb)

Fixes root cause #9. set_find_available_date now writes both keys on every refresh — the legacy per-date key and the batched shared-namespace field — so a booking/inventory change can no longer leave the batched cache stale.

def set_find_available_date(date, slug = nil)
  computed_data = calc_find_available_date(date, slug)

  # Legacy per-date key (date embedded in the key)
  data_store.mapped_hmset(find_available_date_cache_key(date, slug), [date], date) do |data|
    data[date] = computed_data
  end

  # Batched shared-namespace field (date is the hash field). Written WITHOUT
  # resetting the shared hash TTL — the hash spans many dates, so resetting
  # expiry to one date's midnight would prematurely drop the others.
  data_store.write_mapped_field(find_available_date_cache_namespace(slug), date, computed_data)

  computed_data
end

DataStore#write_mapped_field HSETs the single field with the same MSGPACK packing the batched reader decodes, and deliberately does not call expire (unlike mapped_hmset) so unrelated future dates in the shared hash keep their TTL.

Retro-fix of already-stale entries (runbook): the code fix is forward-only — entries already stale in Redis at deploy time self-heal only on the next booking/inventory change or at TTL. To clear them proactively, HDEL only the provably-stale batched fields (batched = NOT_BOOKABLE-default and the legacy per-date key for the same date = availability:true). This leaves genuine sold-outs and all correct entries untouched; each deleted field lazily re-warms to DB truth on the next read. On 2026-07-30 this cleared 586 stale fields across 28 keys out of ~1.03M total date entries (the rest were correct or genuinely not-bookable). Do not blanket-delete all *:find_available_date:by_hmset keys — that nukes ~962k correct entries and triggers a needless re-warm storm.

Performance Considerations

Latency Characteristics

Race Window Analysis

Timeline of Concurrent Bookings:

User A: [DB Lock] → [Update] → [Commit] → [Kafka Event]
User B: ............[Wait for A's lock]......[DB Lock] → [Update] → [Commit]
Cache:  [Stale]................................[Worker processes A]...[Worker processes B]
                                              ↑
                                     100ms-2s window where B sees stale cache

Why This Is OK:

• User B’s inv_lvl1_available? checks DB with row lock (authoritative)
• Even if cache shows seat_left = 1, DB lock ensures accurate availability
• Cache is for pre-filtering, not final decision

Under High Load (Black Friday Scenario)

Troubleshooting Guide

Diagnostic: Compare DB vs Cache

# In Rails console
restaurant = Restaurant.find(6559)
date = Date.parse('2025-12-21')
start_time = '17:00'

# 1. Check DB value (authoritative)
inventory = restaurant.inventories.find_by(date: date, start_time: start_time)
db_seat_left = inventory.seat_left
puts "DB seat_left: #{db_seat_left}"

# 2. Check cache value
inv_checker = InvCheckerFactory.new(restaurant.id, restaurant.time_zone).create_inv_checker_service
inv_checker.for_dine_in = true
cached_seat_left = inv_checker.seat_left(date, start_time)
puts "Cache seat_left: #{cached_seat_left}"

# 3. Force cache refresh
Inventory::RefreshSeatLeftsWorker.perform_async({
  restaurant_id: restaurant.id,
  date: date.to_s,
  start_times: [start_time],
  service_type: 'dine_in'
})

Investigation Checklist

When cache staleness is reported:

1. Check Kafka logs: Were RESET_CACHE events sent?

   # Search Kafka logs for restaurant and date
   grep "restaurant_id.*6559" kafka.log | grep "2025-12-21"
   

2. Check Sidekiq logs: Did Inventory::UpdateCacheWorker run?

   # In Rails console
   Sidekiq::Stats.new.queues['inventory']
   # Check for backed-up jobs
   

3. Check reservation audit: Which reservations affected this time slot?

   Reservation.where(restaurant_id: 6559, date: '2025-12-21')
     .where('start_time <= ? AND end_time >= ?', '17:00', '17:00')
     .order(created_at: :desc)
   

4. Verify UpdateMirageCache triggered:

   # Check recent reservation
   r = Reservation.last
   r.saved_changes # Should include monitored attributes
   r.instance_variable_get(:@datetimes_to_clear) # Should have datetimes
   

Common Issues & Fixes

Monitoring & Alerts

Recommended Metrics

1. Cache Miss Rate: How often is cache empty when it should have data?
2. Staleness Duration: Time between DB update and cache update
3. Kafka Consumer Lag: Messages pending in topic
4. Worker Queue Depth: Jobs in :inventory queue
5. Discrepancy Count: DB vs cache mismatches per hour

Alert Thresholds

# Example monitoring code
class InventoryCacheMonitor
  def check_staleness(restaurant_id, date, start_time)
    db_value = fetch_db_seat_left(restaurant_id, date, start_time)
    cache_value = fetch_cache_seat_left(restaurant_id, date, start_time)

    if db_value != cache_value
      alert = {
        type: 'cache_staleness',
        restaurant_id: restaurant_id,
        date: date,
        start_time: start_time,
        db_value: db_value,
        cache_value: cache_value,
        staleness: cache_value - db_value
      }

      # Alert if stale for > 5 minutes
      if staleness_duration > 5.minutes
        APMErrorHandler.report('Cache staleness detected', alert)
      end
    end
  end
end

Future Improvements

Short Term (Q1 2026)

1. Add cache refresh API endpoint for manual recovery by ops team
2. Implement cache versioning to handle concurrent updates
3. Add idempotency keys to Kafka messages

Long Term (Q2-Q3 2026)

1. Extract Mirage to microservice for independent scaling
2. Consider Redis Cluster for high availability
3. Implement write-through cache for critical time slots (next 24h)
4. Add read-through cache with DB fallback on miss

References

• Original investigation: docs/note.md (8 root causes identified)
• Architecture analysis: docs/note2.md (solutions implemented)
• Performance analysis: docs/note3.md (Black Friday approach)
• Business requirements: docs/note4.md (booking flows)

Document Owner: Engineering Team
Last Review: July 30, 2026
Next Review: October 2026