Sidekiq Redis DB Per-Namespace Isolation (Staging)

Summary

On EKS staging, every namespace’s hh-server app and Sidekiq workers share one Redis host but must use a distinct Redis logical DB number so jobs enqueued in one namespace are not executed by another namespace’s workers.

When two namespaces share both the host and the DB number, their Sidekiq queues are literally the same Redis lists. A job enqueued by hh-engineering can be picked up and run by a hh-venus worker (and vice-versa), producing the classic symptom:

“Reservation data is created on hh-engineering but executed by hh-venus sidekiq … the reservation data is not found.” — and both /sidekiq/retries dashboards show identical data.

This blocks QA on staging because background work (reservations, mailers, etc.) crosses namespace boundaries non-deterministically.

Why changing the DB number alone did not fix it

The DB number is hardcoded in two independent layers that both have to agree:

1. Rails app — config/initializers/0_0_defined_constants.rb:

   db_num = is_development ? 10 : 11   # legacy: ALL non-dev namespaces → DB 11
   

   Staging deploys run with RAILS_ENV=production, so every staging namespace resolved to DB 11. Setting a different DB on the Redis side (or a different env var the app doesn’t read) had no effect, because the app ignored it and reconnected to DB 11.

2. KEDA ScaledObjects — manifest/base/staging/private/scaledobjects/*.yaml:

   databaseIndex: "11"   # autoscaler polls DB 11 for queue depth
   

   Even if the workers move to another DB, KEDA still polls DB 11, so autoscaling reads the wrong queue depth.

Both layers must point at the same DB for a namespace. Earlier ad-hoc edits left namespaces inconsistent (e.g. hh-ballbot app on DB 13 while its KEDA still polled DB 11; one sidekiq-google-reserve-lp ScaledObject polled DB 12 while its workers used DB 11). Those silent app≠KEDA mismatches are exactly what this fix removes.

On the old DigitalOcean staging cluster the same shared-DB setup did not mix jobs, because there was effectively a single staging app per Redis DB. After the EKS migration multiple namespaces (hh-engineering, hh-venus, hh-syn, hh-ballbot) run against the same Redis host concurrently, so the collision surfaced.

The fix: SIDEKIQ_DB_NUMBER

A single per-namespace value, SIDEKIQ_DB_NUMBER, now drives both layers from one place (the namespace’s buildspec-hh-<ns>.yaml):

Data flow per namespace build (CodeBuild):

buildspec env.variables: SIDEKIQ_DB_NUMBER="<n>"
        │
        ├─► echo "SIDEKIQ_DB_NUMBER=<n>" >> build_time_env ─► Dockerfile cp build_time_env .env
        │        └─► app boots, ENV.fetch('SIDEKIQ_DB_NUMBER') ─► workers connect to DB <n>
        │
        └─► set_env.sh: ["<REDIS_DB>"]="${SIDEKIQ_DB_NUMBER:-11}"
                 └─► sed substitutes <REDIS_DB> in ScaledObjects ─► KEDA polls DB <n>

Because app and KEDA derive from the same build variable, they can never drift.

If SIDEKIQ_DB_NUMBER is unset the app falls back to the legacy 10 (dev) / 11 (non-dev) behaviour and KEDA falls back to 11, so the change is backward compatible.

DB number assignment (staging namespaces)

Each namespace’s *-public and *-private sides share one DB number. Avoid DB 12 (used by the DigitalOcean-staging code path) and DB 15 (FirebaseIdToken cache, see below).

hh-ballbot keeps 13 because its app was already hand-moved to DB 13; this fix only realigns its KEDA databaseIndex to match.

Replication checklist (apply per branch/namespace)

Each staging namespace deploys from its own long-lived branch (hh-engineering, hh-venus, hh-syn, hh-ballbot) with its own copy of the manifests + buildspec. The fix must be applied to each branch with that namespace’s DB number:

1. App — config/initializers/0_0_defined_constants.rb, in the non-DO branch:

   default_db_num = is_development ? 10 : 11
   db_num = ENV.fetch('SIDEKIQ_DB_NUMBER', default_db_num)
   

2. KEDA — replace every hardcoded databaseIndex: "<n>" with databaseIndex: "<REDIS_DB>" in manifest/base/staging/private/scaledobjects/*.yaml and any manifest/overlays/staging/<ns>-private/{scaledobjects,patches}/*.yaml.
3. set_env.sh — in manifest/overlays/staging/<ns>-private/set_env.sh, add to the replacements map:

   ["<REDIS_DB>"]="${SIDEKIQ_DB_NUMBER:-<n>}"
   

   (<ns>-public has no ScaledObjects, so it does not need this.)
4. buildspec — in buildspec-hh-<ns>.yaml:
   • add SIDEKIQ_DB_NUMBER: "<n>" under env.variables
   • after the build_time_env SSM writes, append echo "SIDEKIQ_DB_NUMBER=${SIDEKIQ_DB_NUMBER}" >> build_time_env

Verify after deploy

# Workers: confirm the app connects to the expected DB
kubectl --context <eks-staging> -n <ns>-private exec deploy/hungryhub-sidekiq -- \
  ruby -e 'require "./config/environment"; puts REDIS_FOR_SIDEKIQ_CONFIG[:url]'

# KEDA: confirm databaseIndex matches
kubectl --context <eks-staging> -n <ns>-private get scaledobject sidekiq-default -o yaml \
  | grep databaseIndex

# Cross-check: the two /sidekiq/retries dashboards should now show DIFFERENT data
#   https://hh-engineering.my.id/sidekiq/retries
#   https://hh-venus.my.id/sidekiq/retries

Related / out of scope

• FirebaseIdToken (config/initializers/2_firebase_id_token.rb) uses the same shared Redis host on hardcoded DB 15. It is also shared across namespaces, but it is a token cache (not a job queue), so cross-namespace sharing does not mix jobs. Left as-is by this fix; revisit if token-cache isolation is ever needed.
• Flipper (config/initializers/3_flipper.rb) reuses REDIS_FOR_SIDEKIQ_CONFIG, so feature-flag state is now also isolated per namespace on staging — desirable.

See also: Sidekiq KEDA Autoscaling — Architecture & Tuning, Domain and Namespace List.