Sidekiq KEDA Autoscaling — Architecture & Tuning Guide

Architecture

Sidekiq workers run as Kubernetes Deployments scaled by KEDA ScaledObject resources. KEDA watches Redis list lengths (queue depths) and adjusts replica counts.

Redis queue:critical (list length)
        │
        ▼
KEDA ScaledObject (sidekiq-critical)
        │  pollingInterval: 10s
        │  listLength trigger: 1 job → scale up
        ▼
Deployment: hungryhub-sidekiq-critical
        │  minReplicaCount: 3
        │  maxReplicaCount: 10
        ▼
Sidekiq pods → process MarkTransactionAsPaidWorker, MyFirebaseWorker

ScaledObject Locations

Critical Queue Workers

Workers on :critical queue (must never be backlogged):

These two workers are the direct path from payment webhook → user sees confirmation. Any latency here causes payment stuck-loading on hh-pegasus.

Known Incident: Payment Stuck-Loading (2026-05-31)

Root cause: KEDA minReplicaCount: 1 + cold-start latency (~30-60s). When sidekiq-critical pods were idle and a payment webhook arrived, KEDA took one full polling interval to detect the job and another to spin up a pod. During this window MarkTransactionAsPaidWorker and MyFirebaseWorker sat queued — Firebase never updated — hh-pegasus loading screen hung.

Mitigation applied (2026-06-01):

• minReplicaCount bumped from 1 → 3 (PR #8160)
• Ensures 3 workers always available, eliminates cold-start for normal traffic

Root cause of original value: listLength: 30 was set by Cahyanto in PR #7358 (commit ea8b56fa, 2025-10-28) as a copy-paste during namespace restructure with no intentional tuning.

Recommended ScaledObject Config (Critical) — TODO

These changes are not yet applied. Open task:

spec:
  pollingInterval: 10        # was 30 — detect jobs faster
  cooldownPeriod: 300
  minReplicaCount: 3         # done — was 1
  maxReplicaCount: 10
  triggers:
    - type: redis
      metadata:
        listName: queue:critical
        listLength: "1"      # was 30 — scale on first job
        databaseIndex: "11"

Grafana Monitoring

Dashboard: Sidekiq Dashboard (uid SwCwV7qkz)

Key panel: Critical — sidekiq_queue_latency_seconds{name="critical"}

Alert: fires when min_over_time(sidekiq_queue_latency_seconds{name="critical"}[2m]) > threshold for 1 minute pending period. Alert managed by Budi (last updated 2026-03-04).

Observed pattern (2026-06-01): latency spikes are short (≤9s) and isolated — two spikes at ~12:00 and ~01:10 in a 24h window. This confirms cold-start, not sustained overload. With minReplicaCount: 3 these spikes should disappear.

Prometheus exporter: sidekiq-metrics-sidekiq-prometheus-exporter.monitoring.svc.cluster.local:80

Other Queues