RSpec Debugging Guide

This document explains how to debug hanging RSpec tests and at_exit hook issues.

Quick Start

When your RSpec tests hang or get stuck:

1. Enable debug mode:

   RSPEC_DEBUG_HANGING=true bundle exec rspec
   

2. Find the process ID (it will be displayed when debug mode starts)

3. Send debugging signals:

   # Get thread backtraces (most useful for hanging tests)
   kill -USR1 <pid>
   
   # Get system/memory statistics
   kill -USR2 <pid>
   
   # Graceful shutdown with full debug info
   kill -INT <pid>  # or Ctrl+C
   

Environment Variables

Example Usage

Basic debugging of hanging tests:

RSPEC_DEBUG_HANGING=true bundle exec rspec spec/models/user_spec.rb

Debug with custom at_exit timeout:

RSPEC_DEBUG_HANGING=true RSPEC_AT_EXIT_TIMEOUT=60 bundle exec rspec

In CI/GitHub Actions:

- name: Run tests with hanging detection
  env:
    RSPEC_DEBUG_HANGING: true
    RSPEC_AT_EXIT_TIMEOUT: 120
  run: bundle exec rspec

Debug Output Explanation

Thread Backtraces (USR1 Signal)

🔴 MAIN THREAD
  ID: 70123456789012
  Status: sleep
  Priority: 0
  Alive: ✅ yes
  Backtrace (15 frames):
    0: 🟢 /workspaces/hh-server/spec/models/user_spec.rb:45:in `sleep'
    1: 🟢 /workspaces/hh-server/app/services/user_service.rb:23:in `process'
    2: 🟡 /gems/activerecord-6.1.7/lib/active_record/base.rb:123:in `transaction'

Thread indicators:

• 🔴 Main thread
• 🔵 Other threads
• 🟢 Application code (your project)
• 🟡 Gem code
• ⚪ Ruby core/stdlib

System Stats (USR2 Signal)

Shows:

• Process information (PID, memory usage)
• System memory status
• Ruby object counts
• Garbage collection statistics
• Open file descriptors

Common Hanging Scenarios

1. Database Transactions

Symptoms: Tests hang during database operations Debug: Look for threads stuck in ActiveRecord transaction code Solution: Check for unclosed transactions, deadlocks

2. Network Requests

Symptoms: Tests hang during HTTP requests Debug: Look for threads in net/http or faraday code Solution: Add timeouts, check network connectivity

3. Background Jobs

Symptoms: Tests hang during Sidekiq/background job processing Debug: Look for multiple threads in job processing code Solution: Use inline job processing in tests

4. File I/O

Symptoms: Tests hang during file operations Debug: Check file descriptor count, look for I/O operations in backtraces Solution: Ensure files are properly closed, check disk space

5. at_exit Hooks

Symptoms: Tests complete but process hangs during shutdown Debug: Automatic timeout warning will trigger after 30 seconds Solution: Review gems that register at_exit hooks

Manual Debugging in Test Code

You can also trigger debugging manually from within your tests:

# In a test file
require 'spec_helper'

RSpec.describe "Something" do
  it "debugs when needed" do
    # Manually dump thread information
    RSpecDebugger.dump_threads

    # Show system statistics
    RSpecDebugger.dump_system_stats

    # Both together
    RSpecDebugger.debug_current_state
  end
end

Troubleshooting

No debug output when sending signals

• Verify the process is still running: ps aux | grep rspec
• Check that you’re using the correct PID
• Ensure debug mode is enabled (RSPEC_DEBUG_HANGING=true)

Process still hangs after debugging

• Use kill -9 <pid> as last resort (loses debug info)
• Check for external dependencies (databases, Redis, etc.)
• Review test setup/teardown for resource cleanup

Debug output is too verbose

• Focus on the main thread first (🔴)
• Look for application code frames (🟢) vs gem code (🟡)
• Check the first few frames of the backtrace

Integration with APM

For production-like debugging, this system works well with the APM instrumentation:

# In your test
ElasticAPM.with_span('debug_hanging_test', 'test') do
  RSpecDebugger.debug_current_state
  # Your test code here
end

Performance Impact

• Minimal overhead when RSPEC_DEBUG_HANGING=false (default)
• Low overhead when enabled but not actively debugging
• Higher overhead only when signals are sent or timeouts trigger
• Safe for CI use - only activates when explicitly enabled