Response Cache Compression

Overview

The ResponseCacheConcern module now includes automatic compression for large cached values to minimize Redis/Valkey storage usage while maintaining optimal performance.

How It Works

Compression Strategy

• Threshold: Only data larger than 2KB is compressed
• Algorithm: Zlib compression with BEST_COMPRESSION level
• Marker: Compressed data is prefixed with ZLIB_COMPRESSED_V1: marker
• Smart Compression: Only applies compression if it actually reduces size

Backward Compatibility

The implementation is fully backward compatible with existing cached data:

1. Reading old cache: Uncompressed data is detected automatically and used as-is
2. Reading new cache: Compressed data is detected by marker and decompressed
3. Writing new cache: All new writes use the compression logic
4. Error handling: If decompression fails, falls back to original data

Performance Benefits

• Small data (<2KB): No compression overhead, same performance as before
• Large data (>2KB): Compression applied, significant storage savings
• Example compression ratios:
  • JSON data: 70-90% size reduction
  • HTML data: 60-80% size reduction
  • Repeated data: Up to 95% size reduction

Code Changes

Constants

COMPRESSION_THRESHOLD = 2.kilobytes
COMPRESSION_MARKER = "ZLIB_COMPRESSED_V1:".freeze

New Methods

1. compress_if_needed(data)

   • Compresses data if size > threshold
   • Adds marker to compressed data
   • Returns original if compression doesn’t reduce size

2. decompress_if_needed(data)

   • Detects compression marker
   • Decompresses if marker present
   • Returns original data if not compressed (backward compatible)
   • Handles decompression errors gracefully

Modified Methods

cache_in_redis now:

• Compresses data before storing in Redis
• Decompresses data when reading from Redis
• Tracks compression stats (original size, compressed size, ratio)
• Logs compression information in debug mode

Logging

When debug_params is provided, logs include compression metrics:

{
  cache_hit: false,
  compressed: true,
  original_size: 5432,
  stored_size: 1234,
  # ... other params
}

Debug logs also show:

[cacheable] render miss filling cache: response_cache:key (compressed: 5432 -> 1234 bytes, 77.3% saved)

Migration to Production

Zero-Downtime Deployment

The implementation supports zero-downtime deployment:

1. Deploy new code: Compression logic is backward compatible
2. Old cache: Continues to work, read as uncompressed
3. New cache: Written with compression
4. Gradual transition: Cache naturally transitions to compressed format as it’s refreshed

No Manual Migration Needed

• No need to clear existing cache
• No special deployment steps required
• Old cache entries expire naturally and are replaced with compressed versions

Testing

The implementation includes comprehensive test coverage:

• ✅ Small data is not compressed (performance)
• ✅ Large data is compressed efficiently
• ✅ Compressed data is decompressed correctly
• ✅ Backward compatibility with old cache
• ✅ Error handling for corrupted compressed data
• ✅ Integration with full cache flow
• ✅ Compression statistics tracking

Run tests:

bundle exec rspec spec/controllers/response_cache_concern_spec.rb

Storage Savings Example

For a typical API response with 10KB of JSON data:

Before compression:

• Storage: 10,240 bytes per key
• 10,000 keys = ~100MB

After compression:

• Storage: ~2,048 bytes per key (80% reduction)
• 10,000 keys = ~20MB

Savings: 80MB (80% reduction)

Monitoring

Monitor compression effectiveness in logs when debug mode is enabled:

my_response_cache cache_key, :json, debug_params: { endpoint: 'api/v1/menus' } do
  # your data
end

Look for log entries showing compression ratios to evaluate effectiveness.