AWS SES Pause: Manual Notification Recovery
Purpose
Use this runbook when AWS SES is paused or degraded and reservation confirmation emails were not sent.
This process identifies impacted reservations and resends email notifications after SES recovers.
Scope
- System: hh-server
- Channel: Email only
- Notification targets:
- Reservation owner confirmation
- Reservation user confirmation
Current script source in hh-server:
- scripts/resend_notifications_dry_run.rb
When to use this runbook
Use this runbook if all conditions below are true:
- AWS SES incident is confirmed.
- Reservation confirmation emails were missed during the incident window.
- SES is healthy again and approved for resend.
Preconditions
- You have access to run Rails runner in production environment.
- You know the incident start and end timestamps in UTC.
- You have approval from on-call lead (or incident commander) before actual resend.
Safety notes
- Always run dry run first.
- Review dry run output and affected reservation sample before any resend.
- The resend flow sends email only.
- Re-running execution can cause duplicate emails. Track the execution window and operator.
Procedure
1) Confirm incident window
Define exact UTC timestamps:
- INCIDENT_START
- INCIDENT_END
Example:
- INCIDENT_START = 2026-05-27 07:56:23 UTC
- INCIDENT_END = 2026-05-28 06:24:00 UTC
2) Prepare script for this incident
In hh-server, open:
- scripts/resend_notifications_dry_run.rb
Update:
- INCIDENT_START
- INCIDENT_END
Confirm query criteria still match the incident:
- Reservation.reached_goal_scope
- created_at between incident start and end
- reservation date/time is today or future
3) Run dry run
From hh-server app context:
RAILS_ENV=production bundle exec rails runner scripts/resend_notifications_dry_run.rb
Before approving execution mode, verify environment output is production:
RAILS_ENV=production bundle exec rails runner 'puts Rails.env'
Review output:
- Total reservations matching criteria
- Detailed preview sample
- Summary by restaurant
Approval checkpoint before execution:
- Counts are reasonable for incident duration.
- Sample rows look valid.
- No obvious out-of-window records.
- Incident lead approves send.
4) Execute resend (only after approval)
In the same script, switch:
- dry_run = true
- to
- dry_run = false
Run again:
RAILS_ENV=production bundle exec rails runner scripts/resend_notifications_dry_run.rb
Expected execution behavior:
- Sends owner confirmation email via NotificationService::Email
- Sends user confirmation email via NotificationService::Email
- Prints counters:
- Emails to owner
- Emails to user
- Failed
5) Verify results
Collect and share in incident channel:
- Incident window used (UTC).
- Total matched reservations from dry run.
- Execution counters (owner sent, user sent, failed).
- Any failures with reservation IDs and error messages.
- Script has been reverted to dry-run mode after completion.
If failures exist:
- Retry only failed reservations after root cause is known.
- Log exact reservation IDs retried.
- Confirm no duplicate resend on successful reservations.
Immediately after execution, revert local script changes:
- Set dry_run back to true.
- Discard local edit if no longer needed.
6) Close-out checklist
- SES status is healthy.
- Dry run evidence captured.
- Execution approved and completed.
- Script reverted to dry-run mode.
- Failure list resolved or tracked.
- Incident summary posted to team channel.
- Follow-up action created to prevent recurrence.
Incident communication template
Use this message in incident channel:
[SES Recovery] Manual notification resend completed
- Incident window (UTC): <start> to <end>
- Dry run matched reservations: <count>
- Sent to owner: <count>
- Sent to user: <count>
- Failed: <count>
- Notes: <errors or N/A>
Improvements recommended
For future incidents, consider adding a dedicated execution script with:
- CLI flags for dry-run and execution mode
- CSV export of impacted reservation IDs
- Idempotency guard to prevent duplicate sends
- Structured logs for audit and postmortem