How to Disable the Weekend Scale Scheduler for a Specific Weekend

Description

The dev environment in ap-southeast-1 runs a cost-saving weekend scale scheduler that scales EKS node groups to zero, stops RDS instances, and (optionally) snapshots/deletes/restores selected Valkey replication groups every Friday 21:00 → Monday 07:00 Bangkok. See the lambda module README for the full design.

Sometimes a teammate needs the dev environment kept active for a specific weekend — for example a customer demo, a load test, an external audit, or a debugging session that cannot be paused. This runbook documents the two ways to do that, in order of preference.

Scope: this runbook is dev only. Prod (202255947274) is never affected by the weekend scheduler.

Prerequisites

AWS CLI installed and authenticated to the sandbox account (079994049689):
```
aws sso login --profile sandbox
aws sts get-caller-identity --profile sandbox
```
Confirm the returned Account is 079994049689.
For the Terraform variable path: write access to the hungryhub-terraform repo and the ability to dispatch the terraform.yml workflow against dev.

Which path to use

Situation	Use
One-off weekend (most common)	Approach A — AWS CLI (5 minutes, no PR, no apply)
The disabled window is longer than 1-2 weeks, or you want it codified in Git	Approach B — Terraform variable (requires PR + apply)
The CLI approach failed or the schedule state is stuck	Open a ticket with DevOps lead (Saiqul Haq)

[Approach A] Disable the EventBridge Schedules via AWS CLI (recommended)

This disables the 6 schedules without destroying the underlying Lambda functions. Reversible in seconds.

List the schedules to confirm names:
```
aws --profile sandbox scheduler list-schedules \
  --name-prefix "dev-" --region ap-southeast-1 \
  --query "Schedules[].Name" --output table
```
Expected names:
- dev-eks-scale-to-zero
- dev-eks-scale-up
- dev-rds-stop
- dev-rds-start
- dev-valkey-snapshot-delete
- dev-valkey-restore
- (plus monthly-lambda-trigger for check-ecr-outdated — leave it alone)

Disable the 6 weekend schedules (run on Thursday evening or before Friday 14:00 UTC):

for SCHED in dev-eks-scale-to-zero dev-eks-scale-up \
             dev-rds-stop dev-rds-start \
             dev-valkey-snapshot-delete dev-valkey-restore; do
  aws --profile sandbox scheduler update-schedule \
    --name "$SCHED" --region ap-southeast-1 \
    --state DISABLED
  echo "Disabled $SCHED"
done

Verify all 6 are disabled:

aws --profile sandbox scheduler list-schedules \
  --region ap-southeast-1 \
  --query "Schedules[?State=='DISABLED'].[Name,State]" --output table

Communicate in #hh-devops: post a one-liner — “Dev weekend scheduler paused from <date> to <date> per request from @<teammate> for <reason>. Will re-enable Monday after standup.”

On the Monday after the disabled weekend, re-enable:

for SCHED in dev-eks-scale-to-zero dev-eks-scale-up \
             dev-rds-stop dev-rds-start \
             dev-valkey-snapshot-delete dev-valkey-restore; do
  aws --profile sandbox scheduler update-schedule \
    --name "$SCHED" --region ap-southeast-1 \
    --state ENABLED
done

The cron expressions are unchanged — Monday 07:00 Bangkok will fire normally next week.

[Approach B] Disable via the Terraform variable

Use this when the disabled window will last multiple weeks and you want the change to survive a teammate being on leave. Requires a PR.

Open a branch in hungryhub-terraform and edit env/dev.tfvars (ask DevOps if you don’t have access):
```
enable_weekend_schedules = false
```
The variable is already declared in lambda/variable.tf:14 and consumed in lambda/main.tf for the 6 weekend scheduler aws_scheduler_schedule resources. Setting it to false removes the 6 schedules via count.
Open a PR titled chore(lambda): disable weekend schedules for dev and request DevOps review.
After merge, dispatch the terraform.yml workflow against dev + lambda to apply.
To re-enable later, set the variable back to true and apply again.

Notes

Why not just delete the schedules via the AWS console? Possible, but the schedules are managed by Terraform — deleting them out-of-band will cause drift on the next terraform plan and a re-apply will recreate them mid-weekend.
Valkey restore plan in S3 is independent. The dev-scale-scheduler-role IAM role has a known gap: it is missing s3:GetObject on the Valkey restore-plan bucket (see follow-up issue). The restore Lambda will execute but fail to read the plan. If the team-mate also expects Valkey replication groups to come back up after the disabled window, file a separate ticket.
Escalation. If the AWS CLI approach fails with AccessDeniedException, your SSO role lacks scheduler:UpdateSchedule. Open a ticket with DevOps lead Saiqul Haq — do not run terraform destroy to force-stop the schedules.
Audit trail. The PR (Approach B) or the Slack post in #hh-devops (Approach A) is the audit trail. Either is sufficient.
What about the monthly check-ecr-outdated Lambda? It runs on the 1st of every month and is a separate concern. It is dev-only as of PR #365 and is unaffected by the weekend scheduler disable.

Keyboard shortcuts