ClickHouse Infrastructure Documentation

Overview

ClickHouse at HungryHub is deployed using ClickStack Helm Chart (v3.0.1) inside an EKS cluster. This stack includes ClickHouse Server, ClickHouse Keeper (ZooKeeper replacement), and HyperDX (observability dashboard).

Main file: hungryhub-terraform/helm-releases/clickhouse.tf

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        EKS Cluster                          │
│                                                             │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │  ClickHouse  │  │  ClickHouse  │  │     HyperDX       │  │
│  │   Server     │  │   Keeper     │  │   (UI Dashboard)  │  │
│  │  (port 8123  │  │  (port 9000  │  │   (port 3000)     │  │
│  │  native:9000)│  │  internal)   │  │                   │  │
│  └──────┬───────┘  └──────────────┘  └───────────────────┘  │
│         │                                                    │
│  ┌──────┴───────────────────────────────────┐                │
│  │        Kubernetes Services               │                │
│  │  clickhouse (ClusterIP)                  │                │
│  │  clickhouse-headless (ClusterIP=None)    │                │
│  └──────────────────────────────────────────┘                │
│         │                                                    │
│  ┌──────┴───────────────────────────────────┐                │
│  │     NetworkPolicy (restricted ingress)    │                │
│  │  - clickstack pods                       │                │
│  │  - hyperdx pods                          │                │
│  │  - additional workloads via variable     │                │
│  └──────────────────────────────────────────┘                │
└─────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    │                   │
              ┌─────▼─────┐     ┌──────▼──────┐
              │    ALB     │     │ Cloudflare  │
              │  (dev only)│     │  (prod)     │
              │ HTTPS:8123 │     │  Tunnel     │
              └────────────┘     └─────────────┘

Cluster Sizing

Resources

Kubernetes Resources

AWS Resources (dev only)

Secrets Management

ClickHouse credentials are managed through AWS Secrets Manager + Kubernetes Secrets:

Flow

AWS Secrets Manager (/hungryhub/{env})
  ├── CLICKSTACK_MONGO_URI_{DEV|PROD} → HyperDX MongoDB Atlas
  └── SENDGRID_API_KEY → HyperDX SMTP

Random Generation (Terraform)
  ├── clickhouse_password (admin) → K8s Secret: clickhouse-credentials
  └── clickhouse_default_password → K8s Secret: clickhouse-default-password

Rotation

• Admin password: Auto-generated (32 chars, special). Rotated when namespace or storage_class changes.
• Default password: Auto-generated (32 chars, special). Overridden via var.clickhouse_default_password from CI pipeline.
• Both passwords are not stored in Terraform state or git in plaintext.

Networking & Ingress

NetworkPolicy

Only the following pods may access ClickHouse:

• clickstack-clickhouse-clickhouse pods (namespace: clickhouse)
• hyperdx pods (namespace: clickhouse)
• Additional workloads via var.clickhouse_allowed_ingress_workloads

ALB Ingress (dev only)

• Host: clickhouse.hhstaging.dev
• TLS: ACM cert (*.hhstaging.dev) from route53 remote state
• Security Group: Inbound only from clickhouse_ingress_allowed_cidrs (default: WireGuard VPN)
• Access Logs: S3 bucket from s3 remote state

Prod Access

Prod uses Cloudflare Tunnel + Zero Trust, not ALB.

Remote State Dependencies

# helm-releases/provider.tf
terraform_remote_state:
  vpc:
    key: vpc/vpc-state.tfstate
    outputs: vpc_id, private_subnets, private_subnets_cidr, db_subnets_cidr, eks_cluster_name

  eks:
    key: eks/eks-state.tfstate
    outputs: eks_cluster_id

  route53:
    key: route53/route53-state.tfstate
    outputs: cloudflare_acm (ACM cert ARN)

  s3:
    key: s3/s3-state.tfstate
    outputs: alb_logs_s3_bucket

Helm Values Template

File: helm-releases/values/clickhouse.yaml.tft

Monitoring (dev only)

File: helm-releases/clickhouse-monitoring.tf

ServiceMonitor

• Scrapes ClickHouse native Prometheus endpoint (port 9363)
• Interval: 15s, timeout: 10s
• Target: clickstack-clickhouse-clickhouse-headless Service

PrometheusRule Alerts

Outputs

clickhouse_http_endpoint    = "http://clickstack-clickhouse.clickhouse.svc.cluster.local:8123"
clickhouse_native_endpoint  = "clickstack-clickhouse.clickhouse.svc.cluster.local:9000"
hyperdx_ui_endpoint         = "http://clickstack-hyperdx.clickhouse.svc.cluster.local:3000"
clickhouse_alb_dns_name     = ""  # (dev only, populated after ALB provisioned)
clickhouse_alb_zone_id      = ""  # (dev only)
clickhouse_alb_security_group_id = ""  # (dev only)

Variables

DMS (Database Migration Service)

Overview

DMS is used for two pipelines:

1. MySQL → Aurora MySQL (CDC replication)
2. MySQL → S3 Parquet (dev only — data pipeline)

Main file: hungryhub-terraform/dms/main.tf, dms_s3_pipeline.tf

Architecture

┌──────────────────────────────────────────────────────┐
│                     DMS Pipeline                     │
│                                                      │
│  ┌──────────┐     ┌──────────────┐     ┌──────────┐ │
│  │  MySQL   │────▶│ Replication  │────▶│  Target  │ │
│  │  Source   │     │  Instance    │     │          │ │
│  │  (RDS)   │     │  dms.t3.large│     │          │ │
│  └──────────┘     └──────────────┘     └──────────┘ │
│       │                                      │       │
│       │                                      ▼       │
│       │                              ┌──────────────┐│
│       │                              │ Aurora MySQL  ││
│       │                              │ (prod target) ││
│       │                              └──────────────┘│
│       │                                              │
│       │              ┌──────────────┐                │
│       └─────────────▶│  S3 Bucket   │ (dev only)     │
│                      │  dms-target  │                │
│                      │  (Parquet)   │                │
│                      └──────────────┘                │
└──────────────────────────────────────────────────────┘

DMS Replication Instance

Pipeline 1: MySQL → Aurora MySQL (CDC)

Source Endpoint

Target Endpoint

Replication Tasks

• start_replication_task = false (manual start)
• Task settings: task_settings.json

Event Subscription

• Categories: failure, state change, deletion, configuration change, creation
• SNS Topic: {env}-hungryhub-dms

Pipeline 2: MySQL → S3 Parquet (dev only)

Gated: var.environment == "dev"

Source Endpoint

Target Endpoint

S3 Output Settings

• Row group length: 10,000
• Data page size: 1 MB
• Dict page size limit: 1 MB
• Max file size: 1 MB
• CDC batch interval: 180s
• CDC min file size: 32 KB

Table Mappings

• 104 selection rules
• 4 column-type transformations (date → string(50)):
  • vouchers.expiry_date
  • updated_inventories.start_date
  • updated_inventories.end_date
  • reservations.date
• File: table_mappings_s3_pipeline.json

S3 Target Bucket

Bucket: dms-target-hungryhub-{env}-s3

IAM Roles

dms-s3 Policy

{
  "s3:PutObject", "s3:GetObject", "s3:DeleteObject",
  "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts",
  "s3:ListBucketMultipartUploads", "s3:PutObjectTagging"
}
# Resource: dms-target bucket/*

Security Groups

DMS SG ({env}-dms)

• Inbound: MySQL 3306 from private + db subnets
• Egress: All (0.0.0.0/0)

ClickHouse ALB SG (clickhouse-alb) — dev only

• Inbound: HTTPS 8123 from clickhouse_ingress_allowed_cidrs
• Egress: All (0.0.0.0/0)

RDS Connections

Source RDS (External)

Host: hh-production-db-master.cgfqflgzmmxy.ap-southeast-1.rds.amazonaws.com
Port: 3306
Engine: MySQL

Dev MySQL Proxy RDS

Host: hungryhub-dev-mysql-proxy.proxy-c2i8lks3rhzq.ap-southeast-1.rds.amazonaws.com
Port: 3306
Engine: MySQL

Target Aurora Cluster

Endpoint: data.terraform_remote_state.rds.outputs.mysql_rds_cluster_endpoint
User: data.terraform_remote_state.rds.outputs.mysql_rds_user_name
Password: data.terraform_remote_state.rds.outputs.mysql_rds_password

DMS Outputs

dms_security_group_id              # ID of DMS security group
dms_s3_target_bucket_name          # dms-target-hungryhub-{env}-s3
dms_s3_pipeline_replication_instance_arn  # Replication instance ARN
dms_mysql_to_s3_task_arn           # MySQL→S3 task ARN (dev only)

Relationship: ClickHouse ↔ DMS ↔ S3 ↔ RDS

                    ┌─────────────┐
                    │  ClickHouse │
                    │  (EKS)      │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │ MongoDB  │ │ HyperDX  │ │ Metrics  │
        │ Atlas    │ │  (UI)    │ │Prometheus│
        └──────────┘ └──────────┘ └──────────┘

                    ┌─────────────┐
                    │     DMS     │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  MySQL   │ │  Aurora  │ │  S3      │
        │  Source  │ │  Target  │ │ (Parquet)│
        │  (RDS)   │ │  (RDS)   │ │          │
        └──────────┘ └──────────┘ └──────────┘

Notes:

• ClickHouse and DMS are not directly connected. They are separate infrastructure.
• ClickHouse receives data from APM agents (OpenTelemetry) and queries from HyperDX/BI tools.
• DMS moves data MySQL → Aurora (CDC) and MySQL → S3 (analytics pipeline).
• DMS S3 bucket can be used as a data source for ClickHouse (via S3 table function or materialized view), but no direct connection is defined in Terraform currently.