APM Setup Guide — K8s Deployment, ConfigMap & Verification

Complete guide for setting up Elastic APM in any HungryHub Rails app, covering the full stack from gem installation to K8s ConfigMap deployment and verification.

For Ruby instrumentation patterns (spans, error handling, concerns), see src/PricingEngine/010-APM-MONITORING.md.

Table of Contents

1. APM Server URLs
2. Per-Namespace Configuration
3. K8s ConfigMap Setup
4. Build Pipeline Integration
5. Production vs Staging Tuning
6. Verification Steps
7. Troubleshooting

1. APM Server URLs

Two APM servers exist — one for staging (DigitalOcean) and one for production (EKS internal).

APM Credentials (from AWS Secrets Manager):

2. Per-Namespace Configuration

Each K8s namespace has its own ConfigMap with service_name and environment. The server_url is the same within each cluster.

Staging (eks-dev-262)

Dev-preview namespaces (Pattern C — elastic_apm.yml format):

Note: dev-preview-* namespaces use service_name: hh-server (not per-service names) and individual environment values per namespace. These use Pattern C (elastic_apm.yml format).

Production (eks-prod-21)

Note: hungryhub namespace uses service_name: hungryhubcom and environment: hungryhubcom (not production) — this is intentional.

3. K8s ConfigMap Setup

Every app deployment needs a ConfigMap with Elastic APM config. The ConfigMap is mounted as a volume and read by the elastic-apm gem.

3.1 ConfigMap Template

# base/configmaps/elastic-apm-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic-apm-config
data:
  ELASTIC_APM_SERVICE_NAME: "<SERVICE_NAME>"
  ELASTIC_APM_SERVER_URL: "<SERVER_URL>"
  ELASTIC_APM_ENVIRONMENT: "<ENVIRONMENT>"
  ELASTIC_APM_ENABLED: "true"
  ELASTIC_APM_LOG_LEVEL: "info"
  ELASTIC_APM_POOL_SIZE: "10"
  ELASTIC_APM_TRANSACTION_SAMPLE_RATE: "1"
  ELASTIC_APM_API_BUFFER_SIZE: "256"
  ELASTIC_APM_API_REQUEST_SIZE: "500kb"
  ELASTIC_APM_API_REQUEST_TIME: "5s"
  ELASTIC_APM_CAPTURE_HEADERS: "true"
  ELASTIC_APM_CAPTURE_ENV: "true"
  ELASTIC_APM_SPAN_FRAMES_MIN_DURATION: "5ms"
  ELASTIC_APM_STACK_TRACE_LIMIT: "50"
  ELASTIC_APM_TRANSACTION_MAX_SPANS: "300"

3.2 Three Templating Patterns

Pattern A — ERB (hh-server): Uses <%= ENV['...'] %> placeholders, substituted by envsubst at deploy time.

# overlays/staging/hh-engineering-public/patches/elastic-apm-config.yaml
data:
  ELASTIC_APM_SERVER_URL: "<%= ENV['APM_SERVER_URL'] %>"
  ELASTIC_APM_ENVIRONMENT: "engineering"

Pattern B — Placeholder (pricing-engine): Uses <PLACEHOLDER> strings, replaced by kustomize or shell scripts.

# overlays/do/prod/patches/elastic-apm-config.yaml
data:
  ELASTIC_APM_SERVER_URL: "<APM_SERVER_URL>"
  ELASTIC_APM_ENVIRONMENT: "<ENVIRONMENT>"

Pattern C — elastic_apm.yml (dev-preview): Uses a single YAML file in the ConfigMap data key, with ERB templating for server_url. Used in dev-preview-* namespaces.

# dev-preview-engineering-public elastic-apm-config
data:
  elastic_apm.yml: |
    service_name: "hh-server"
    server_url: <%= ENV['APM_SERVER_URL'] %>
    environment: "engineering"
    log_level: info
    log_path: 'log/elastic_apm.log'
    api_request_size: "500kb"
    api_request_time: "10s"
    capture_headers: false
    capture_env: false
    pool_size: 5
    transaction_sample_rate: 1
    api_buffer_size: 512

3.3 Deployment Structure (hh-server)

hh-server/
├── manifest/
│   ├── base/
│   │   ├── staging/public/configmaps/elastic-apm-config.yaml   # Base staging defaults
│   │   └── prod/public/configmaps/elastic-apm-config.yaml      # Base production defaults
│   └── overlays/
│       ├── staging/
│       │   ├── hh-engineering-public/patches/elastic-apm-config.yaml  # ERB override
│       │   └── hh-ballbot-public/patches/elastic-apm-config.yaml     # Placeholder override
│       └── prod/
│           └── ...

3.4 Deployment Structure (pricing-engine)

pricing-engine/
├── manifest/
│   ├── base/
│   │   └── configmaps/elastic-apm-config.yaml              # Base with placeholders
│   └── overlays/
│       ├── do/prod/patches/elastic-apm-config.yaml         # DigitalOcean prod
│       ├── eks/dev262/patches/elastic-apm-config.yaml      # EKS dev/staging
│       └── eks/prod/patches/elastic-apm-config.yaml        # EKS production

4. Build Pipeline Integration

4.1 AWS Secrets Manager

APM_SERVER_URL is stored in AWS Secrets Manager and injected at deploy time via BuildSpec.

Path convention: /hungryhub/<Environment>:APM_SERVER_URL

# buildspec-hh-engineering.yaml
env:
  variables:
    ECR_LOGIN_ACCOUNT_ID: "627608802312"
    APP_NAME: "hh-server"
  parameter-store:
    APM_SERVER_URL: "/hungryhub/$Environment:APM_SERVER_URL"
    ELASTICSEARCH_SERVER_URL: "/hungryhub/$Environment:ELASTICSEARCH_SERVER_URL"
    # ... other secrets

4.2 envsubst for Config Substitution

Some overlays use set_env.sh + envsubst to substitute environment variables into ConfigMap files at deploy time.

# set_env.sh (hh-engineering-public)
export ELASTICSEARCH_SERVER_URL="http://opendistro-es-node1:9200"
export ELASTICSEARCH_INDEX_SERVER="app-log-staging-server"
# ... more OpenSearch indices

# Substitute into fluent-bit.conf template
envsubst < fluent-bit.conf.template > fluent-bit.conf

5. Production vs Staging Tuning

Key differences:

• Staging: Full sampling (1.0), smaller pool (5), verbose logging (info)
• Production: Reduced sampling (0.7), larger pool (15), minimal logging (warn)
• Some namespaces have custom overrides (e.g., hh-ballbot uses api_request_size: 1500kb, api_request_time: 29s for longer requests)

6. Verification Steps

6.1 Check ConfigMap Values

# Staging (eks-dev-262)
kubectl config use-context eks-dev-262
kubectl get configmap elastic-apm-config -n <namespace> -o yaml

# Production (eks-prod-21)
kubectl config use-context eks-prod-21
kubectl get configmap elastic-apm-config -n <namespace> -o yaml

6.2 Verify APM Server Connectivity

# From within the cluster
kubectl run apm-test --rm -it --image=curlimages/curl -- \
  curl -v http://20.0.30.229:8300

# Expected: HTTP 200 with APM server info JSON

6.3 Check APM Agent Status in Logs

# Check app logs for APM startup
kubectl logs -n <namespace> deployment/<app-name> | grep -i "elastic"

# Look for:
# "ElasticAPM started" or "Connecting to APM Server"

6.4 Verify Traces in Kibana

1. Open Kibana APM UI:

   • Staging: https://apm.hh-engineering.my.id
   • Production: https://apm.hungryhub.com

2. Navigate to APM > Services

3. Find your service_name in the list

4. Generate some traffic to the app

5. Check for:

   • Transactions (requests)
   • Spans (database queries, external calls)
   • Errors (if any)

6.5 Verify via Elasticsearch API

# Check APM indices exist
curl -u elastic:sGQJBy1Gbg7qfYbbFlWd \
  "https://apm.hungryhub.com/_cat/indices/apm*?v"

# Search for recent transactions from your service
curl -u elastic:sGQJBy1Gbg7qfYbbFlWd \
  "https://apm.hungryhub.com/apm-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "service.name": "<SERVICE_NAME>" } },
          { "range": { "@timestamp": { "gte": "now-1h" } } }
        ]
      }
    },
    "size": 5
  }'

7. Troubleshooting

APM agent not starting

• Check ELASTIC_APM_AGENT=1 is set in Dockerfile or deployment env
• Verify ENABLE_ELASTIC_APM=true in ConfigMap
• Check initializer isn’t overriding to enabled: false

No data in Kibana

1. Verify server_url matches the correct APM server for your cluster
2. Check service_name is unique and matches the Kibana service list
3. Ensure the namespace ConfigMap is actually mounted (check kubectl describe pod)
4. Check APM server logs: kubectl logs -n monitoring deployment/apm-server

Raw placeholders in ConfigMap

If <APM_SERVER_URL> or <%= ENV['APM_SERVER_URL'] %> appears literally:

• The set_env.sh / envsubst step didn’t run
• The kustomize overlay patch didn’t apply
• Re-run the build pipeline or manually apply the correct values

High memory usage

• Reduce pool_size (default 10, prod 15)
• Reduce transaction_max_spans (prod: 700, staging: 300)
• Set transaction_sample_rate lower (e.g., 0.3)

Quick Reference