HH_LOGGER → OpenSearch: Safe Access Runbook

Verified against Aiven-hosted OpenSearch 2.17.1 / Lucene 9.11.1 behind logs.hungryhub.com (OpenSearch Dashboards), 2026-08-06. Authentication findings reflect live-endpoint tests; see §2 for the two-layer flow and what is actually required.

This runbook explains the HH_LOGGER shipping pipeline, how to authenticate to the OpenSearch Dashboards API at logs.hungryhub.com, the active index patterns for every cluster family, safe field use, and count-only query templates. It deliberately avoids raw log retrieval, document _source reads, and exposure of any free-text or contextual fields.

Companion docs:

• Cluster & credential context: Credential Access Monitoring, Logs & APM
• APM error retrieval (different system): Kibana APM Error Reports — API Retrieval & Interpretation Runbook
• How to reach Kibana staging logs: Access Staging Logs on Kibana

Table of contents

1. Shipping architecture: HH_LOGGER → OpenSearch
2. Authentication — two-layer flow
   • §2.1 Cloudflare Access layer
   • §2.2 OpenSearch Dashboards native auth
   • §2.3 Python credential-loading example
3. Index patterns and rotation
4. Field mapping and safe field use
5. Count-only query templates
   • §5.1 Severity totals (last 24 h)
   • §5.2 PR #8586 — redis_cache_store_invalid_ttl_write_rejected
   • §5.3 PR #8584 — sidekiq_unique_job.timeout
   • §5.4 Cross-environment index breakdown
6. Field-capability checks
7. Index rotation and interpreting zero hits
8. Cosmos clusters — no OpenSearch shipping
9. Troubleshooting authentication errors
10. Evidence limitations

1. Shipping architecture: HH_LOGGER → OpenSearch

hh-server application code
  └── HH_LOGGER  (config/initializers/2_logging.rb)
        └── CustomLogger.new('log/hh-server.log')
              └── LogStashLogger → structured JSON → log/hh-server.log

Fluent Bit sidecar (every hh-server pod)
  └── tails log/hh-server.log under tag: custom-log-technical
        └── opensearch OUTPUT plugin
              Logstash_Format On
              → renames JSON field  timestamp  →  @timestamp  in the indexed document
              → writes to index prefix  <cluster-prefix>-custom-technical-index-<date>

Source references for the above:

• manifest/base/prod/public/configmaps/fluent-bit.conf.tmpl — prod rotation config
• manifest/base/prod-support/public/configmaps/fluent-bit.conf.tmpl — prod-support / staging rotation
• manifest/overlays/prod/hh-*/set_opensearch_env.sh — per-cluster prefix values

The Logstash_Format On directive is the reason documents carry @timestamp (not timestamp). Always filter on @timestamp in all range queries.

2. Authentication — two-layer flow

logs.hungryhub.com uses two independent auth layers that must both be satisfied:

Client
  │
  ▼
Cloudflare Access (layer 1)
  │  blocks anonymous requests
  │  accepts: browser-based OAuth / SAML session (→ CF_Authorization cookie)
  │  does NOT accept: CF service-token headers for this application (see §2.1)
  │
  ▼ (with valid CF_Authorization cookie in request)
OpenSearch Dashboards native auth (layer 2)
  │  POST /auth/login  {username, password}  → session cookie (1 h TTL)
  │  accepts: HH_LOGS_OPENSEARCH_USERNAME / HH_LOGS_OPENSEARCH_PASSWORD from .env
  │
  ▼ (with OSD session cookie)
OpenSearch Dashboards API  (/_search, /_field_caps, /_cat/indices, …)

§2.1 Cloudflare Access layer

Observed behavior (live-endpoint verification, 2026-08-06):

Conclusion: The CF_Access_Client_Id and CF_Access_Client_Secret values in .env are not accepted by the CF Access policy protecting logs.hungryhub.com. Service-token programmatic access is not available for this application.

Obtaining a CF_Authorization cookie requires completing a browser-based Cloudflare Access authentication flow (Google SSO or equivalent) at logs.hungryhub.com. Automation that wants to call the OSD API must present that browser-obtained cookie on every request.

Troubleshooting by status code: see §9.

§2.2 OpenSearch Dashboards native auth

Once a valid CF_Authorization cookie is present, the OSD session can be obtained:

POST /auth/login
Content-Type: application/json
osd-xsrf: true
Cookie: CF_Authorization=<browser-obtained token>

{"username": "<HH_LOGS_OPENSEARCH_USERNAME>", "password": "<HH_LOGS_OPENSEARCH_PASSWORD>"}

A successful 200 response sets an OSD session cookie (TTL ≈ 1 hour). All subsequent API calls must carry both the CF_Authorization cookie and the OSD session cookie.

§2.3 Python credential-loading example

The following script demonstrates the secure pattern for loading credentials and making a single probe request. It prints only sanitized HTTP status/category and emits no credential values or raw response content.

#!/usr/bin/env python3
"""
Probe logs.hungryhub.com with credentials from a local .env.
Prints only HTTP status category (2xx / 4xx / 5xx).
Never prints credential values, cookies, response bodies, or log data.

Usage:
  CF_COOKIE=<browser-obtained CF_Authorization value> \
  ENV_FILE=/path/to/.env \
  python3 probe_hh_logs.py
"""
import http.cookiejar
import json
import os
import sys
import urllib.error
import urllib.request

ENV_FILE = os.environ.get("ENV_FILE", "./.env")
# CF_COOKIE must come from a browser-completed CF Access flow.
# Never embed or log this value.
CF_COOKIE_VALUE = os.environ.get("CF_COOKIE", "")


def load_env(path):
    values = {}
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith("#") or "=" not in line:
                continue
            k, v = line.split("=", 1)
            values[k.strip()] = v.strip().strip('"').strip("'")
    return values


def status_category(code):
    if code < 300:
        return "2xx (success)"
    if code < 400:
        return "3xx (redirect)"
    if code < 500:
        return "4xx (client/auth error)"
    return "5xx (server error)"


def main():
    if not CF_COOKIE_VALUE:
        print(
            "CF_COOKIE env var is empty. "
            "Obtain a CF_Authorization cookie via browser login first.",
            file=sys.stderr,
        )
        sys.exit(1)

    env = load_env(ENV_FILE)
    base = env.get("HH_LOGS_OPENSEARCH_BASE_URL", "").rstrip("/")
    username = env.get("HH_LOGS_OPENSEARCH_USERNAME", "")
    password = env.get("HH_LOGS_OPENSEARCH_PASSWORD", "")

    if not all([base, username, password]):
        print("Missing HH_LOGS_OPENSEARCH_* vars in .env", file=sys.stderr)
        sys.exit(1)

    # ── Step 1: OSD native login (credentials constructed in memory) ──────────
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))

    login_payload = json.dumps({"username": username, "password": password}).encode()
    login_req = urllib.request.Request(base + "/auth/login", data=login_payload, method="POST")
    login_req.add_header("Content-Type", "application/json")
    login_req.add_header("osd-xsrf", "true")
    # CF_Authorization cookie provides layer-1 access; value stays in memory.
    login_req.add_header("Cookie", f"CF_Authorization={CF_COOKIE_VALUE}")

    try:
        with opener.open(login_req, timeout=20) as resp:
            login_status = resp.status
    except urllib.error.HTTPError as e:
        print(f"OSD /auth/login: {status_category(e.code)}", file=sys.stderr)
        sys.exit(2)

    print(f"OSD /auth/login: {status_category(login_status)}")
    if login_status >= 300:
        sys.exit(2)

    # ── Step 2: Probe with a size:0 count query ───────────────────────────────
    count_payload = json.dumps({
        "size": 0,
        "query": {"range": {"@timestamp": {"gte": "now-1h", "lte": "now"}}},
    }).encode()
    count_req = urllib.request.Request(
        base + "/*custom-technical-index-*/_search",
        data=count_payload,
        method="POST",
    )
    count_req.add_header("Content-Type", "application/json")
    count_req.add_header("osd-xsrf", "true")
    session_cookies = "; ".join(f"{c.name}={c.value}" for c in cj)
    combined_cookie = f"CF_Authorization={CF_COOKIE_VALUE}"
    if session_cookies:
        combined_cookie += f"; {session_cookies}"
    count_req.add_header("Cookie", combined_cookie)

    try:
        with opener.open(count_req, timeout=20) as resp:
            query_status = resp.status
            body = json.loads(resp.read().decode("utf-8"))
            total = body.get("hits", {}).get("total", {})
            count = total.get("value") if isinstance(total, dict) else total
    except urllib.error.HTTPError as e:
        print(f"count query: {status_category(e.code)}", file=sys.stderr)
        sys.exit(2)

    print(f"count query: {status_category(query_status)}")
    # Emit only the numeric count — no log content, no field values.
    print(f"1h hit count: {count}")


if __name__ == "__main__":
    main()

Smoke-test (fabricated response, no live endpoint or secrets required):

#!/usr/bin/env python3
"""
Smoke-test: feeds a fabricated count response through the output path
to prove no secrets or raw log documents are emitted.
"""
import json

def status_category(code):
    return "2xx (success)" if code < 300 else "4xx" if code < 500 else "5xx"

# Fabricated response — mimics a real OpenSearch _search reply.
fake_body = json.dumps({
    "hits": {"total": {"value": 42, "relation": "eq"}, "hits": []},
    "aggregations": {},
})

body = json.loads(fake_body)
count = body.get("hits", {}).get("total", {}).get("value")

# Only these lines should appear in real usage — no credential echoes.
print(f"OSD /auth/login: {status_category(200)}")
print(f"count query: {status_category(200)}")
print(f"1h hit count: {count}")
# Verify no raw log document is in the output (hits list is empty in fabricated resp).
assert body["hits"]["hits"] == [], "Raw hit documents must not be emitted"
print("smoke-test passed: no secrets, no raw log documents emitted")

Running the smoke-test:

python3 smoke_test_hh_logs.py
# Expected output:
# OSD /auth/login: 2xx (success)
# count query: 2xx (success)
# 1h hit count: 42
# smoke-test passed: no secrets, no raw log documents emitted

3. Index patterns and rotation

All HH_LOGGER events use the suffix -custom-technical-index-<date>. The date format and rotation cadence vary by cluster family:

Active indices confirmed 2026-08-06 (approximate 24 h throughput):

Multi-index wildcard forms (use comma separation for explicit sets or the wildcard forms above for broad searches):

# All production HH_LOGGER (weekly/daily mixed):
prod-hh-*-custom-technical-index-*,.ds-prod-hh-server-custom-technical-index-*

# Cross-environment (all families):
prod-hh-*-custom-technical-index-*,.ds-prod-hh-server-custom-technical-index-*,staging-*-custom-technical-index-*,prod-support-*-custom-technical-index-*

# Broadest (matches all rotation periods, all families):
*custom-technical-index-*

OSD Discover pattern: Use *custom-technical-index-* as the index pattern in OpenSearch Dashboards Discover to span all clusters and rotation periods in one view.

4. Field mapping and safe field use

These fields are confirmed across prod-hh-end-user-public-custom-technical-index-2026.08.31 and .ds-prod-hh-server-custom-technical-index-2026.08.31-000001 via _field_caps and _mapping.

Do not use without first running a _field_caps check:

• payload.* sub-fields other than payload.event — field schema differs between clusters
• business_context.* and request_context.* — contain restaurant/user/request PII; these are off-limits for monitoring use
• host — pod hostname; do not log or include in reports

§4.1 payload field schema difference between cluster families

Before building a cross-cluster query on any payload.* sub-field, verify with _field_caps that the field exists in every target index.

5. Count-only query templates

All templates are read-only (size: 0), bounded by time range, and safe for repeated automated execution. No _source, no document retrieval, no free-text field values in results.

Replace <YYYY.MM.WW> with the current week suffix and <YYYY.MM> with the current month suffix when targeting a specific index.

§5.1 Severity totals (last 24 h)

POST /prod-hh-end-user-public-custom-technical-index-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-24h", "lte": "now" } }
  },
  "aggs": {
    "by_severity": {
      "terms": { "field": "severity.keyword", "size": 10 }
    }
  }
}

Observed distribution (prod-hh-end-user-public, 48 h sample, 2026-08-06): DEBUG 63.9 %, WARN 33.1 %, INFO 1.1 %, ERROR 0.1 %.

§5.2 PR #8586 — redis_cache_store_invalid_ttl_write_rejected

HH_LOGGER.warn('redis_cache_store_invalid_ttl_write_rejected', {...}) in config/initializers/0_4_redis_cache_store_patch.rb. The event identifier is the top-level message field; payload does not carry an event key for this call.

message is a text-only field (no .keyword); use match_phrase:

POST /*custom-technical-index-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-7d", "lte": "now" } } },
        { "match_phrase": { "message": "redis_cache_store_invalid_ttl_write_rejected" } }
      ]
    }
  }
}

7-day count (2026-07-30 → 2026-08-06, all *custom-technical-index-*): 0.

Interpretation: PR #8586 is merged to main (commit 5310dc1a9c) but not yet deployed, or its trigger condition (a write with zero/negative TTL reaching RedisCacheStoreExpiresFix#write_entry) has not occurred. A non-zero count after deployment signals a regression or edge case that the fix itself did not suppress.

§5.3 PR #8584 — sidekiq_unique_job.timeout

HH_LOGGER.info('sidekiq-unique-jobs Timeout', payload) in config/initializers/2_sidekiq.rb, where payload = { event: 'sidekiq_unique_job.timeout', ... }. Level is INFO (not WARN). The event identifier is payload.event:

POST /*custom-technical-index-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-7d", "lte": "now" } } },
        { "term": { "payload.event": "sidekiq_unique_job.timeout" } },
        { "term": { "severity.keyword": "INFO" } }
      ]
    }
  }
}

7-day count (2026-07-30 → 2026-08-06, all *custom-technical-index-*): 0.

Interpretation: PR #8584 is merged to main (commit fed6d54e31) but not yet deployed, or the sidekiq-unique-jobs on.timeout callback has not fired. A non-zero count after deployment confirms the reclassification from exception to structured INFO telemetry is working correctly.

payload.event is only available on Logstash-format clusters. The data-stream cluster (prod-hh-server) does not carry payload.event. Verify with _field_caps before narrowing to specific index patterns.

§5.4 Cross-environment index breakdown

POST /*custom-technical-index-*/_search
{
  "size": 0,
  "query": {
    "range": { "@timestamp": { "gte": "now-1h", "lte": "now" } }
  },
  "aggs": {
    "by_index": {
      "terms": { "field": "_index", "size": 20 }
    }
  }
}

This returns a count-by-index breakdown without retrieving any documents. Useful for confirming which clusters are actively shipping events.

6. Field-capability checks

Before building queries that assume a field exists across multiple clusters, verify with _field_caps:

GET /*custom-technical-index-*/_field_caps?fields=@timestamp,severity,message,payload.event,payload.latency,payload.queue

The response fields map shows the type for each field per index family. A field that appears in one family but not another will only be listed for the indices where it exists. Key things to check:

• message.keyword — confirmed absent; do not use with term queries.
• payload.event — confirmed present on Logstash-format clusters, absent on the prod-hh-server data-stream cluster.
• Any new payload.* sub-field introduced by a PR — verify presence before alerting on it.

7. Index rotation and interpreting zero hits

Rotation periods:

• Weekly indices roll over at the start of each ISO week. Queries spanning a week boundary must use prod-hh-*-custom-technical-index-* to pick up both sides. The date suffix format YYYY.MM.WW uses zero-padded ISO week number.
• Monthly indices roll on the first of each month. Use staging-*-custom-technical-index-* or prod-support-*-custom-technical-index-* to span months automatically.
• Data-stream indices roll daily. OSD manages the backing index automatically; the .ds- prefix and the backing-index suffix are internal and should not be targeted directly — use the wildcard form.

Interpreting zero hits:

A count of zero for a specific event does not mean the event cannot occur. Consider:

1. Code merged but not deployed. A feature lands in main but the pod rollout has not propagated to all clusters. The event cannot fire until the new image is running.
2. Trigger condition not met. The event may only fire under specific runtime conditions (e.g., redis_cache_store_invalid_ttl_write_rejected fires only when a write with an invalid TTL reaches the patched code path; under normal operation the fix suppresses it silently).
3. Wrong index family. payload.event is absent from the prod-hh-server data-stream; a query targeting that cluster for a payload.event term will always return zero.
4. Time window too narrow. Infrequent events may not appear in 24 h but appear in a 7-day or 30-day window.
5. Pending rotation. If the query targets now-7d and the current weekly index did not exist seven days ago, older events live in the previous week’s index. The wildcard pattern handles this automatically.

8. Cosmos clusters — no OpenSearch shipping

The following clusters have Fluent Bit configured to read hh-server.log but all OpenSearch OUTPUT blocks are commented out in their manifest patches:

• prod-hh-cosmos-public
• prod-hh-cosmos-private

Source: manifest/overlays/prod/hh-cosmos-public/patches/configmaps/fluent-bit-config.yaml and the equivalent -private patch — every [OUTPUT] stanza is a comment block.

Consequence: HH_LOGGER events from Cosmos pods are not shipped to OpenSearch. No prod-hh-cosmos-*-custom-technical-index-* indices exist. Absence of such indices is not evidence that Cosmos pods produce no events; it is evidence that the shipping path is disabled.

To inspect Cosmos HH_LOGGER events, use kubectl logs on the application container of a Cosmos pod in the appropriate namespace. This requires separate cluster authorization (kubeconfig access to the production hungryhub namespace) and is outside the scope of this runbook.

If the shipping path should be enabled, the commented-out OUTPUT blocks in the above manifests must be re-enabled and the relevant OPENSEARCH_CUSTOM_LOGSTASH_PREFIX environment variables populated for the Cosmos overlays.

9. Troubleshooting authentication errors

10. Evidence limitations

1. Index age not inspected. Only the current rotation period was enumerated. Historical indices (prior weeks / months) follow the same patterns but were not validated individually.

2. Cosmos clusters unresolved. Whether the commented-out Fluent Bit outputs are intentional (logs discarded) or an oversight (should be re-enabled) is not known from the code alone.

3. prod-hh-server data-stream origin. No set_opensearch_env.sh in the Kustomize overlay tree was found that defines the prod-hh-server-* prefix. This cluster likely has its OpenSearch env vars injected via a CodePipeline variable or non-Kustomize config path.

4. payload schema differs between clusters. Logstash-format clusters carry 37 payload.* sub-fields; the data-stream cluster carries only payload.latency and payload.queue. Always run _field_caps before cross-cluster payload queries.

5. Zero-hit window. The 7-day count window for PR #8586 and PR #8584 events reflects 2026-07-30 to 2026-08-06. Both PRs are merged but neither event had fired in that window. This does not mean the events are unreachable.

6. Direct Aiven endpoint. Direct access to the Aiven service endpoint (bypassing Cloudflare) using Basic Auth would also work, but requires network-level access (VPC peering or private endpoint) not available from developer workstations.

7. No prod-hh-end-user-private-*, prod-hh-syn-private-*, or prod-hh-vendor-private-* indices. These private-cluster patterns produced no results. The private clusters may share pods with their public counterparts or have been decommissioned.