Airbyte Integration Guidelines

Context & Objective

Following the recent high-load incident on RDS caused by Full Refresh operations, this document defines the standard operating procedures for data synchronization.

How to set the Airbyte Source

Access the Airbyte Web Portal Open https://airbyte.hungryhub.com/ and log in using your Airbyte account.
Create a New Source The first step in creating a connection is defining the source (where your data comes from).
- Click + New Source.
- Select the appropriate source connector (e.g., MySQL, Google Analytics). Note: A setup guide specific to your chosen connector will appear on the right side of the page. You can follow that guide for additional details.
Input Connection Details Enter the Host, Port, User, Password, and Database name.
Select an Update Method Airbyte offers two methods for replicating data. Choose the one that fits your infrastructure:
- Option A: Scan Changes with User-Defined Cursor
  - How it works: Uses a custom cursor column in your source table (e.g., updated_at).
  - When to use: Use this if your server does not support binary logging (binlog), if your dataset is small, or if you only need a snapshot of your table.
- Option B: Read Changes using Change Data Capture (CDC)
  - How it works: Reads the binary log to capture changes in real-time.
  - Requirements: You must enable binary logging on your MySQL server. (Cloud providers like AWS/GCP usually have a one-click option for this; self-managed servers require configuration file changes).
  - Settings: If you choose CDC, set the “Invalid CDC Position Behavior” to either Fail sync or Re-sync data.
Finalize Setup Click the Set up source button. It will take approximately 5 minutes for Airbyte to test and connect to the source.

How to set the Airbyte Destination

Access the Airbyte Web Portal Open https://airbyte.hungryhub.com/ , log in using your Airbyte account and click on Destinations in the menu.
Create a New Destination
- Click + New destination.
- Pick the destination connector. Note: HungryHub uses ClickHouse as the Airbyte destination.
Input Connection Details Enter the Host, Port, Protocol, Database, Username, and Password. Note: Similar to the source setup, a guide will appear on the right side of the page to assist you.
Finalize Setup Click the Set up destination button. It will take approximately 5 minutes for Airbyte to test and connect to the destination.

How to set the Airbyte Connection

A Connection links your Source to your Destination and defines how data should move between them.

Access the Airbyte Web Portal Open https://airbyte.hungryhub.com/ and log in with your Airbyte account.
Initiate the Connection

Click the + New connection button.
Select the Source you created earlier.
Select the Destination where the data will be sent.

Configure Streams and Sync Modes

Replicate Source: Choose the recommended “Replicate Source” option to see available data.
Select Streams: Toggle the switch for the streams (database tables) you wish to sync.
Select Sync Mode: Choose between Incremental or Full Refresh. Note on Cursors:
```
  *   **CDC Sources:** If you enabled CDC on the source, the cursor is handled automatically.
```
- User-Defined Sources: If not using CDC, you must select a “Cursor Field” for tracking updates. It is best to use a column that updates automatically, such as updated_at. Note on Sync Modes: For a more detailed explanation of the different Sync Modes, please scroll down to the relevant section of this document.

Connection Settings Input the following configuration details:

```
  *   **Connection Name:** A unique name for this pipeline.
```
- Schedule Type: How often the sync should run (e.g., scheduled, Manual, Cron).
- Replication Frequency: The specific time interval (e.g., Every hour).
- Destination Namespace: Select Destination-defined.

Configure Schema Change Behavior Find the field labeled “When the source schema changes, I want to…”. This setting controls what Airbyte does when your source database structure changes (e.g., a developer adds a new column). Choose the option that best fits your needs:

Propagate field changes only
- Behavior: Automatically updates the destination table if columns are added or removed in existing tables. It ignores completely new tables.
- Best for: Stable products where you want column updates to flow automatically but don’t want unexpected new tables appearing.
Propagate all field and stream changes
- Behavior: Total automation. If you add a new column OR a new table in the source, Airbyte automatically syncs it to the destination.
- Best for: Development environments or early-stage startups where speed is prioritized over strict governance.
Approve all changes myself
- Behavior: Airbyte detects the change but takes no action. The sync continues using the old schema. You must manually click “Approve” in the UI to apply the change.
- Best for: Critical production pipelines where you cannot risk a schema change breaking downstream dashboards.
Stop future syncs
- Behavior: The “Emergency Brake.” If any change is detected, the connection is immediately disabled/paused.
- Best for: Highly sensitive environments where data consistency is paramount and unexpected changes are treated as incidents.

Finalize Click Finish & sync to save the connection and trigger the first sync immediately.

Sync Modes: When to Use What

Sync Mechanics:

Incremental: Reads only data that changed since the last run.
Full Refresh: Reads the entire dataset every run.
Append: Adds new records to the end of the table (keeps history/duplicates).
Deduped: Merges updates to keep only the single latest version of a record.
Overwrite: Deletes the data and replaces it with a fresh copy.

A. Incremental Sync (Append + Deduped)

Status: Recommended for Production

How it works: Airbyte reads a Cursor Field (e.g., updated_at) to identify new or modified rows. It uses the Primary Key to deduplicate rows in the destination, ensuring only the latest version exists.
When to use:
- Tables with frequent updates (Users, Orders, Transactions).
- Large datasets (>100k rows).
Requirements: A reliable Cursor Field (timestamp) and a Primary Key.

B. Full Refresh (Overwrite)

Status: Restricted

How it works: Deletes existing data in the destination and re-syncs the entire source table from scratch.
When to use:
- Small, static lookup tables (e.g., country_codes).
- Tables with no reliable cursor/timestamp.
- Dev/Staging environments for initial testing.

Airbyte Sync Modes Comparison

Sync Mode	How Source is Read	How Destination is Written	Handling of Duplicates	Best Used When…
Incremental (Append + Deduped)	Reads only new or modified records (using a Cursor field like `updated_at`).	Appends data to a history table, then updates the final table to show only the latest version.	Removes duplicates. Ensures the final table has only one row per Primary Key.	You need a production-ready mirror of your database (e.g., `users` table) and want to minimize load.
**Full Refresh	Overwrite**	Reads the entire source table every time.	Deletes the existing destination table and replaces it with the new data.	No duplicates (from previous runs), but relies on the source having unique data.
**Incremental	Append**	Reads only new or modified records.	Appends new rows to the end of the destination table. Never deletes old rows.	Keeps duplicates. If a row is updated in the source, you get both the old and new versions in the destination.
**Full Refresh	Append**	Reads the entire source table every time.	Appends the full dataset to the end of the destination table.	Creates massive duplicates. Every sync adds a full copy of the table.
**Full Refresh	Overwrite + Deduped**	Reads the entire source table every time.	Overwrites the “raw” data, then rebuilds the final table ensuring uniqueness.	Removes duplicates within the source snapshot. Ensures the final table is unique by Primary Key.

Keyboard shortcuts

HungryHub Knowledge Base