Debezium BigQuery Consumers

This project adds BigQuery sink consumers to Debezium Server. These consumers replicate change data capture (CDC) events from databases to Google BigQuery in real time.

Debezium BigQuery Consumers:
- bigquerybatch Consumer - Uses standard BigQuery Load Jobs (no streaming ingestion fees).
- bigquerystream Consumer - Uses high-throughput BigQuery Storage Write API with real-time streaming and optional CDC Upsert support.

Key Features

Batch & Streaming Modes: Support for both standard BigQuery Load Jobs and the high-throughput BigQuery Storage Write API.
CDC Upsert & Deletion Handling: Real-time deduplication and UPSERT mode using BigQuery CDC.
Embedded Storage Extensions: Built-in BigQuery implementations for Debezium Offset Storage (BigqueryOffsetBackingStore) and Schema History (BigquerySchemaHistory).
Dynamic Batch Optimization: Configurable batch size wait strategies (MaxBatchSizeWait, DynamicBatchSizeWait) to optimize file sizes and upload intervals.
Nested JSON Serialization: Configurable handling of nested record structures as JSON strings (debezium.sink.batch.nested-as-json).

Build and Install from Source

Prerequisites

JDK 21 or later
Apache Maven 3.6.3 or later

Installation Steps

Clone the repository:

git clone https://github.com/memiiso/debezium-server-bigquery.git
cd debezium-server-bigquery

Build and package:

mvn clean package -Passembly -DskipTests

Unzip the distribution package:

unzip debezium-server-bigquery-dist/target/debezium-server-bigquery-dist*.zip -d appdist
cd appdist

Configure the application: Edit conf/application.properties (refer to application.properties.example for baseline settings).
Run the server:
```
bash run.sh
```

Running via Docker

Build and run using the provided multi-stage Dockerfile:

# Build the container image
docker build -t debezium-server-bigquery .

# Run with custom configuration and data volumes
docker run -d --name debezium-bigquery \
  -v $(pwd)/conf:/app/conf \
  -v $(pwd)/data:/app/data \
  debezium-server-bigquery

Contributing

We welcome contributions of any kind! Feel free to report issues, suggest improvements, or submit pull requests.