faucetstream
Get started

Connectors / Warehouses

Google BigQuery

sink

Google BigQuery streaming-insert sink

Install

cargo install faucet-cli --features sink-bigquery
faucet init --sink bigquery
# Cargo.toml — use it as a library
cargo add faucet-sink-bigquery

Starter config

sink:
  type: bigquery
  config:
    project_id: ""    # REQUIRED — GCP project ID.
    dataset_id: ""    # REQUIRED — BigQuery dataset ID.
    table_id: ""    # REQUIRED — BigQuery table ID.
    auth:
      type: service_account_key_path    # one of: service_account_key_path, service_account_key, application_default
      config:
        path: ""    # REQUIRED — Filesystem path to the service-account JSON key.
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_key
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
    # batch_size: 1000    # Maximum rows per `tabledata.insertAll` request.
    # insert_id_field: ""    # Optional record field whose value is sent as the BigQuery streaming `insertId` for each row.
    # job_timeout: 3600    # Longest the sink waits, in seconds, for one BigQuery job it started (a MERGE, the overwrite swap, a cleanup DELETE, DDL, a load job) before requesting its cancellation and failing the write.
    # write_mode: null    # Write semantics for a sink.
    # key: []    # Key columns.
    # delete_marker: null    # Optional.
    # rollback: null    # Runtime rollback settings (#706): the run id to journal under, the run-id column, and whether to journal before-images / keep a replaced table.
    # scope: null    # Scoped/windowed overwrite (#518): with `write_mode: overwrite`, replace only the rows matching this scope (e.g.
    # create_table: true    # Create the target table (and its dataset) if it does not exist, inferring the schema from the first written page.
    # schema: null    # **Experimental** (PRINCIPLES.md §3): this field may change shape in a minor release; changes are called out in the changelog.
    # location: ""    # Location (region or multi-region, e.g.
    # media_load: true    # Bucket-free bulk load for the **append** and **overwrite** write paths (default **`true`**, #612).
    # bulk_load: null    # Arrow columnar **load-job** mode (#380): buffer Arrow `RecordBatch`es to Parquet, stage them on a GCS bucket, then run a BigQuery `PARQUET` load job (`jobs.insert`) instead of the per-row `insertAll` path.

Capabilities

Delivery Exactly-once (atomic watermark)
Write modes append, upsert, delete, overwrite
Exactly-once writes Yes
Schema evolution Yes
Scoped cleanup Yes
Staged bulk load Yes
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
project_id
required		 string — GCP project ID.
dataset_id
required		 string — BigQuery dataset ID.
table_id
required		 string — BigQuery table ID.
auth
required		 object — Authentication credentials. YAML/JSON key is auth for consistency with every other connector's auth block.
batch_size integer 1000 Maximum rows per tabledata.insertAll request. Defaults to DEFAULT_BATCH_SIZE.
insert_id_field string — Optional record field whose value is sent as the BigQuery streaming insertId for each row. BigQuery uses insertId for best-effort de-duplication over a short window, so a stable per-row key here makes streaming inserts resilient to transport retries (which are otherwise at-least-once and can produce duplicate rows) (#78/#31). When None (the default) no insertId is sent. A row missing the field is inserted without an insertId (no dedup for that row).
job_timeout integer 3600 Longest the sink waits, in seconds, for one BigQuery job it started (a MERGE, the overwrite swap, a cleanup DELETE, DDL, a load job) before requesting its cancellation and failing the write. Default 3600; 0 waits for as long as the job runs. After the cancel the sink waits up to 30 s for the job to stop; a job that finished successfully before the cancel took effect is treated as completed, and the error says when a job could not be confirmed stopped (it may still commit).
write_mode object append Append (default), upsert, or delete.
key array [] Key columns. Required and non-empty for upsert/delete; ignored for append.
delete_marker object — Optional. Upsert only: rows whose field matches one of values are deletes; all others are upserts. The marker field is stripped from upsert rows before writing.
rollback object — Runtime rollback settings (#706): the run id to journal under, the run-id column, and whether to journal before-images / keep a replaced table. Injected per invocation by the CLI when a config has a top-level rollback: block — not meant to be written by hand.
scope object — Scoped/windowed overwrite (#518): with write_mode: overwrite, replace only the rows matching this scope (e.g. a partition/date window) instead of the whole table — the out-of-scope rows are preserved.
create_table boolean true Create the target table (and its dataset) if it does not exist, inferring the schema from the first written page. Enabled by default: a first-ever sync cannot assume the destination table already exists. Set to false to require the table to pre-exist and fail fast when it is missing (e.g. the schema is managed externally and a missing table signals a typo).
schema any — **Experimental** (PRINCIPLES.md §3): this field may change shape in a minor release; changes are called out in the changelog.
location string — Location (region or multi-region, e.g. US, EU, us-central1) used only when create_table has to create the dataset. None uses the BigQuery job's default location. Ignored once the dataset exists.
media_load boolean true Bucket-free bulk load for the **append** and **overwrite** write paths (default **true**, #612).
bulk_load object — Arrow columnar **load-job** mode (#380): buffer Arrow RecordBatches to Parquet, stage them on a GCS bucket, then run a BigQuery PARQUET load job (jobs.insert) instead of the per-row insertAll path. Only present in arrow builds; drives the columnar fast path the pipeline negotiates when the source and sink are both columnar. Load jobs are append/truncate only, so the sink advertises columnar support **only** when the write mode is append — upsert/delete stay on the MERGE path.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub