faucetstream
Get started

Connectors / Warehouses

Google BigQuery

source

Google BigQuery query source

Install

cargo install faucet-cli --features source-bigquery
faucet init --source bigquery
# Cargo.toml — use it as a library
cargo add faucet-source-bigquery

Starter config

source:
  type: bigquery
  config:
    project_id: ""    # REQUIRED — GCP project ID against which the query is billed and run.
    auth:
      type: service_account_key_path    # one of: service_account_key_path, service_account_key, application_default
      config:
        path: ""    # REQUIRED — Filesystem path to the service-account JSON key.
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_key
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
    query: ""    # REQUIRED — SQL statement to execute.
    # use_legacy_sql: false    # Whether to use BigQuery's legacy SQL dialect.
    # location: ""    # Optional location override for non-`US` jobs (`"EU"`, `"asia-east1"`).
    # max_results_per_page: 1000    # Maximum rows per page when calling `jobs.getQueryResults`.
    # params: []    # Positional bind parameters for the query, sent as `POSITIONAL` query parameters in declaration order before any context-derived values.
    # statement_timeout: 60    # Per-statement server-side timeout.
    # poll_timeout: 300    # Maximum wall-clock time the source will spend polling `jobs.getQueryResults` for a job that keeps reporting `jobComplete=false`, before giving up with `FaucetError::Source`.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # read_api: false    # Arrow columnar **Storage Read API** mode (#380).
    # read_table: ""    # Table to read in `read_api` mode: `dataset.table` (billed to `project_id`) or a fully-qualified `project.dataset.table`.
    # row_restriction: ""    # Optional Storage Read API `row_restriction` — a SQL predicate (e.g.
    # selected_fields: []    # Optional column projection for `read_api` mode — the columns to read.
    # max_streams: 1    # Maximum number of Storage Read API streams to request (`read_api` mode).
    # stream_concurrency: 4    # How many Storage Read streams to consume concurrently (#621).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery Yes
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
project_id
required		 string — GCP project ID against which the query is billed and run.
auth
required		 object — Authentication — the auth field, consistent with every other connector.
query
required		 string — SQL statement to execute. May contain ${field.path} placeholders that are resolved against the parent-record context at runtime as positional ? markers; matched values are appended to params when the query is sent.
use_legacy_sql boolean false Whether to use BigQuery's legacy SQL dialect. Defaults to false (Standard SQL). Set to true only for tables that use legacy [project:dataset.table] references.
location string — Optional location override for non-US jobs ("EU", "asia-east1"). When None, BigQuery uses the default location for the queried tables.
max_results_per_page integer 1000 Maximum rows per page when calling jobs.getQueryResults. Smaller values trade more HTTP round-trips for lower memory; larger values trade memory for fewer requests. Defaults to 1000.
params array [] Positional bind parameters for the query, sent as POSITIONAL query parameters in declaration order before any context-derived values. Each value is typed from its JSON form: an integer as INT64, any other number as FLOAT64, a boolean as BOOL, and everything else (strings, arrays, objects, null) as STRING.
statement_timeout integer 60 Per-statement server-side timeout. Forwarded to the timeoutMs field on jobs.query. Defaults to 60 seconds. If BigQuery does not finish the query within this window it responds with jobComplete=false; the source then polls jobs.getQueryResults until the job completes.
poll_timeout integer 300 Maximum wall-clock time the source will spend polling jobs.getQueryResults for a job that keeps reporting jobComplete=false, before giving up with FaucetError::Source. Without this cap a job that never completes would loop forever. Defaults to 300 seconds. Set to 0 to disable the cap and poll indefinitely. Only the *completion* wait is bounded; once the job is complete, ordinary pageToken paging is unaffected.
batch_size integer 1000 Records per emitted StreamPage. Rows returned by BigQuery are re-framed into pages of this size — every time the buffer reaches batch_size, a page is yielded. Defaults to DEFAULT_BATCH_SIZE.
read_api boolean false Arrow columnar **Storage Read API** mode (#380). When true, the source reads read_table directly via the BigQuery Storage Read gRPC API as Arrow RecordBatches (no jobs.query), driving the columnar fast path when the sink is also columnar and decoding Arrow → JSON on the row path otherwise. Requires a binary built with this crate's arrow feature and a read_table; the query field is ignored in this mode. Full extract only — no incremental bookmark.
read_table string — Table to read in read_api mode: dataset.table (billed to project_id) or a fully-qualified project.dataset.table. Required when read_api is set; ignored otherwise.
row_restriction string — Optional Storage Read API row_restriction — a SQL predicate (e.g. state = "CA") pushed to the read session so BigQuery filters rows server-side. Only used in read_api mode.
selected_fields array — Optional column projection for read_api mode — the columns to read. Empty means all columns.
max_streams integer 1 Maximum number of Storage Read API streams to request (read_api mode). Defaults to 1 (a single ordered stream). Higher values let BigQuery shard large tables; the source reads the returned streams sequentially.
stream_concurrency integer 4 How many Storage Read streams to consume concurrently (#621).
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub