faucetstream
Get started

Connectors / Warehouses

Cloud Spanner

source

Google Cloud Spanner query source (streaming SQL, incremental replication)

Install

cargo install faucet-cli --features source-spanner
faucet init --source spanner
# Cargo.toml — use it as a library
cargo add faucet-source-spanner

Starter config

source:
  type: spanner
  config:
    project_id: ""    # REQUIRED — GCP project ID that owns the Spanner instance.
    instance: ""    # REQUIRED — Spanner instance ID.
    database: ""    # REQUIRED — Database name within the instance.
    # auth: { type: service_account_key_path }    # one of: service_account_key_path, service_account_key, application_default
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_key
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
    # max_sessions: 100    # Upper bound on pooled Spanner sessions (server-side resources).
    # emulator_host: ""    # Emulator endpoint override (`host:port`, e.g.
    query: ""    # REQUIRED — GoogleSQL query to run.
    # params: {}    # Named bind parameters for the query.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # exact_staleness_secs: 0    # Read at a timestamp this many seconds in the past instead of a strong read.
    # replication: { type: full }    # one of: full, incremental
      # --- Alternative variants — replace the block above with one of these ---
      # type: incremental
      # column: ""    # REQUIRED
      # initial_value: null    # REQUIRED
    # state_key: ""    # Explicit state-store key for the bookmark.
    # shard: null    # Optional primary-key range sharding for clustered (Mode B) execution.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental Yes
Dataset discovery Yes
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
project_id
required		 string — GCP project ID that owns the Spanner instance.
instance
required		 string — Spanner instance ID.
database
required		 string — Database name within the instance.
query
required		 string — GoogleSQL query to run. Use named parameters (@name) for params, and the named parameter @bookmark to bind the incremental cursor server-side.
auth object type: application_default Credentials. Defaults to Application Default Credentials.
max_sessions integer 100 Upper bound on pooled Spanner sessions (server-side resources). Channels are sized automatically at one per 100 sessions.
emulator_host string — Emulator endpoint override (host:port, e.g. localhost:9010). When set, the connection is plaintext and unauthenticated — exactly what SPANNER_EMULATOR_HOST does, but scoped to this connector so parallel pipelines/tests don't race on a process-global env var. The env var is still honored when this field is unset.
params object {} Named bind parameters for the query. Values must be scalars (string, integer, float, or boolean) — Spanner parameters are typed, and null/array/object values have no unambiguous parameter type.
batch_size integer 1000 Records per emitted StreamPage. 0 emits the whole result set as a single page. Defaults to DEFAULT_BATCH_SIZE.
exact_staleness_secs integer — Read at a timestamp this many seconds in the past instead of a strong read. Stale reads can be served by any replica, offloading the leader — at the cost of missing rows committed within the staleness window.
replication object type: full Replication mode. Defaults to SpannerReplication::Full.
state_key string — Explicit state-store key for the bookmark. When unset, a key is derived from the database path and a query fingerprint.
shard object — Optional primary-key range sharding for clustered (Mode B) execution.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub