faucetstream
Get started

Connectors / Warehouses

Cloud Spanner

sink

Google Cloud Spanner mutation sink (upsert/delete, exactly-once)

Install

cargo install faucet-cli --features sink-spanner
faucet init --sink spanner
# Cargo.toml — use it as a library
cargo add faucet-sink-spanner

Starter config

sink:
  type: spanner
  config:
    project_id: ""    # REQUIRED — GCP project ID that owns the Spanner instance.
    instance: ""    # REQUIRED — Spanner instance ID.
    database: ""    # REQUIRED — Database name within the instance.
    # auth: { type: service_account_key_path }    # one of: service_account_key_path, service_account_key, application_default
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_key
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
    # max_sessions: 100    # Upper bound on pooled Spanner sessions (server-side resources).
    # emulator_host: ""    # Emulator endpoint override (`host:port`, e.g.
    table_name: ""    # REQUIRED — Target table name.
    # create_table: true    # Create the target table if it does not exist, inferring the columns from the first written page (#580).
    # batch_size: 1000    # Maximum rows per Spanner commit.
    # ddl_timeout_secs: 300    # Upper bound (seconds) on waiting for a schema-change operation (`ALTER TABLE …` long-running operation) to complete.
    # write_mode: null    # Write semantics for a sink.
    # key: []    # Key columns.
    # delete_marker: null    # Optional.
    # rollback: null    # Runtime rollback settings (#706): the run id to journal under, the run-id column, and whether to journal before-images / keep a replaced table.

Capabilities

Delivery Exactly-once (atomic watermark)
Write modes append, upsert, delete
Exactly-once writes Yes
Schema evolution Yes
Scoped cleanup Yes
Staged bulk load No
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
project_id
required		 string — GCP project ID that owns the Spanner instance.
instance
required		 string — Spanner instance ID.
database
required		 string — Database name within the instance.
table_name
required		 string — Target table name.
auth object type: application_default Credentials. Defaults to Application Default Credentials.
max_sessions integer 100 Upper bound on pooled Spanner sessions (server-side resources). Channels are sized automatically at one per 100 sessions.
emulator_host string — Emulator endpoint override (host:port, e.g. localhost:9010). When set, the connection is plaintext and unauthenticated — exactly what SPANNER_EMULATOR_HOST does, but scoped to this connector so parallel pipelines/tests don't race on a process-global env var. The env var is still honored when this field is unset.
create_table boolean true Create the target table if it does not exist, inferring the columns from the first written page (#580). Enabled by default: a first-ever sync cannot assume the destination already exists.
batch_size integer 1000 Maximum rows per Spanner commit. Defaults to DEFAULT_BATCH_SIZE.
ddl_timeout_secs integer 300 Upper bound (seconds) on waiting for a schema-change operation (ALTER TABLE … long-running operation) to complete. Defaults to 300.
write_mode object append Append (default), upsert, or delete.
key array [] Key columns. Required and non-empty for upsert/delete; ignored for append.
delete_marker object — Optional. Upsert only: rows whose field matches one of values are deletes; all others are upserts. The marker field is stripped from upsert rows before writing.
rollback object — Runtime rollback settings (#706): the run id to journal under, the run-id column, and whether to journal before-images / keep a replaced table. Injected per invocation by the CLI when a config has a top-level rollback: block — not meant to be written by hand.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub