faucetstream
Get started

Connectors / Warehouses

Snowflake

sink

Snowflake SQL REST API sink

Install

cargo install faucet-cli --features sink-snowflake
faucet init --sink snowflake
# Cargo.toml — use it as a library
cargo add faucet-sink-snowflake

Starter config

sink:
  type: snowflake
  config:
    account: ""    # REQUIRED — Snowflake account identifier (e.g.
    warehouse: ""    # REQUIRED — Warehouse to use for the session.
    database: ""    # REQUIRED — Database name.
    schema: ""    # REQUIRED — Schema name.
    table: ""    # REQUIRED — Target table name.
    # role: ""    # Optional role to assume for the session; without it Snowflake uses the user's default role.
    # statement_timeout: 0    # Server-side statement timeout, sent as the `timeout` field of every statement.
    # create_table: true    # Create the target table if it does not exist, inferring the column **names** from the first written page (#580), and add a nullable `STRING` column for any field a later page brings that the table lacks.
    # commit_rows: 0    # Commit-group size for the cross-page accumulator (#617).
    # commit_bytes: 0    # Estimated-bytes counterpart of `commit_rows` (#617).
    auth:
      type: key_pair    # one of: key_pair, oauth
      config:
        user: ""    # REQUIRED — The Snowflake user account name.
        private_key_pem: ""    # REQUIRED — PEM-encoded RSA private key.
      # --- Alternative variants — replace the block above with one of these ---
      # type: oauth
      # config:
        # token: ""    # REQUIRED
    # batch_size: 1000    # Maximum number of records sent per Snowflake SQL REST API request.
    # poll_timeout: 172800    # Maximum wall-clock time to wait for an asynchronously-executed INSERT to finish.
    # bulk_load: null    # Arrow columnar **bulk-load** mode (#381): buffer Arrow `RecordBatch`es to Parquet, upload to an external cloud stage's backing storage, then `COPY INTO <table> FROM @stage FILE_FORMAT=(TYPE=PARQUET)` over the SQL REST API.

Capabilities

Delivery Exactly-once (atomic watermark)
Write modes append
Exactly-once writes Yes
Schema evolution No
Scoped cleanup No
Staged bulk load Yes
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
account
required		 string — Snowflake account identifier (e.g. "xy12345.us-east-1").
warehouse
required		 string — Warehouse to use for the session.
database
required		 string — Database name.
schema
required		 string — Schema name.
table
required		 string — Target table name.
auth
required		 object — Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog. A shared provider must yield a Bearer or Token credential, which maps onto SnowflakeAuth::OAuth; key-pair JWT is always inline.
role string — Optional role to assume for the session; without it Snowflake uses the user's default role.
statement_timeout integer 0 Server-side statement timeout, sent as the timeout field of every statement. Snowflake **cancels** a statement that runs longer. Defaults to 0: Snowflake's maximum.
create_table boolean true Create the target table if it does not exist, inferring the column **names** from the first written page (#580), and add a nullable STRING column for any field a later page brings that the table lacks. Enabled by default: a first-ever sync cannot assume the destination already exists.
commit_rows integer — Commit-group size for the cross-page accumulator (#617).
commit_bytes integer — Estimated-bytes counterpart of commit_rows (#617). Rows are a poor proxy for how much work a warehouse commit is; this bounds the buffered size. 0 means no byte limit; see commit_rows for the default when both are unset.
batch_size integer 1000 Maximum number of records sent per Snowflake SQL REST API request. Defaults to DEFAULT_BATCH_SIZE (1000), which matches the documented sweet spot for the SQL REST API.
poll_timeout integer 172800 Maximum wall-clock time to wait for an asynchronously-executed INSERT to finish. Snowflake's SQL REST API answers an accepted but not-yet-finished statement with HTTP 202 and a statementHandle; the sink polls GET /api/v2/statements/{handle} until the statement reports success before counting the rows as written. Without this the sink would report success the moment Snowflake *accepted* the statement, losing durability. The statement is cancelled server-side when the cap is reached. Defaults to 172800 seconds (48 hours, Snowflake's own maximum statement time), so a large commit group on a small warehouse is never cut short by the client. Set to 0 to poll indefinitely.
bulk_load object — Arrow columnar **bulk-load** mode (#381): buffer Arrow RecordBatches to Parquet, upload to an external cloud stage's backing storage, then COPY INTO <table> FROM @stage FILE_FORMAT=(TYPE=PARQUET) over the SQL REST API. Requires a binary built with this crate's arrow feature; a config that sets it on an arrow-off build is rejected at construction.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub