faucetstream
Get started

Connectors / Warehouses

Snowflake

source

Snowflake query source (SQL REST API)

Install

cargo install faucet-cli --features source-snowflake
faucet init --source snowflake
# Cargo.toml — use it as a library
cargo add faucet-source-snowflake

Starter config

source:
  type: snowflake
  config:
    account: ""    # REQUIRED — Snowflake account identifier (e.g.
    warehouse: ""    # REQUIRED — Warehouse to use for the session.
    database: ""    # REQUIRED — Database name.
    schema: ""    # REQUIRED — Schema name.
    # role: ""    # Optional role to assume for the session.
    auth:
      type: key_pair    # one of: key_pair, oauth
      config:
        user: ""    # REQUIRED — The Snowflake user account name.
        private_key_pem: ""    # REQUIRED — PEM-encoded RSA private key.
      # --- Alternative variants — replace the block above with one of these ---
      # type: oauth
      # config:
        # token: ""    # REQUIRED
    query: ""    # REQUIRED — SQL statement to execute.
    # params: []    # Positional bind parameters for the query, one per `?` marker the query holds, in order (a `?` inside a string literal, quoted identifier or comment is not a marker).
    # statement_timeout: 0    # Server-side statement timeout, sent as the `timeout` field of `POST /api/v2/statements`.
    # poll_timeout: 172800    # Maximum wall-clock time the source will spend polling an asynchronous statement (one for which the initial `POST /api/v2/statements` returns HTTP 202) before giving up with `FaucetError::Source`.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # partition_concurrency: 4    # How many result **partitions** to fetch concurrently (#621).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery Yes
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
account
required		 string — Snowflake account identifier (e.g. "xy12345.us-east-1").
warehouse
required		 string — Warehouse to use for the session.
database
required		 string — Database name.
schema
required		 string — Schema name.
auth
required		 object — Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog. A shared provider must yield a Bearer or Token credential, which maps onto SnowflakeAuth::OAuth; key-pair JWT is always inline.
query
required		 string — SQL statement to execute. May contain ${field.path} placeholders that are resolved against the parent-record context at runtime; resolved values are sent as positional bind parameters at the position of their token, interleaved with params in textual order.
role string — Optional role to assume for the session.
params array [] Positional bind parameters for the query, one per ? marker the query holds, in order (a ? inside a string literal, quoted identifier or comment is not a marker). Snowflake's SQL REST API uses 1-based positional binds in the JSON request body (see the bindings field in the [SQL API docs](https://docs.snowflake.com/en/developer-guide/sql-api/submitting-requests#using-bind-variables-in-a-statement)).
statement_timeout integer 0 Server-side statement timeout, sent as the timeout field of POST /api/v2/statements. Snowflake **cancels** a statement that runs longer (error 000630). Defaults to 0: Snowflake's maximum. This is not how long the submit waits before going async — that is fixed by the SQL API; see poll_timeout for the client cap.
poll_timeout integer 172800 Maximum wall-clock time the source will spend polling an asynchronous statement (one for which the initial POST /api/v2/statements returns HTTP 202) before giving up with FaucetError::Source. Without this cap a statement that never finishes would loop forever. The statement is cancelled server-side when the cap is reached. Defaults to 172800 seconds (48 hours, Snowflake's own maximum statement time), so a long extract is never cut short by the client. Set to 0 to disable the cap and poll indefinitely.
batch_size integer 1000 Records per emitted StreamPage.
partition_concurrency integer 4 How many result **partitions** to fetch concurrently (#621).
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub