faucetstream
Get started

Connectors / Files & object stores

Delta Lake

sink

Apache Delta Lake sink (append-only; local FS or S3/Azure/GCS)

Install

cargo install faucet-cli --features sink-delta
faucet init --sink delta
# Cargo.toml — use it as a library
cargo add faucet-sink-delta

Starter config

sink:
  type: delta
  config:
    table_uri: ""    # REQUIRED — Delta table location URI: `file:///abs/path`, `s3://bucket/key`, `abfss://…`, `gs://bucket/key`.
    # credentials: { type: default }    # one of: default, aws, azure, gcp
      # --- Alternative variants — replace the block above with one of these ---
      # type: aws
      # config:
        # access_key_id: ""
        # secret_access_key: ""
        # session_token: ""
        # region: ""
        # endpoint_url: ""
        # allow_http: false
      #
      # type: azure
      # config:
        # account_name: ""
        # access_key: ""
        # sas_token: ""
      #
      # type: gcp
      # config:
        # service_account_path: ""
        # service_account_key: ""
    # storage_options: {}    # Extra `storage_options` passed verbatim to delta-rs (e.g.
    # create_table: true    # Create the table (and its schema, from the inferred record shape) on the first write when it does not already exist.
    # partition_by: []    # Partition columns, applied only when the table is created.
    # schema_sample_size: 100    # Records sampled to infer the Arrow/Delta schema when creating the table.
    # batch_size: 1000    # Re-chunk size for `Sink::write_batch`.
    # target_file_size: 0    # Optional target data-file size (bytes) hint for the writer's rollover.

Capabilities

Delivery At-least-once
Write modes append
Exactly-once writes No
Schema evolution No
Scoped cleanup No
Staged bulk load No
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
table_uri
required		 string — Delta table location URI: file:///abs/path, s3://bucket/key, abfss://…, gs://bucket/key. A bare local path is accepted and treated as file://.
credentials object type: default Object-store credentials. Defaults to the ambient provider chain.
storage_options object {} Extra storage_options passed verbatim to delta-rs (e.g. a MinIO endpoint override, or any object-store key the DeltaCredentials enum does not model). Keys set here **win** over credential-derived keys. Keys that name an endpoint (*endpoint*, *_url, *_uri, *_host, a proxy, an authority host) or a local file (*_file, *_path, a service-account or application-credentials file) are checked like the typed fields by a host that confines configs (faucet serve).
create_table boolean true Create the table (and its schema, from the inferred record shape) on the first write when it does not already exist. When false, an absent table fails the first write with a typed error.
partition_by array [] Partition columns, applied only when the table is created. Ignored when appending to an existing table (its stored partitioning is used).
schema_sample_size integer 100 Records sampled to infer the Arrow/Delta schema when creating the table.
batch_size integer 1000 Re-chunk size for Sink::write_batch. Incoming pages larger than this are sliced into batch_size-row Arrow batches before writing. 0 writes each page through as one batch.
target_file_size integer — Optional target data-file size (bytes) hint for the writer's rollover. Advisory — delta-rs rolls files at buffer boundaries near this size.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub