faucetstream
Get started

Connectors / Files & object stores

Delta Lake

source

Apache Delta Lake source (local FS or S3/Azure/GCS; time travel, projection)

Install

cargo install faucet-cli --features source-delta
faucet init --source delta
# Cargo.toml — use it as a library
cargo add faucet-source-delta

Starter config

source:
  type: delta
  config:
    table_uri: ""    # REQUIRED — Delta table location URI: `file:///abs/path`, `s3://bucket/key`, `abfss://…`, `gs://bucket/key`.
    # credentials: { type: default }    # one of: default, aws, azure, gcp
      # --- Alternative variants — replace the block above with one of these ---
      # type: aws
      # config:
        # access_key_id: ""
        # secret_access_key: ""
        # session_token: ""
        # region: ""
        # endpoint_url: ""
        # allow_http: false
      #
      # type: azure
      # config:
        # account_name: ""
        # access_key: ""
        # sas_token: ""
      #
      # type: gcp
      # config:
        # service_account_path: ""
        # service_account_key: ""
    # storage_options: {}    # Extra `storage_options` passed verbatim to delta-rs (e.g.
    # version: 0    # Time travel: read the table as of this version.
    # timestamp: ""    # Time travel: read the table as of this RFC 3339 timestamp.
    # columns: []    # Projection pushdown: only read these columns.
    # batch_size: 1000    # Page size hint for `stream_pages`.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
table_uri
required		 string — Delta table location URI: file:///abs/path, s3://bucket/key, abfss://…, gs://bucket/key. A bare local path is accepted and treated as file://.
credentials object type: default Object-store credentials. Defaults to the ambient provider chain.
storage_options object {} Extra storage_options passed verbatim to delta-rs (e.g. a MinIO endpoint override, or any object-store key the DeltaCredentials enum does not model). Keys set here **win** over credential-derived keys. Keys that name an endpoint (*endpoint*, *_url, *_uri, *_host, a proxy, an authority host) or a local file (*_file, *_path, a service-account or application-credentials file) are checked like the typed fields by a host that confines configs (faucet serve).
version integer — Time travel: read the table as of this version. Mutually exclusive with timestamp.
timestamp string — Time travel: read the table as of this RFC 3339 timestamp. Mutually exclusive with version.
columns array [] Projection pushdown: only read these columns. Empty = all columns.
batch_size integer 1000 Page size hint for stream_pages. Governs how many rows accumulate before a StreamPage is emitted. 0 is the "no batching" sentinel — one page per data file, however many row groups it holds.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub