faucetstream
Get started

Connectors / Files & object stores

Apache Parquet

source

Apache Parquet file source (local, glob, or S3)

Install

cargo install faucet-cli --features source-parquet
faucet init --source parquet
# Cargo.toml — use it as a library
cargo add faucet-source-parquet

Starter config

source:
  type: parquet
  config:
    source:
      type: local_path    # one of: local_path, glob, s3
      path: ""    # REQUIRED
      # --- Alternative variants — replace the block above with one of these ---
      # type: glob
      # pattern: ""    # REQUIRED
      #
      # type: s3
      # bucket: ""    # REQUIRED
      # key: null
      # prefix: null
      # region: null
      # endpoint_url: null
      # credentials: null
    # batch_size: 1000    # Arrow `RecordBatch` size used as the per-page hint when streaming.
    # columns: null    # Optional column projection (column pruning).
    # concurrency: 4    # Maximum number of files to read concurrently for `Glob` / S3 prefix modes.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
source
required		 object — Where to read Parquet from — a local file, a local glob pattern, or S3.
batch_size integer 1000 Arrow RecordBatch size used as the per-page hint when streaming.
columns array — Optional column projection (column pruning).
concurrency integer 4 Maximum number of files to read concurrently for Glob / S3 prefix modes. Ignored for single-file modes.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub