Install
cargo install faucet-cli --features sink-parquet faucet init --sink parquet
# Cargo.toml — use it as a library cargo add faucet-sink-parquet
Starter config
sink:
type: parquet
config:
destination:
type: local_path # one of: local_path, s3
path: "" # REQUIRED
# --- Alternative variants — replace the block above with one of these ---
# type: s3
# bucket: "" # REQUIRED
# prefix: ""
# region: null
# endpoint_url: null
# credentials: null
# allow_http: false
# schema: { type: inferred } # one of: inferred, explicit
# --- Alternative variants — replace the block above with one of these ---
# type: explicit
# compression: "" # Parquet compression codec. — one of: uncompressed, snappy, gzip, zstd, lz4
# row_group_size: 1048576 # Maximum rows per Parquet row group.
# max_rows_per_file: null # Roll over to a new file once the current writer has accepted this many rows.
# max_bytes_per_file: null # Roll over to a new file once the writer's `bytes_written()` exceeds this threshold.
# batch_size: 1000 # Re-chunk size for `Sink::write_batch`.
Capabilities
| Delivery | At-least-once |
|---|---|
| Write modes | append |
| Exactly-once writes | No |
| Schema evolution | No |
| Scoped cleanup | No |
| Staged bulk load | No |
| Run rollback | No |
| Compression | Yes |
Config fields
| Field | Type | Default | Description |
|---|---|---|---|
destinationrequired | object | — | Where to write the Parquet files (local filesystem or S3). |
schema | object | — | How the Arrow schema is determined. None means infer from the first batch using DEFAULT_SAMPLE_SIZE records. |
compression | string | snappy | Compression codec for column data. |
row_group_size | integer | 1048576 | Maximum rows per Parquet row group. Defaults to DEFAULT_ROW_GROUP_SIZE. |
max_rows_per_file | integer | — | Roll over to a new file once the current writer has accepted this many rows. None writes everything to a single file (until flush()). |
max_bytes_per_file | integer | — | Roll over to a new file once the writer's bytes_written() exceeds this threshold. Compared after each batch — so the actual file size may slightly exceed the limit by one batch worth of data. |
batch_size | integer | 1000 | Re-chunk size for Sink::write_batch. When batch_size > 0 and an incoming page exceeds this many records, the sink slices the page into batch_size-sized chunks and runs the internal write path once per chunk. When batch_size = 0 (the "no batching" sentinel) the page is written through as-is. Defaults to DEFAULT_BATCH_SIZE. |
Get started
Your first pipeline runs in five minutes.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli