faucetstream
Get started

Connectors / Files & object stores

Amazon S3

sink

AWS S3 object sink

Install

cargo install faucet-cli --features sink-s3
faucet init --sink s3
# Cargo.toml — use it as a library
cargo add faucet-sink-s3

Starter config

sink:
  type: s3
  config:
    bucket: ""    # REQUIRED — S3 bucket name.
    # prefix: ""    # Key prefix for written objects.
    # path: ""    # Object key template inside the bucket, after `prefix` — the file sink's `path` (#777).
    # if_exists: null    # What happens when a file this run writes already exists (`if_exists`).
    # write_mode: null    # Pipeline-level write mode (`write_mode`).
    # format: null    # On-the-wire format of objects written by the S3 sink.
    # region: ""    # AWS region.
    # endpoint_url: ""    # Custom endpoint URL for S3-compatible services (e.g.
    # credentials: { type: default }    # one of: default, profile, access_key, assume_role, web_identity, anonymous
      # --- Alternative variants — replace the block above with one of these ---
      # type: profile
      # config:
        # name: ""    # REQUIRED
      #
      # type: access_key
      # config:
        # access_key_id: ""    # REQUIRED
        # secret_access_key: ""    # REQUIRED
        # session_token: ""
      #
      # type: assume_role
      # config:
        # role_arn: ""    # REQUIRED
        # session_name: ""
        # external_id: ""
      #
      # type: web_identity
      #
      # type: anonymous
    # file_extension: ".jsonl"    # File extension for written objects (default: `.jsonl`).
    # max_records_per_file: 0    # Maximum records per object.
    # max_bytes_per_file: 0    # Maximum **bytes** per object before rolling to a new one (#618).
    # concurrency: 10    # Maximum number of uploads in flight (default: 10).
    # batch_size: 1000    # Records per S3 object when `max_records_per_file` is not set (without `path`).
    # compression: null    # User-facing compression config.
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # parquet: { ... }    # Parquet write options.
    # json_lines: { ... }    # JSON Lines write options.
    # encryption: null    # Encrypt objects at rest (#777; the `encryption` feature).
    # scratch_dir: ""    # Directory the sink builds each object in before uploading it (default: the system temporary directory).

Capabilities

Delivery At-least-once
Write modes append, overwrite
Exactly-once writes No
Schema evolution No
Scoped cleanup No
Staged bulk load No
Run rollback No
Compression Yes

Config fields

FieldTypeDefaultDescription
bucket
required		 string — S3 bucket name.
prefix string "" Key prefix for written objects.
path string — Object key template inside the bucket, after prefix — the file sink's path (#777). May contain {part} (numbered objects) and ${now.*} tokens; a trailing / is a directory of part-{part}<extension> objects. When set, file_extension is not used. Unset: every run writes new objects named <prefix><run id>-<part><file_extension> (<run id> is a fresh time-ordered UUID, <part> is 00001, 00002, …).
if_exists object replace What to do when an object of the same name already exists (if_exists): replace it (default), append to it (JSON Lines, CSV, raw text) or fail with error (#777). Needs path. mode is accepted as another name for this key, and overwrite / error_if_exists for its values.
write_mode object append append (default) or overwrite: write the run's objects into a hidden swap area and move them into place only after a successful run, then remove objects of an earlier run that match path and were not rewritten (#777). The move is one object at a time, so a reader listing the destination while it runs can see new objects beside old ones; a move that was interrupted is finished by the next run. Needs path.
format object json_lines Object format (default: json_lines). Set to parquet (with the arrow feature) to write Parquet objects and enable the columnar fast path.
region string — AWS region. None uses the SDK default.
endpoint_url string — Custom endpoint URL for S3-compatible services (e.g. MinIO).
credentials object type: default How to authenticate (default: the AWS default chain, i.e. the identity of the host faucet runs on). access_key names keys in the config; anonymous sends unsigned requests (public buckets).
file_extension string .jsonl File extension for written objects (default: .jsonl). Not used when path is set.
max_records_per_file integer — Maximum records per object. Since #618 the sink **accumulates across write_batch calls** and rolls to a new object when this (or max_bytes_per_file) is reached, so a small upstream page no longer means a small object. None removes the record cap; with neither cap set the whole run lands in one object, closed at flush.
max_bytes_per_file integer — Maximum **bytes** per object before rolling to a new one (#618).
concurrency integer 10 Maximum number of uploads in flight (default: 10). A page that rolls into several objects uploads them concurrently, up to this many, while the next object is encoded; a large object's multipart parts are also uploaded up to this many at a time. flush returns only after every upload has landed.
batch_size integer 1000 Records per S3 object when max_records_per_file is not set (without path). Records accumulate across Sink::write_batch calls and an object closes once it holds batch_size records, or at flush. Defaults to DEFAULT_BATCH_SIZE.
compression string | object auto Compression codec applied to each uploaded object body. Defaults to CompressionConfig::Auto — resolves against the object name (path's, else file_extension, so .jsonl.gz triggers gzip). Requires the crate-local compression feature. Note: this sink does **not** set the S3 Content-Encoding header, so consumers must decompress explicitly.
csv object delimiter: ","has_headers: truequote: "on_unknown_field: widen CSV dialect, used when format: csv (#604).
excel object header_row: 0 Worksheet name, used when format: xlsx (#604).
xml object record_element: recordroot_element: records Record framing, used when format: xml (#604).
avro object codec: null Writer schema and block codec, used when format: avro (#719).
parquet object row_group_size: 1048576 Parquet writer options, used when format: parquet (#777): compression (default zstd), row_group_size, explicit schema.
json_lines object pretty: false JSON Lines writer options (pretty), used when format: json_lines.
encryption object — Encrypt objects at rest (#777; the encryption feature). Scratch files are not encrypted while the run is in progress; those holding plaintext are kept in a private directory.
scratch_dir string — Directory the sink builds each object in before uploading it (default: the system temporary directory). JSON Lines and raw text go up in parts as they are written and need no scratch space; every other format needs room for each object being built (up to concurrency of them while uploads are in flight). A private subdirectory is created in it for each sink.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub