SFTP

Install

cargo install faucet-cli --features source-sftp
faucet init --source sftp

# Cargo.toml — use it as a library
cargo add faucet-source-sftp

Starter config

source:
  type: sftp
  config:
    host: ""    # REQUIRED — Server hostname or IP address.
    # port: 22    # Server port (default: 22).
    username: ""    # REQUIRED — SSH username.
    # known_hosts: { type: strict }    # one of: strict, accept_new, insecure
      # --- Alternative variants — replace the block above with one of these ---
      # type: accept_new
      #
      # type: insecure
    # connect_timeout_secs: 30    # Seconds the TCP connect, SSH handshake and authentication together may take, and then each SFTP channel's start (default 30).
    # keepalive_interval_secs: 15    # Seconds between SSH keepalives on a quiet connection (default 15); the connection is dropped after three go unanswered.
    path: ""    # REQUIRED — Remote path to read: a directory whose files are listed and streamed, or a single file.
    # glob: null    # Optional filename glob (`*` / `?`) applied to the basenames of files in a directory listing.
    # format: null    # Format of the remote files read by the SFTP source.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # concurrency: 4    # Maximum number of files read concurrently (default: 4).
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # orc: { ... }    # ORC read options.
    # parquet: { ... }    # Parquet reader options (the `parquet:` block of an object-store source).
    # max_object_bytes: 2147483648    # Largest file, in bytes, a format read whole into memory (JSON array, raw text, CSV, XML, Excel, Avro, ORC) may reach; a larger one fails with an error naming it instead of exhausting memory.
    # max_decoded_bytes: 0    # Most bytes one object may decode to in memory where a format can expand past its input: an Avro file's records (per batch where the file streams), an Excel workbook's parts and shared strings.
    # max_listed_entries: 1000000    # Entries one directory listing may scan (default 1,000,000).
    # list_timeout_secs: 300    # Seconds the directory listing may take (default 300), including the stat of each symbolic link in it; a slower listing fails the run with an error naming this field.

Capabilities

Config fields