Azure Blob Storage

Install

cargo install faucet-cli --features sink-azure-blob
faucet init --sink azure-blob

# Cargo.toml — use it as a library
cargo add faucet-sink-azure-blob

Starter config

sink:
  type: azure-blob
  config:
    container: ""    # REQUIRED — Blob container / ADLS Gen2 filesystem name.
    # account: ""    # Storage-account name.
    # auth: { type: account_key }    # one of: account_key, sas_token, connection_string, managed_identity, service_principal, default
      # --- Alternative variants — replace the block above with one of these ---
      # type: sas_token
      # config:
        # sas_token: ""    # REQUIRED
      #
      # type: connection_string
      # config:
        # connection_string: ""    # REQUIRED
      #
      # type: managed_identity
      # config:
        # client_id: ""
      #
      # type: service_principal
      # config:
        # client_id: ""    # REQUIRED
        # client_secret: ""    # REQUIRED
        # tenant_id: ""    # REQUIRED
        # authority_host: ""
      #
      # type: default
    # endpoint: ""    # Custom blob endpoint (e.g.
    # allow_http: false    # Permit plaintext HTTP (required for a local Azurite emulator).
    # use_emulator: false    # Target the Azurite storage emulator with its well-known `devstoreaccount1` credentials.
    # timeout_secs: 0    # Seconds one request (including reading its body) may take.
    # connect_timeout_secs: 10    # Seconds to wait for a connection to the storage endpoint.
    # max_retries: 10    # Retries of a failed request, including resuming an interrupted body read.
    # retry_timeout_secs: 600    # Seconds after a request first went out during which it may still be retried or its body resumed.
    # prefix: ""    # Object-name prefix for written objects.
    # path: ""    # Object key template inside the container, after `prefix` — the file sink's `path` (#777).
    # if_exists: null    # What happens when a file this run writes already exists (`if_exists`).
    # write_mode: null    # Pipeline-level write mode (`write_mode`).
    # format: null    # On-the-wire format of objects written by the Azure Blob sink (#604).
    # file_extension: ".jsonl"    # File extension for written objects (default `.jsonl`).
    # max_records_per_file: 0    # Hard cap on records per uploaded object.
    # max_bytes_per_file: 0    # Maximum **bytes** per object before rolling to a new one (#618).
    # concurrency: 10    # Maximum number of uploads in flight (default 10): a page that rolls into several objects uploads them concurrently while the next object is encoded, and a large blob's blocks go up this many at a time.
    # batch_size: 1000    # Records per object when `max_records_per_file` is not set (without `path`).
    # compression: null    # User-facing compression config.
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # parquet: { ... }    # Parquet write options.
    # json_lines: { ... }    # JSON Lines write options.
    # encryption: null    # Encrypt blobs at rest (#777; the `encryption` feature).
    # scratch_dir: ""    # Directory the sink builds each blob in before uploading it (default: the system temporary directory).

Capabilities

Config fields