Azure Blob Storage

Install

cargo install faucet-cli --features source-azure-blob
faucet init --source azure-blob

# Cargo.toml — use it as a library
cargo add faucet-source-azure-blob

Starter config

source:
  type: azure-blob
  config:
    container: ""    # REQUIRED — Blob container / ADLS Gen2 filesystem name.
    # account: ""    # Storage-account name.
    # auth: { type: account_key }    # one of: account_key, sas_token, connection_string, managed_identity, service_principal, default
      # --- Alternative variants — replace the block above with one of these ---
      # type: sas_token
      # config:
        # sas_token: ""    # REQUIRED
      #
      # type: connection_string
      # config:
        # connection_string: ""    # REQUIRED
      #
      # type: managed_identity
      # config:
        # client_id: ""
      #
      # type: service_principal
      # config:
        # client_id: ""    # REQUIRED
        # client_secret: ""    # REQUIRED
        # tenant_id: ""    # REQUIRED
        # authority_host: ""
      #
      # type: default
    # endpoint: ""    # Custom blob endpoint (e.g.
    # allow_http: false    # Permit plaintext HTTP (required for a local Azurite emulator).
    # use_emulator: false    # Target the Azurite storage emulator with its well-known `devstoreaccount1` credentials.
    # timeout_secs: 0    # Seconds one request (including reading its body) may take.
    # connect_timeout_secs: 10    # Seconds to wait for a connection to the storage endpoint.
    # max_retries: 10    # Retries of a failed request, including resuming an interrupted body read.
    # retry_timeout_secs: 600    # Seconds after a request first went out during which it may still be retried or its body resumed.
    # prefix: ""    # Object name prefix filter.
    # include: null    # Glob over the whole object key selecting which listed objects to read (e.g.
    # object_keys: []    # Explicit object names.
    # file_format: null    # Format of objects stored in the container.
    # max_objects: 0    # Hard cap on the number of objects read (after listing).
    # concurrency: 10    # Maximum concurrent object reads (default: 10).
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # verify_length: true    # Verify each blob's byte length against the length the store advertises, failing the read with `FaucetError::Source` on a short (truncated) or over-long transfer.
    # verify_checksum: false    # Not supported on Azure Blob: `object_store` exposes no `Content-MD5` attribute and an Azure ETag is not a content hash, so there is nothing to verify against.
    # max_object_bytes: 2147483648    # Largest object, once decompressed, that a format read whole (JSON array, raw text, CSV, XML, Excel, Avro, ORC, buffered Parquet) may reach; a larger one fails with an error instead of exhausting memory.
    # max_decoded_bytes: 0    # Most bytes one object may decode to in memory where a format can expand past its input: an Avro file's records (per batch where the file streams), an Excel workbook's parts and shared strings.
    # compression: null    # User-facing compression config.
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # orc: { ... }    # ORC read options.
    # parquet: { ... }    # Parquet reader options (the `parquet:` block of an object-store source).

Capabilities

Config fields