faucetstream
Get started

Connectors / APIs & web

XML / SOAP

source

XML / SOAP API source with XML→JSON conversion

Install

cargo install faucet-cli --features source-xml
faucet init --source xml
# Cargo.toml — use it as a library
cargo add faucet-source-xml

Starter config

source:
  type: xml
  config:
    base_url: ""    # REQUIRED — Base URL of the API.
    path: ""    # REQUIRED — Request path (appended to base_url).
    method: ""    # REQUIRED — HTTP method (GET or POST for SOAP).
    auth:
      type: none    # one of: none, bearer, basic, custom
      # --- Alternative variants — replace the block above with one of these ---
      # type: bearer
      # config:
        # token: ""    # REQUIRED
      #
      # type: basic
      # config:
        # username: ""    # REQUIRED
        # password: ""    # REQUIRED
      #
      # type: custom
      # config:
        # headers: {}    # REQUIRED
    # body: ""    # Optional request body (e.g.
    # soap: null    # Optional first-class SOAP ergonomics block.
    # records_element_path: ""    # Dot-separated path to the repeating element in the XML response (e.g.
    # pagination: { type: PageNumber }    # one of: PageNumber, Offset, BodyCursor
      # --- Alternative variants — replace the block above with one of these ---
      # type: Offset
      # offset_param: ""    # REQUIRED
      # limit_param: ""    # REQUIRED
      # limit: 0    # REQUIRED
      # stop_when_short: false
      #
      # type: BodyCursor
      # next_token_path: ""    # REQUIRED
      # next_body: ""    # REQUIRED
    # max_pages: 0    # Maximum number of pages to fetch.
    # query_params: {}    # Query parameters to include in every request.
    # decode: []    # Response-decode pipeline (#540): a declarative chain applied to the raw response body before record extraction (`extract` an element's text → `base64`/`gunzip`/`unzip` → `parse` csv/xlsx/xml/json).
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # tls: null    # Optional client-certificate (mutual TLS) config.
    # timeout: 30    # Per-request timeout in seconds.
    # connect_timeout: 10    # Timeout for establishing a connection, in seconds.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
base_url
required		 string — Base URL of the API.
path
required		 string — Request path (appended to base_url).
method
required		 string — HTTP method (GET or POST for SOAP).
auth
required		 object — Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog.
body string — Optional request body (e.g. a raw SOAP envelope). Mutually exclusive with SoapConfig::body_inner when a soap block is set.
soap object — Optional first-class SOAP ergonomics block. When present, the source assembles the SOAP envelope, injects the version-appropriate headers, and (by default) resolves records_element_path relative to Envelope.Body. Sugar over the raw body path — see SoapConfig.
records_element_path string — Dot-separated path to the repeating element in the XML response (e.g. "Envelope.Body.GetUsersResponse.Users.User").
pagination object — Pagination configuration.
max_pages integer — Maximum number of pages to fetch.
query_params object {} Query parameters to include in every request. Empty by default.
decode array — Response-decode pipeline (#540): a declarative chain applied to the raw response body before record extraction (extract an element's text → base64/gunzip/unzip → parse csv/xlsx/xml/json). When non-empty, records come from the decoded output instead of records_element_path.
batch_size integer 1000 Records per emitted StreamPage. The event-driven XML parser accumulates matched subtrees into a buffer and yields whenever the buffer reaches this size. Defaults to DEFAULT_BATCH_SIZE.
tls object — Optional client-certificate (mutual TLS) config. When set, the source presents a client certificate on every request (data + inline auth token request). Requires the crate's mtls feature.
timeout integer 30 Per-request timeout in seconds. Covers one HTTP request (a stalled peer fails the request, so retries and resilience can act), not the whole run. null disables it. Defaults to 30.
connect_timeout integer 10 Timeout for establishing a connection, in seconds. null disables it. Defaults to 10.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub