Connectors / APIs & web

WebSocket

source

WebSocket streaming source

Install

cargo install faucet-cli --features source-websocket
faucet init --source websocket
# Cargo.toml — use it as a library
cargo add faucet-source-websocket

Starter config

source:
  type: websocket
  config:
    url: ""    # REQUIRED — WebSocket endpoint, `ws://` or `wss://`.
    # auth: { type: none }    # one of: none, bearer, custom
      # --- Alternative variants — replace the block above with one of these ---
      # type: bearer
      # config:
        # token: ""    # REQUIRED
      #
      # type: custom
      # config:
        # headers: {}    # REQUIRED
    # subscribe_messages: []    # Subscription frames sent (in order) immediately after every (re)connect.
    # message_format: null    # How each incoming frame is converted into a record.
    # on_parse_error: null    # What to do when a `Json`-mode frame is not valid JSON.
    # envelope: false    # `false` (default) emits the record raw; `true` wraps it as `{ data, received_at, url }`.
    # ping_interval: 0    # If set, send a WebSocket Ping frame on this interval (seconds) to keep the connection alive through proxies/load balance…
    # max_messages: 0    # Stop after this many messages.
    # idle_timeout: 0    # Stop after this many seconds with no message.
    # reconnect: false    # Reconnect on transport error / non-1000 close.
    # reconnect_backoff: 1    # Base wait (seconds) for reconnect backoff.
    # reconnect_max_attempts: 0    # Cap on *consecutive* failed reconnects (resets on a received data frame — a handshake the server accepts and then closes still counts).
    # max_message_bytes: 0    # Bound the max WebSocket message/frame size (bytes) to prevent runaway memory.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # connect_timeout: 30    # Seconds allowed for the TCP connect + upgrade handshake and for sending the subscribe frames (default 30).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
url
required
string — WebSocket endpoint, ws:// or wss://. Supports {placeholder} parent-matrix context substitution.
auth object type: none Authentication applied to the HTTP upgrade request. Either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog.
subscribe_messages array [] Subscription frames sent (in order) immediately after every (re)connect. Empty = send nothing.
message_format object json How to interpret each incoming frame.
on_parse_error object fail In Json mode, what to do when a frame is not valid JSON.
envelope boolean false false (default) emits the record raw; true wraps it as { data, received_at, url }.
ping_interval integer — If set, send a WebSocket Ping frame on this interval (seconds) to keep the connection alive through proxies/load balancers.
max_messages integer — Stop after this many messages. At least one of max_messages / idle_timeout must be set.
idle_timeout integer — Stop after this many seconds with no message. The idle clock keeps ticking across reconnect gaps, so it also caps a connection outage.
reconnect boolean false Reconnect on transport error / non-1000 close.
reconnect_backoff integer 1 Base wait (seconds) for reconnect backoff. Default 1s.
reconnect_max_attempts integer — Cap on *consecutive* failed reconnects (resets on a received data frame — a handshake the server accepts and then closes still counts). None = unlimited (then idle_timeout is the natural cap).
max_message_bytes integer — Bound the max WebSocket message/frame size (bytes) to prevent runaway memory. None = tungstenite default (64 MiB message / 16 MiB frame).
batch_size integer 1000 Records per emitted StreamPage. Default DEFAULT_BATCH_SIZE. 0 drains the entire run window into a single page (same sentinel as the Kafka source).
connect_timeout integer 30 Seconds allowed for the TCP connect + upgrade handshake and for sending the subscribe frames (default 30). Expiry counts as a connect failure, so reconnect retries it.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli