Connectors / Streaming & queues

NATS

source

NATS source; subject/JetStream drain with idle/max termination

Install

cargo install faucet-cli --features source-nats
faucet init --source nats
# Cargo.toml — use it as a library
cargo add faucet-source-nats

Starter config

source:
  type: nats
  config:
    # servers: ["nats://127.0.0.1:4222"]    # One or more NATS server URLs, e.g.
    # auth: { type: none }    # one of: none, token, user_password, n_key, creds_file
      # --- Alternative variants — replace the block above with one of these ---
      # type: token
      # config:
        # token: ""    # REQUIRED
      #
      # type: user_password
      # config:
        # username: ""    # REQUIRED
        # password: ""    # REQUIRED
      #
      # type: n_key
      # config:
        # nkey: ""    # REQUIRED
      #
      # type: creds_file
      # config:
        # path: ""    # REQUIRED
    # tls: null    # `tls:` on the wire: a flag or a settings block.
    # name: ""    # Optional client connection name (surfaced in NATS server monitoring).
    # ignore_discovered_servers: false    # Connect only to `servers`, ignoring the cluster members a server advertises.
    subject: ""    # REQUIRED — Subject to subscribe to.
    # queue_group: ""    # Optional queue group for core-NATS load-balanced subscriptions — all subscribers sharing a group split the subject's messages.
    # jetstream_stream: ""    # JetStream stream name.
    # jetstream_consumer: ""    # Name of the durable JetStream (pull) consumer to bind to.
    # max_messages: 0    # Stop after this many messages have been drained.
    # idle_timeout_secs: 0    # Stop after this many seconds elapse with no new message.
    # batch_size: 1000    # Messages per emitted `StreamPage`.
    # progress_interval_secs: 10    # JetStream only: send an in-progress ack (`+WPI`) every this many seconds for every message pulled but not yet acked (while a page is assembled and while the sink writes it), resetting the consumer's `ack_wait` so a slow page is not redelivered into the same run.
    # value_format: null    # How `NatsSourceConfig::value_format` decodes a payload.
    # include_metadata: false    # Wrap each record as `{ subject, sequence, message_id, payload }` so a downstream consumer can deduplicate redeliveries: `sequence` is the JetStream stream sequence (`null` in core mode) and `message_id` the `Nats-Msg-Id` header (`null` when absent).
    # on_decode_error: null    # `NatsSourceConfig::on_decode_error`.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
subject
required
string — Subject to subscribe to. Supports NATS wildcards (* for one token, > for the remaining tokens). In JetStream mode the durable consumer's own filter subject governs delivery and this field is informational.
servers array ["nats://127.0.0.1:4222"] One or more NATS server URLs, e.g. ["nats://127.0.0.1:4222"]. The client connects to the first reachable server and uses the rest for failover.
auth object type: none Authentication mode. Defaults to none (anonymous).
tls boolean | object false TLS: true to require it with the platform roots, or a block with ca_cert_path / ca_cert_pem, client_cert_path + client_key_path and insecure_skip_verify (which also requires TLS). Defaults to false.
name string — Optional client connection name (surfaced in NATS server monitoring).
ignore_discovered_servers boolean — Connect only to servers, ignoring the cluster members a server advertises. Defaults to false.
queue_group string — Optional queue group for core-NATS load-balanced subscriptions — all subscribers sharing a group split the subject's messages. Ignored in JetStream mode.
jetstream_stream string — JetStream stream name. When set together with jetstream_consumer the source pulls from a durable JetStream consumer instead of a core-NATS subscription.
jetstream_consumer string — Name of the durable JetStream (pull) consumer to bind to. Required when jetstream_stream is set; the consumer must already exist on the server.
max_messages integer — Stop after this many messages have been drained. At least one of max_messages / idle_timeout_secs must be set so a run terminates.
idle_timeout_secs integer — Stop after this many seconds elapse with no new message. At least one of max_messages / idle_timeout_secs must be set so a run terminates.
batch_size integer 1000 Messages per emitted StreamPage. Drained messages accumulate in an in-memory buffer and a page is yielded when the buffer reaches this size (or when the run terminates with a partial buffer). Defaults to DEFAULT_BATCH_SIZE.
progress_interval_secs integer 10 JetStream only: send an in-progress ack (+WPI) every this many seconds for every message pulled but not yet acked (while a page is assembled and while the sink writes it), resetting the consumer's ack_wait so a slow page is not redelivered into the same run. Keep it below the consumer's ack_wait (30 s by default). 0 disables. Default 10.
value_format object auto How a message payload becomes a record: auto (default — JSON when it parses, else the UTF-8 text), json, string or bytes (base64). A payload the format cannot represent fails the run rather than being altered (#789 MSG-39).
include_metadata boolean false Wrap each record as { subject, sequence, message_id, payload } so a downstream consumer can deduplicate redeliveries: sequence is the JetStream stream sequence (null in core mode) and message_id the Nats-Msg-Id header (null when absent). Default false (the payload alone).
on_decode_error object fail What a payload value_format cannot decode does: fail (default) fails the run; skip logs a warning and drops it — a JetStream message is terminated so it is not redelivered.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli