Connectors / Streaming & queues

RabbitMQ

source

RabbitMQ (AMQP 0.9.1) source; queue consume, ack after sink confirm, idle/max termination

Install

cargo install faucet-cli --features source-rabbitmq
faucet init --source rabbitmq
# Cargo.toml — use it as a library
cargo add faucet-source-rabbitmq

Starter config

source:
  type: rabbitmq
  config:
    # url: ""    # Full AMQP URI, e.g.
    # host: ""    # Broker host.
    # port: 0    # Broker port.
    # vhost: ""    # Virtual host.
    # auth: { type: none }    # one of: none, plain, external
      # --- Alternative variants — replace the block above with one of these ---
      # type: plain
      # config:
        # username: ""    # REQUIRED
        # password: ""    # REQUIRED
      #
      # type: external
    # tls: { ... }    # TLS (`amqps://`) settings.
    # connection_name: ""    # Client connection name shown in the RabbitMQ management UI.
    # heartbeat_secs: 0    # Requested heartbeat interval in seconds (`0` disables heartbeats).
    # connect_timeout_secs: 30    # Upper bound, in seconds, on establishing the connection (TCP + AMQP handshake, including any retries).
    # connect_retries: 0    # Extra TCP connect attempts (exponential backoff) before giving up.
    queue: ""    # REQUIRED — Queue to consume from.
    # declare_queue: true    # Declare the queue before consuming (idempotent when it already exists with the same properties).
    # queue_durable: true    # Whether a declared queue is durable (survives a broker restart).
    # bindings: []    # Exchange → queue bindings to declare before consuming.
    # prefetch: 0    # Consumer QoS window: the most unacknowledged deliveries the broker sends ahead.
    # ack_mode: null    # When deliveries are acknowledged to the broker.
    # value_format: null    # How a message body maps to and from a JSON record.
    # on_decode_error: null    # What to do with a message whose body does not decode under `value_format`.
    # include_metadata: false    # Wrap each record as `{ "data": <body>, "exchange", "routing_key", "delivery_tag", "redelivered", "headers", … }` instead of emitting the decoded body alone.
    # consumer_tag: ""    # Consumer tag.
    # max_messages: 0    # Stop after this many messages.
    # idle_timeout_secs: 0    # Stop after this many seconds with no new message.
    # batch_size: 1000    # Messages per emitted page (and per acknowledgement under `on_sink_confirm`).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
queue
required
string — Queue to consume from.
url string — Full AMQP URI, e.g. amqp://user:pass@rabbit:5672/%2f (the vhost is URL-encoded; %2f is /). amqps:// selects TLS. Mutually exclusive with host/port/vhost.
host string — Broker host. Defaults to 127.0.0.1 when url is not set.
port integer — Broker port. Defaults to 5672 (5671 with TLS).
vhost string — Virtual host. Defaults to /.
auth object type: none Authentication. Defaults to RabbitMqAuth::None.
tls object enabled: false TLS settings. Disabled by default.
connection_name string — Client connection name shown in the RabbitMQ management UI.
heartbeat_secs integer — Requested heartbeat interval in seconds (0 disables heartbeats). Unset uses the broker's proposal.
connect_timeout_secs integer 30 Upper bound, in seconds, on establishing the connection (TCP + AMQP handshake, including any retries). Defaults to 30.
connect_retries integer 0 Extra TCP connect attempts (exponential backoff) before giving up. Defaults to 0 — an unreachable broker fails immediately.
declare_queue boolean true Declare the queue before consuming (idempotent when it already exists with the same properties). Defaults to true. Set false to consume a queue managed elsewhere (it must exist).
queue_durable boolean true Whether a declared queue is durable (survives a broker restart). Defaults to true. Must match an existing queue's durability.
bindings array — Exchange → queue bindings to declare before consuming.
prefetch integer — Consumer QoS window: the most unacknowledged deliveries the broker sends ahead. 0 is unlimited. Defaults to batch_size (capped at 65535; unlimited when batch_size is 0 or larger than 65535). Under ack_mode: on_sink_confirm it must be 0 or ≥ batch_size, or the broker would stop delivering before a page could fill.
ack_mode object on_sink_confirm When deliveries are acknowledged. Defaults to on_sink_confirm.
value_format object json How message bodies decode into records. Defaults to json.
on_decode_error object fail What to do with a body that fails to decode. Defaults to fail.
include_metadata boolean false Wrap each record as { "data": <body>, "exchange", "routing_key", "delivery_tag", "redelivered", "headers", … } instead of emitting the decoded body alone. Defaults to false.
consumer_tag string — Consumer tag. Defaults to a broker-generated one.
max_messages integer — Stop after this many messages. At least one of max_messages / idle_timeout_secs must be set so a run terminates.
idle_timeout_secs integer — Stop after this many seconds with no new message. At least one of max_messages / idle_timeout_secs must be set so a run terminates.
batch_size integer 1000 Messages per emitted page (and per acknowledgement under on_sink_confirm). Defaults to DEFAULT_BATCH_SIZE. 0 drains the whole run window into one page.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli