Connectors / Streaming & queues
RabbitMQ
sourceRabbitMQ (AMQP 0.9.1) source; queue consume, ack after sink confirm, idle/max termination
Install
cargo install faucet-cli --features source-rabbitmq faucet init --source rabbitmq
# Cargo.toml — use it as a library cargo add faucet-source-rabbitmq
Starter config
source:
type: rabbitmq
config:
# url: "" # Full AMQP URI, e.g.
# host: "" # Broker host.
# port: 0 # Broker port.
# vhost: "" # Virtual host.
# auth: { type: none } # one of: none, plain, external
# --- Alternative variants — replace the block above with one of these ---
# type: plain
# config:
# username: "" # REQUIRED
# password: "" # REQUIRED
#
# type: external
# tls: { ... } # TLS (`amqps://`) settings.
# connection_name: "" # Client connection name shown in the RabbitMQ management UI.
# heartbeat_secs: 0 # Requested heartbeat interval in seconds (`0` disables heartbeats).
# connect_timeout_secs: 30 # Upper bound, in seconds, on establishing the connection (TCP + AMQP handshake, including any retries).
# connect_retries: 0 # Extra TCP connect attempts (exponential backoff) before giving up.
queue: "" # REQUIRED — Queue to consume from.
# declare_queue: true # Declare the queue before consuming (idempotent when it already exists with the same properties).
# queue_durable: true # Whether a declared queue is durable (survives a broker restart).
# bindings: [] # Exchange → queue bindings to declare before consuming.
# prefetch: 0 # Consumer QoS window: the most unacknowledged deliveries the broker sends ahead.
# ack_mode: null # When deliveries are acknowledged to the broker.
# value_format: null # How a message body maps to and from a JSON record.
# on_decode_error: null # What to do with a message whose body does not decode under `value_format`.
# include_metadata: false # Wrap each record as `{ "data": <body>, "exchange", "routing_key", "delivery_tag", "redelivered", "headers", … }` instead of emitting the decoded body alone.
# consumer_tag: "" # Consumer tag.
# max_messages: 0 # Stop after this many messages.
# idle_timeout_secs: 0 # Stop after this many seconds with no new message.
# batch_size: 1000 # Messages per emitted page (and per acknowledgement under `on_sink_confirm`).
Capabilities
| Replay | Non-deterministic replay |
|---|---|
| Exactly-once replay | No |
| Incremental | No |
| Dataset discovery | No |
| Reports lag | No |
| Compression | No |
Config fields
| Field | Type | Default | Description |
|---|---|---|---|
queuerequired | string | — | Queue to consume from. |
url | string | — | Full AMQP URI, e.g. amqp://user:pass@rabbit:5672/%2f (the vhost is URL-encoded; %2f is /). amqps:// selects TLS. Mutually exclusive with host/port/vhost. |
host | string | — | Broker host. Defaults to 127.0.0.1 when url is not set. |
port | integer | — | Broker port. Defaults to 5672 (5671 with TLS). |
vhost | string | — | Virtual host. Defaults to /. |
auth | object | type: none | Authentication. Defaults to RabbitMqAuth::None. |
tls | object | enabled: false | TLS settings. Disabled by default. |
connection_name | string | — | Client connection name shown in the RabbitMQ management UI. |
heartbeat_secs | integer | — | Requested heartbeat interval in seconds (0 disables heartbeats). Unset uses the broker's proposal. |
connect_timeout_secs | integer | 30 | Upper bound, in seconds, on establishing the connection (TCP + AMQP handshake, including any retries). Defaults to 30. |
connect_retries | integer | 0 | Extra TCP connect attempts (exponential backoff) before giving up. Defaults to 0 — an unreachable broker fails immediately. |
declare_queue | boolean | true | Declare the queue before consuming (idempotent when it already exists with the same properties). Defaults to true. Set false to consume a queue managed elsewhere (it must exist). |
queue_durable | boolean | true | Whether a declared queue is durable (survives a broker restart). Defaults to true. Must match an existing queue's durability. |
bindings | array | — | Exchange → queue bindings to declare before consuming. |
prefetch | integer | — | Consumer QoS window: the most unacknowledged deliveries the broker sends ahead. 0 is unlimited. Defaults to batch_size (capped at 65535; unlimited when batch_size is 0 or larger than 65535). Under ack_mode: on_sink_confirm it must be 0 or ≥ batch_size, or the broker would stop delivering before a page could fill. |
ack_mode | object | on_sink_confirm | When deliveries are acknowledged. Defaults to on_sink_confirm. |
value_format | object | json | How message bodies decode into records. Defaults to json. |
on_decode_error | object | fail | What to do with a body that fails to decode. Defaults to fail. |
include_metadata | boolean | false | Wrap each record as { "data": <body>, "exchange", "routing_key", "delivery_tag", "redelivered", "headers", … } instead of emitting the decoded body alone. Defaults to false. |
consumer_tag | string | — | Consumer tag. Defaults to a broker-generated one. |
max_messages | integer | — | Stop after this many messages. At least one of max_messages / idle_timeout_secs must be set so a run terminates. |
idle_timeout_secs | integer | — | Stop after this many seconds with no new message. At least one of max_messages / idle_timeout_secs must be set so a run terminates. |
batch_size | integer | 1000 | Messages per emitted page (and per acknowledgement under on_sink_confirm). Defaults to DEFAULT_BATCH_SIZE. 0 drains the whole run window into one page. |
Get started
Your first pipeline runs in five minutes.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli