faucetstream
Get started

Connectors / Streaming & queues

AWS SQS

source

AWS SQS source; long-poll ReceiveMessage, delete-after-confirm

Install

cargo install faucet-cli --features source-sqs
faucet init --source sqs
# Cargo.toml — use it as a library
cargo add faucet-source-sqs

Starter config

source:
  type: sqs
  config:
    queue_url: ""    # REQUIRED — SQS queue URL (e.g.
    # region: ""    # AWS region.
    # endpoint_url: ""    # Custom endpoint URL for LocalStack / VPC endpoints.
    # credentials: { type: default }    # one of: default, profile, access_key, assume_role, web_identity
      # --- Alternative variants — replace the block above with one of these ---
      # type: profile
      # config:
        # name: ""    # REQUIRED
      #
      # type: access_key
      # config:
        # access_key_id: ""    # REQUIRED
        # secret_access_key: ""    # REQUIRED
        # session_token: ""
      #
      # type: assume_role
      # config:
        # role_arn: ""    # REQUIRED
        # session_name: ""
        # external_id: ""
      #
      # type: web_identity
    # idle_timeout_secs: 0    # Stop after this many seconds without a new message.
    # max_messages: 0    # Stop after this many messages in total.
    # wait_time_seconds: 10    # Long-poll wait per `ReceiveMessage` call, in seconds (0–20).
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # visibility_extension_secs: 60    # Visibility timeout, in seconds, the source keeps renewing on every message it has received but not yet deleted (while a page is assembled and while the sink writes it), so a slow page is not redelivered into the same run.
    # include_metadata: false    # Wrap each record as `{ message_id, attributes, payload }`: the SQS `MessageId` (for downstream deduplication of redeliveries) and the message attributes, which are requested only when this is on.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
queue_url
required		 string — SQS queue URL (e.g. https://sqs.us-east-1.amazonaws.com/1234/my-q).
region string — AWS region. None uses the SDK default chain (env, profile, IMDS).
endpoint_url string — Custom endpoint URL for LocalStack / VPC endpoints.
credentials object type: default AWS credentials. Defaults to the SDK default provider chain.
idle_timeout_secs integer — Stop after this many seconds without a new message. At least one of idle_timeout_secs and max_messages must be set (mirrors the Kafka / Kinesis sources) — a batch run must terminate.
max_messages integer — Stop after this many messages in total.
wait_time_seconds integer 10 Long-poll wait per ReceiveMessage call, in seconds (0–20). Default 10.
batch_size integer 1000 Records per emitted StreamPage. 0 is the "no batching" sentinel: one page per assembled drain buffer. Default 1000.
visibility_extension_secs integer 60 Visibility timeout, in seconds, the source keeps renewing on every message it has received but not yet deleted (while a page is assembled and while the sink writes it), so a slow page is not redelivered into the same run. Each receive requests this window and it is renewed every third of it, so a queue whose own visibility timeout is shorter still holds the messages. 0 disables renewal and leaves the queue's own visibility timeout in charge. Default 60.
include_metadata boolean false Wrap each record as { message_id, attributes, payload }: the SQS MessageId (for downstream deduplication of redeliveries) and the message attributes, which are requested only when this is on. Default false (the decoded body alone).
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub