AWS Kinesis

Install

cargo install faucet-cli --features source-kinesis
faucet init --source kinesis

# Cargo.toml — use it as a library
cargo add faucet-source-kinesis

Starter config

source:
  type: kinesis
  config:
    stream_name: ""    # REQUIRED — Kinesis Data Stream name.
    # region: ""    # AWS region.
    # endpoint_url: ""    # Custom endpoint URL for LocalStack / VPC endpoints.
    # credentials: { type: default }    # one of: default, profile, access_key, assume_role, web_identity
      # --- Alternative variants — replace the block above with one of these ---
      # type: profile
      # config:
        # name: ""    # REQUIRED
      #
      # type: access_key
      # config:
        # access_key_id: ""    # REQUIRED
        # secret_access_key: ""    # REQUIRED
        # session_token: ""
      #
      # type: assume_role
      # config:
        # role_arn: ""    # REQUIRED
        # session_name: ""
        # external_id: ""
      #
      # type: web_identity
    # start_position: { type: trim_horizon }    # one of: trim_horizon, latest, at_timestamp, at_sequence_number, after_sequence_number
      # --- Alternative variants — replace the block above with one of these ---
      # type: latest
      #
      # type: at_timestamp
      # timestamp_secs: 0    # REQUIRED
      #
      # type: at_sequence_number
      # sequence: ""    # REQUIRED
      #
      # type: after_sequence_number
      # sequence: ""    # REQUIRED
    # shard_ids: []    # Explicit shard-id allowlist; empty = every shard.
    # include_closed: false    # Also consume closed (post-resharding) shards still inside the retention window that have no bookmark.
    # poll_interval_ms: 1000    # Base wait between `GetRecords` calls per shard, in milliseconds.
    # records_per_request: 500    # `GetRecords` `Limit` per request (1–10000).
    # shard_concurrency: 4    # Max concurrent `GetRecords` calls; every shard is still read, in turn.
    # idle_termination_secs: 0    # Stop after this many seconds without a new record across all shards.
    # max_messages: 0    # Stop after this many records in total.
    # value_format: null    # How each record's `Data` blob is decoded into the emitted `data` field.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # on_decode_error: null    # `KinesisSourceConfig::on_decode_error`.

Capabilities

Config fields