faucetstream
Get started

Connectors / Streaming & queues

AWS Kinesis

sink

AWS Kinesis Data Streams producer with partial-failure retry

Install

cargo install faucet-cli --features sink-kinesis
faucet init --sink kinesis
# Cargo.toml — use it as a library
cargo add faucet-sink-kinesis

Starter config

sink:
  type: kinesis
  config:
    stream_name: ""    # REQUIRED — Kinesis Data Stream name.
    # region: ""    # AWS region.
    # endpoint_url: ""    # Custom endpoint URL for LocalStack / VPC endpoints.
    # credentials: { type: default }    # one of: default, profile, access_key, assume_role, web_identity
      # --- Alternative variants — replace the block above with one of these ---
      # type: profile
      # config:
        # name: ""    # REQUIRED
      #
      # type: access_key
      # config:
        # access_key_id: ""    # REQUIRED
        # secret_access_key: ""    # REQUIRED
        # session_token: ""
      #
      # type: assume_role
      # config:
        # role_arn: ""    # REQUIRED
        # session_name: ""
        # external_id: ""
      #
      # type: web_identity
    # partition_key: { type: random }    # one of: random, field, jsonpath, hash, static
      # --- Alternative variants — replace the block above with one of these ---
      # type: field
      # name: ""    # REQUIRED
      #
      # type: jsonpath
      # path: ""    # REQUIRED
      #
      # type: hash
      # path: ""    # REQUIRED
      #
      # type: static
      # value: ""    # REQUIRED
    # explicit_hash_key: { type: none }    # one of: none, field, jsonpath
      # --- Alternative variants — replace the block above with one of these ---
      # type: field
      # name: ""    # REQUIRED
      #
      # type: jsonpath
      # path: ""    # REQUIRED
    # value_format: null    # How each record is encoded into the Kinesis `Data` blob.
    # batch_size: 500    # Max entries per `PutRecords` request (1–500).
    # max_record_size_bytes: 1048576    # Per-record size ceiling in bytes (data + partition key; ≤ 1 MiB).
    # max_request_bytes: 5242880    # Per-request size ceiling in bytes (≤ 5 MiB); batches re-chunk to it.
    # concurrency: 4    # Bounded concurrent in-flight `PutRecords` requests.
    # retry: null    # Per-record retry for partial failures, grouped (#654 M20).
    # retry_max_attempts: 5    # **Deprecated** — use `retry.max_attempts`.
    # retry_initial_backoff_ms: 100    # **Deprecated** — use `retry.initial_backoff_ms`.
    # retry_max_backoff_ms: 30000    # **Deprecated** — use `retry.max_backoff_ms`.

Capabilities

Delivery At-least-once
Write modes append
Exactly-once writes No
Schema evolution No
Scoped cleanup No
Staged bulk load No
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
stream_name
required		 string — Kinesis Data Stream name.
region string — AWS region. None uses the SDK default chain.
endpoint_url string — Custom endpoint URL for LocalStack / VPC endpoints.
credentials object type: default AWS credentials. Defaults to the SDK default provider chain.
partition_key object type: random Partition-key derivation strategy. Default: random.
explicit_hash_key object type: none Optional explicit hash-key override.
value_format object json Record payload encoding. Default: json.
batch_size integer 500 Max entries per PutRecords request (1–500). Default 500.
max_record_size_bytes integer 1048576 Per-record size ceiling in bytes (data + partition key; ≤ 1 MiB). Oversized records fail per-record (DLQ-routable), never sent.
max_request_bytes integer 5242880 Per-request size ceiling in bytes (≤ 5 MiB); batches re-chunk to it.
concurrency integer 4 Bounded concurrent in-flight PutRecords requests. Default 4.
retry object — Per-record retry for partial failures, grouped (#654 M20). When present it supersedes the three deprecated flat retry_* keys below.
retry_max_attempts integer 5 **Deprecated** — use retry.max_attempts.
retry_initial_backoff_ms integer 100 **Deprecated** — use retry.initial_backoff_ms.
retry_max_backoff_ms integer 30000 **Deprecated** — use retry.max_backoff_ms.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub