Amazon DynamoDB

Install

cargo install faucet-cli --features source-dynamodb
faucet init --source dynamodb

# Cargo.toml — use it as a library
cargo add faucet-source-dynamodb

Starter config

source:
  type: dynamodb
  config:
    table_name: ""    # REQUIRED — Table name.
    # region: ""    # AWS region.
    # endpoint_url: ""    # Custom endpoint URL (DynamoDB Local, LocalStack, VPC endpoints).
    # credentials: { type: default }    # one of: default, profile, access_key, assume_role, web_identity
      # --- Alternative variants — replace the block above with one of these ---
      # type: profile
      # config:
        # name: ""    # REQUIRED
      #
      # type: access_key
      # config:
        # access_key_id: ""    # REQUIRED
        # secret_access_key: ""    # REQUIRED
        # session_token: ""
      #
      # type: assume_role
      # config:
        # role_arn: ""    # REQUIRED
        # session_name: ""
        # external_id: ""
      #
      # type: web_identity
    # mode: null    # What the source reads.
    # segments: 1    # Parallel scan segments (`TotalSegments`), 1–1000000.
    # max_concurrency: 8    # Segments read concurrently.
    # index_name: ""    # Secondary index to scan or query.
    # projection: ""    # `ProjectionExpression` — the attributes to return.
    # filter_expression: ""    # `FilterExpression` applied server-side after the read.
    # key_condition_expression: ""    # `KeyConditionExpression`.
    # expression_attribute_names: {}    # `ExpressionAttributeNames` (`#name` → attribute).
    # expression_attribute_values: {}    # `ExpressionAttributeValues` (`:value` → plain JSON value).
    # consistent_read: false    # Strongly consistent reads (table and local secondary indexes only).
    # scan_index_forward: true    # Query order: ascending sort key when `true` (default).
    # page_limit: 0    # `Limit` per Scan/Query request (items evaluated per call).
    # stream_arn: ""    # Stream ARN.
    # start_position: null    # Where a stream shard starts when no bookmark exists for it.
    # on_gap: null    # What to do when the bookmark points before the Streams trim horizon (a resume after more than 24 hours, a re-enabled str…
    # resnapshot_deletes: true    # `on_gap: resnapshot`: after the re-read, delete the destination rows of items it did not see (default `true`; the sink must support cleanup).
    # resnapshot_max_keys: 1000000    # `on_gap: resnapshot`: the most distinct keys the re-read tracks for its delete (default 1,000,000; memory grows with it).
    # poll_interval_ms: 1000    # Base wait between `GetRecords` calls per shard, in ms (floored at 200).
    # records_per_request: 1000    # `GetRecords` `Limit` per request (1–1000).
    # shard_concurrency: 4    # Shards read concurrently.
    # idle_termination_secs: 0    # Streams: stop after this many seconds without a new record.
    # max_messages: 0    # Streams: stop after this many change records.
    # retry: { ... }    # Retry/backoff settings exposed on both connector configs.
    # batch_size: 1000    # Records per emitted page.
    # value_format: null    # How records carry attribute values: `plain` JSON (the default) or DynamoDB's `typed` attribute JSON (`{"pk": {"S": "a"}, "tags": {"SS": [..]}}`).

Capabilities

Config fields