Amazon Redshift

source

Amazon Redshift query source (PostgreSQL wire; streaming rows, incremental replication)

Install

cargo install faucet-cli --features source-redshift
faucet init --source redshift

# Cargo.toml — use it as a library
cargo add faucet-source-redshift

Starter config

source:
  type: redshift
  config:
    # host: ""    # Cluster / endpoint host (e.g.
    # port: 5439    # Port.
    database: ""    # REQUIRED — Database name.
    # user: ""    # Database user.
    credentials:
      type: password    # one of: password, iam, redshift_data_api
      config:
        password: ""    # REQUIRED — The password (use `${env:…}` / `${vault:…}` to inject it, never a literal).
      # --- Alternative variants — replace the block above with one of these ---
      # type: iam
      # config:
        # region: ""
        # cluster_identifier: ""
        # db_user: ""
        # workgroup_name: ""
        # auto_create: false
        # db_groups: []
        # duration_secs: 0
        # aws_credentials: null
        # endpoint_url: ""
      #
      # type: redshift_data_api
      # config:
        # region: ""
        # cluster_identifier: ""
        # workgroup_name: ""
        # secret_arn: ""
        # db_user: ""
        # aws_credentials: null
        # endpoint_url: ""
    # tls: true    # Whether to require TLS.
    # tls_mode: null    # Explicit TLS mode; overrides `tls`.
    # ssl_root_cert: ""    # PEM file of the CA that signed the server certificate (the Redshift CA bundle), for `tls_mode: verify_ca | verify_full`.
    query: ""    # REQUIRED — SQL query to execute.
    # params: []    # Positional bind parameters for the query, applied in `$1, $2, …` order before any context- or bookmark-derived values.
    # max_connections: 10    # Maximum number of connections in the pool.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # replication: { type: full }    # one of: full, incremental
      # --- Alternative variants — replace the block above with one of these ---
      # type: incremental
      # column: ""    # REQUIRED
      # initial_value: null    # REQUIRED
    # state_key: ""    # Explicit state-store key for the incremental bookmark.

Capabilities

Replay	Non-deterministic replay
Exactly-once replay	No
Incremental	Yes
Dataset discovery	No
Reports lag	No
Compression	No

Config fields

Field	Type	Default	Description
`database` required	`string`	`—`	Database name.
`credentials` required	`object`	`—`	Authentication credentials.
`query` required	`string`	`—`	SQL query to execute. May contain `${field.path}` parent-context tokens (resolved to positional binds at runtime) and, for incremental replication, a `${bookmark}` token.
`host`	`string`	`""`	Cluster / endpoint host (e.g. `my-cluster.abc123.us-east-1.redshift.amazonaws.com`). Required except with `credentials: { type: redshift_data_api }`.
`port`	`integer`	`5439`	Port. Defaults to `DEFAULT_PORT` (5439).
`user`	`string`	`""`	Database user. Required for `password`; for a provisioned `iam` cluster it is the default `db_user`; unused by `redshift_data_api`.
`tls`	`boolean`	`true`	Whether to require TLS. Defaults to `true` (Redshift clusters require SSL by default). `true` maps to `sslmode=require` (encrypted, but the server certificate is not verified); `false` maps to `sslmode=prefer` (opportunistic TLS with plaintext fallback). Ignored when `tls_mode` is set.
`tls_mode`	`object`	`—`	Explicit TLS mode; overrides `tls`. `verify_ca` / `verify_full` check the server certificate against `ssl_root_cert` (or, without one, the Mozilla root store bundled into the binary — not the operating system's), the latter also its host name.
`ssl_root_cert`	`string`	`—`	PEM file of the CA that signed the server certificate (the Redshift CA bundle), for `tls_mode: verify_ca \| verify_full`.
`params`	`array`	`[]`	Positional bind parameters for the query, applied in `$1, $2, …` order before any context- or bookmark-derived values. Defaults to empty.
`max_connections`	`integer`	`10`	Maximum number of connections in the pool. Defaults to 10.
`batch_size`	`integer`	`1000`	Records per emitted `StreamPage`. Rows are drained from the `sqlx` cursor and yielded whenever the buffer reaches this size. Defaults to `DEFAULT_BATCH_SIZE`.
`replication`	`object`	`type: full`	Replication mode. Defaults to `RedshiftReplication::Full`.
`state_key`	`string`	`—`	Explicit state-store key for the incremental bookmark. When unset, a key is derived from the host / database and a query fingerprint.

Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh

brew install faucet-hq/faucet-stream/faucet-cli

Linux & macOS Homebrew

Read the guide → Star on GitHub