faucetstream
Get started

Connectors / Databases

Redis

source

Redis (streams, lists, keys) source

Install

cargo install faucet-cli --features source-redis
faucet init --source redis
# Cargo.toml — use it as a library
cargo add faucet-source-redis

Starter config

source:
  type: redis
  config:
    url: ""    # REQUIRED — Redis connection URL (e.g.
    source_type:
      type: List    # one of: List, Stream, Keys
      key: ""    # REQUIRED — The list key.
      # --- Alternative variants — replace the block above with one of these ---
      # type: Stream
      # key: ""    # REQUIRED
      # group: ""
      # consumer: ""
      # count: 0
      #
      # type: Keys
      # pattern: ""    # REQUIRED
    # tls: { ... }    # Server verification and client-certificate settings for a connector that dials a TLS server itself (Redis, NATS, Elasticsearch).
    # max_records: 0    # Optional maximum number of records to return.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # parse_json: null    # Which Redis values the source parses as JSON.
    # binary: null    # How the source renders a value that is not valid UTF-8.
    # max_tracked_keys: 1000000    # `Keys` mode: how many matched key names the scan remembers to emit a key `SCAN` returns twice only once (16 bytes plus set overhead each, about 48 MB at the default 1,000,000).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
url
required		 string — Redis connection URL (e.g. "redis://127.0.0.1:6379").
source_type
required		 object — The type of Redis data structure to read from.
tls object — TLS settings for a rediss:// URL: a private CA, a client certificate for mutual TLS, or insecure_skip_verify.
max_records integer — Optional maximum number of records to return.
batch_size integer 1000 Records per emitted StreamPage. Each mode maps the knob onto its native paging primitive:
parse_json object all Which values are parsed as JSON. all (default) parses any value that is valid JSON, so "42" becomes 42 and "true" becomes true; containers parses only objects and arrays, so "1.10", "true" and long numeric ids stay the strings Redis holds; none keeps every value a string.
binary object base64 What to do with a value that is not valid UTF-8 (msgpack, gzip, …): base64 (default) emits it base64-encoded, lossy replaces invalid sequences with U+FFFD, error fails the run naming the key.
max_tracked_keys integer 1000000 Keys mode: how many matched key names the scan remembers to emit a key SCAN returns twice only once (16 bytes plus set overhead each, about 48 MB at the default 1,000,000). Past the limit the scan stops remembering new keys, so a later duplicate can be emitted again (a warning says so) while memory stays bounded. 0 turns de-duplication off.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub