faucetstream
Get started

Connectors / Databases

Elasticsearch

source

Elasticsearch search / scroll source

Install

cargo install faucet-cli --features source-elasticsearch
faucet init --source elasticsearch
# Cargo.toml — use it as a library
cargo add faucet-source-elasticsearch

Starter config

source:
  type: elasticsearch
  config:
    base_url: ""    # REQUIRED — Base URL of the Elasticsearch cluster (e.g.
    index: ""    # REQUIRED — Index name to search.
    # query: {"match_all":{}}    # Elasticsearch query DSL.
    # scroll_timeout: "5m"    # Scroll context keep-alive (e.g.
    auth:
      type: none    # one of: none, basic, bearer, api_key
      # --- Alternative variants — replace the block above with one of these ---
      # type: basic
      # config:
        # username: ""    # REQUIRED
        # password: ""    # REQUIRED
      #
      # type: bearer
      # config:
        # token: ""    # REQUIRED
      #
      # type: api_key
      # config:
        # key: ""    # REQUIRED
    # max_pages: 0    # Maximum number of scroll pages to fetch.
    # batch_size: 1000    # Records per emitted `StreamPage`, which is also the `size` parameter passed to the Elasticsearch scroll API (`POST /{index}/_search?scroll={timeout}&size={batch_size}`).
    # connect_timeout_secs: 10    # Seconds to wait for a TCP/TLS connection to the cluster.
    # request_timeout_secs: 300    # Seconds one search or scroll request may take end to end before it fails the run (the source does not retry it).
    # tls: { ... }    # Server verification and client-certificate settings for a connector that dials a TLS server itself (Redis, NATS, Elasticsearch).

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery Yes
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
base_url
required		 string — Base URL of the Elasticsearch cluster (e.g. "http://localhost:9200").
index
required		 string — Index name to search.
auth
required		 object — Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog.
query any match_all: {} Elasticsearch query DSL. Defaults to {"match_all": {}}.
scroll_timeout string 5m Scroll context keep-alive (e.g. "5m"), renewed by every scroll request — so it must outlast the slowest page *write* between two requests, or the context expires mid-run. Defaults to "5m".
max_pages integer — Maximum number of scroll pages to fetch. None means no limit.
batch_size integer 1000 Records per emitted StreamPage, which is also the size parameter passed to the Elasticsearch scroll API (POST /{index}/_search?scroll={timeout}&size={batch_size}). Each scroll response becomes exactly one StreamPage. Defaults to DEFAULT_BATCH_SIZE.
connect_timeout_secs integer 10 Seconds to wait for a TCP/TLS connection to the cluster. Default 10.
request_timeout_secs integer 300 Seconds one search or scroll request may take end to end before it fails the run (the source does not retry it). Default 300, so a half-open connection or a wedged node cannot hang a run forever.
tls object — TLS settings for an https:// cluster: a private CA, a client certificate for mutual TLS, or insecure_skip_verify.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub