Connectors / Databases

MySQL

source

MySQL query source

Install

cargo install faucet-cli --features source-mysql
faucet init --source mysql
# Cargo.toml — use it as a library
cargo add faucet-source-mysql

Starter config

source:
  type: mysql
  config:
    connection_url: ""    # REQUIRED — MySQL connection URL (e.g.
    query: ""    # REQUIRED — SQL query to execute.
    # max_connections: 10    # Maximum number of connections in the pool.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # shard: null    # Optional primary-key range sharding for clustered (Mode B) execution.
    # read_timeout_secs: 3600    # Longest the source waits on the server for the next row (or for a whole result in `fetch_all`), in seconds, before the read fails.
    # json_big_numbers: null    # What to do with a number inside a JSON / JSONB column that a 64-bit float cannot hold exactly.
    # net_write_timeout_secs: 3600    # The session's `net_write_timeout`, in seconds: how long the server waits for the source to read more of a result before it aborts the query.
    # replication: { type: full }    # one of: full, incremental
      # --- Alternative variants — replace the block above with one of these ---
      # type: incremental
      # column: ""    # REQUIRED
      # initial_value: null    # REQUIRED
    # state_key: ""    # Explicit state-store key for the incremental bookmark.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental Yes
Dataset discovery Yes
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
connection_url
required
string — MySQL connection URL (e.g. mysql://user:pass@host/db).
query
required
string — SQL query to execute.
max_connections integer 10 Maximum number of connections in the pool. Defaults to 10.
batch_size integer 1000 Records per emitted StreamPage. Rows are drained from the sqlx cursor and yielded whenever the buffer reaches this size. Defaults to DEFAULT_BATCH_SIZE.
shard object — Optional primary-key range sharding for clustered (Mode B) execution.
read_timeout_secs integer 3600 Longest the source waits on the server for the next row (or for a whole result in fetch_all), in seconds, before the read fails. Defaults to 3600; 0 waits forever.
json_big_numbers object fail What to do with a number inside a JSON column that a 64-bit float cannot represent exactly (more than about 17 significant digits, or beyond the u64 / i64 range): fail (default) fails the read with an error naming the column, string emits the number as a JSON string holding its exact digits (with one warning per column).
net_write_timeout_secs integer 3600 The session's net_write_timeout, in seconds: how long the server waits for the source to read more of a result before it aborts the query. The pipeline stops reading while the sink writes a page, so the server default (60 s) kills large extracts behind a slow sink. Defaults to 3600; 0 keeps the server's setting.
replication object type: full Replication mode. Defaults to MysqlReplication::Full.
state_key string — Explicit state-store key for the incremental bookmark. When unset, a key is derived from the connection host and a fingerprint of the database URL and query.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli