ClickHouse

sink

ClickHouse sink (HTTP INSERT … FORMAT JSONEachRow; optional async inserts)

Install

cargo install faucet-cli --features sink-clickhouse
faucet init --sink clickhouse

# Cargo.toml — use it as a library
cargo add faucet-sink-clickhouse

Starter config

sink:
  type: clickhouse
  config:
    # url: ""    # Full base URL of the ClickHouse HTTP interface, e.g.
    # host: ""    # Hostname of the ClickHouse server.
    # http_port: 0    # HTTP-interface port used with `host`.
    # tls: false    # Use `https://` instead of `http://` when building the base URL from `host`.
    # database: "default"    # Target database.
    # user: ""    # ClickHouse user.
    # password: ""    # ClickHouse password.
    table: ""    # REQUIRED — Target table.
    # batch_size: 1000    # Records per `INSERT` request.
    # create_table: true    # Create the target table if it does not exist, inferring the columns from the first written page (#580).
    # commit_rows: 0    # Commit-group size for the cross-page accumulator (#617).
    # commit_bytes: 0    # Estimated-bytes counterpart of `commit_rows` (#617).
    # async_insert: false    # Enable ClickHouse [asynchronous inserts](https://clickhouse.com/docs/en/optimize/asynchronous-inserts) (`async_insert=1`): the server buffers rows and flushes them in the background, which greatly improves throughput for many small inserts.
    # wait_for_async_insert: true    # When `async_insert` is enabled, whether the server waits for the buffered rows to be flushed before acknowledging the request (`wait_for_async_insert=1`).
    # staging: null    # Staged bulk load (#528): stage each page to S3/GCS and have the ClickHouse server pull it with `s3()` / `gcs()` instead of an `INSERT … FORMAT JSONEachRow` body.

Capabilities

Delivery	At-least-once
Write modes	append
Exactly-once writes	No
Schema evolution	No
Scoped cleanup	No
Staged bulk load	Yes
Run rollback	No
Compression	No

Config fields

Field	Type	Default	Description
`table` required	`string`	`—`	Target table. May be schema-qualified (`db.table`); each identifier segment is quoted before use.
`url`	`string`	`—`	Full base URL of the ClickHouse HTTP interface, e.g. `"http://localhost:8123"`. Mutually exclusive with `host`. A trailing slash is trimmed.
`host`	`string`	`—`	Hostname of the ClickHouse server. Mutually exclusive with `url`; combined with `http_port` and `tls` to build the base URL.
`http_port`	`integer`	`—`	HTTP-interface port used with `host`. Defaults to `DEFAULT_HTTP_PORT` (`8123`).
`tls`	`boolean`	`false`	Use `https://` instead of `http://` when building the base URL from `host`. Ignored when `url` is set. Defaults to `false`.
`database`	`string`	`default`	Target database. Defaults to `DEFAULT_DATABASE` (`"default"`).
`user`	`string`	`—`	ClickHouse user. When set, sent as the `X-ClickHouse-User` header.
`password`	`string`	`—`	ClickHouse password. When set, sent as the `X-ClickHouse-Key` header.
`batch_size`	`integer`	`1000`	Records per `INSERT` request. When the upstream `StreamPage` carries more records than `batch_size`, the sink splits it into `batch_size`-row chunks and issues one `INSERT … FORMAT JSONEachRow` per chunk. `0` forwards the whole page as a single request. Defaults to `DEFAULT_BATCH_SIZE`.
`create_table`	`boolean`	`true`	Create the target table if it does not exist, inferring the columns from the first written page (#580). Enabled by default: a first-ever sync cannot assume the destination already exists. The table is created with the `MergeTree` engine and `ORDER BY tuple()` — the neutral choice when faucet has no key to sort by; define the table yourself and set `create_table: false` when the sort key matters, which it does for any table you intend to query at scale.
`commit_rows`	`integer`	`—`	Commit-group size for the cross-page accumulator (#617).
`commit_bytes`	`integer`	`—`	Estimated-bytes counterpart of `commit_rows` (#617). Rows are a poor proxy for how much work a warehouse commit is; this bounds the buffered size. `0` means no byte limit; see `commit_rows` for the default when both are unset.
`async_insert`	`boolean`	`false`	Enable ClickHouse [asynchronous inserts](https://clickhouse.com/docs/en/optimize/asynchronous-inserts) (`async_insert=1`): the server buffers rows and flushes them in the background, which greatly improves throughput for many small inserts. Defaults to `false`.
`wait_for_async_insert`	`boolean`	`true`	When `async_insert` is enabled, whether the server waits for the buffered rows to be flushed before acknowledging the request (`wait_for_async_insert=1`). Keeping this `true` (the default) preserves faucet's at-least-once durability contract — the bookmark only advances after ClickHouse has durably accepted the batch. Ignored when `async_insert` is `false`.
`staging`	`object`	`—`	Staged bulk load (#528): stage each page to S3/GCS and have the ClickHouse server pull it with `s3()` / `gcs()` instead of an `INSERT … FORMAT JSONEachRow` body. Absent ⇒ the ordinary insert path. Requires the crate's `staging` feature at build time to execute.

Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh

brew install faucet-hq/faucet-stream/faucet-cli

Linux & macOS Homebrew

Read the guide → Star on GitHub