Connector catalog

faucet-stream ships 38 sources and 30 sinks. Each is a Cargo feature (source-<name> / sink-<name>) and an independently published crate. Full API docs are on docs.rs.

Run faucet list to see what’s compiled into your binary, and faucet schema source <name> / faucet schema sink <name> for a connector’s exact config fields. Not sure which to pick? See Choosing a connector.

Legend: ✓ supported · ✗ not applicable. Tier: T1 = passes the faucet-conformance battery in CI; T2 = not yet wired into the battery.

Two “tier” signals, distinct on purpose. The Tier column below (T1/T2/T3) tracks whether a connector is wired into the reusable faucet-conformance test battery in CI. Separately, faucet conformance computes a maturity tier — 🟢 Stable / 🟡 Experimental / 🟠 Beta / ⚪ Draft — from each connector’s config schema and advertised capabilities; that tier shows in faucet list and cli/connectors/registry.json. See Connector conformance & tiers.

Sources

¹⁰ Discover = enumerates the datasets behind the connection for faucet discover (tables / collections / indices / prefixes with schemas + row estimates where the catalog provides them). ᵒ REST supports discovery with an odata: block (via the OData $metadata (EDMX) catalog — one dataset per entity set) or a generic discovery: recipe (a config-driven list → describe → emit pipeline — one dataset per listed object, with a templated source config, typed schema, and optional per-object sink table_id). ¹ Streams = yields records in bounded-memory batches rather than buffering the whole result. ² Resumable = persists a bookmark to a state store so re-runs continue where they left off (incremental replication / CDC / Kafka offsets). ³ Effectively-once = the source emits a complete resume position on every page and replaying from a bookmark continues the record stream at exactly that position (immutable-log sources: CDC WAL/binlog/change streams, Kafka partition offsets); required for the atomic-watermark mechanism behind delivery: exactly_once — see Effectively-once delivery. ⁴ gRPC streams natively in server-streaming mode; unary buffers the single response. ⁵ S3/GCS stream in JSONL and raw-text modes; JSON-array mode buffers one object. ⁶ Webhook is buffer-shaped by nature (it collects POSTs over a window). ⁸ MSSQL is resumable only in replication: incremental mode (it persists a tracking-column bookmark); in full mode it is not. ⁹ The Singer bridge is resumable via the tap’s STATE messages, but the granularity of resume (and whether re-emitted rows overlap) depends on the individual tap — pair it with a keyed/upsert sink for clean, effectively-once (idempotent at-least-once) behavior.

Support tiers (the Tier column above). A connector is Tier-1 ✅ when it invokes and passes the faucet-conformance battery in CI against the connector’s real backend — config-schema validity, bounded-memory streaming, and (where applicable) bookmark round-trip, idempotent replay, truthful capabilities, and errors-not-panics (see the Faucet Connector Protocol spec, docs/spec/faucet-connector-spec-v0.md). Each Tier-1 connector wires the battery from its own tests/conformance.rs; that battery is the tiering mechanism — there is no separate scheme.

ᵐ marks a connector whose battery runs in CI against a wiremock HTTP mock, not a live service instance — the rest, graphql, xml, elasticsearch, bigquery, snowflake, and databricks sources and the http sink. The mock faithfully drives the paging, schema, and error-handling behavior the checks assert, but it is not an end-to-end test against the real system (no credentialed cloud/service backend runs in CI). ᵉ marks a connector whose battery runs against an official emulator in Docker — a real implementation, closer to end-to-end than a wiremock but still not the managed service: the Cloud Spanner pair (Spanner emulator, gRPC), the Pub/Sub source and sink (Pub/Sub emulator, gRPC), and the Azure Blob sink (Azurite). Unmarked T1 ✅ connectors run against a real backend with no emulator caveat — a local filesystem (delta, parquet, csv), or a testcontainers-launched real server (postgres, mysql, mongodb, redis, clickhouse, kafka, …).

The connectors still marked Tier-2 are the ones whose full battery cannot run in CI (so they are not conformance-certified — Tier-2 means “not certified,” not “low quality”; they keep their own extensive wiremock/testcontainers tests): the BigQuery and Snowflake sinks and the Elasticsearch sink are cloud-only and tested against wiremock, which cannot validate real idempotent dedup; the GCS source’s bounded-memory check needs a real gRPC backend (the emulator is REST-only); the GCS sink cannot be durably counted against the emulator; the webhook source is buffer-shaped (no bounded-memory page check); and the Iceberg sink is append-only with a terminal flush that does not fit the effectively-once replay check on iceberg-rust 0.10.0. The Singer bridge ⚠️ passes the battery but is additionally experimental (v0, single-stream).

Streaming: native vs. buffered

The Streams¹ column above is not all-or-nothing — every source participates in the bounded-memory streaming loop, but there are two ways it gets there:

• Native streaming (override). The source reads from its underlying primitive incrementally — a database cursor, a WAL/binlog/change stream, an object read line by line, a scroll cursor, a Kafka partition — and emits each StreamPage as it goes. Memory stays at O(batch_size) no matter how large the result set. These sources override Source::stream_pages.
• Buffered fallback (default). The source implements only the one required method, fetch_with_context; the default stream_pages calls it, buffers the whole result, then chunks the buffer into pages. Correct and still streamed to the sink, but peak memory is the full result set because the fetch buffered it first.

A connector author gets the buffered path for free and opts into native streaming only where the primitive supports it — see ADR 0001 and the stream-pages architecture note.

Sources that override stream_pages for native streaming: rest, graphql, xml, postgres, postgres-cdc, mysql, mysql-cdc, mssql, mssql-cdc, sqlite, mongodb, mongodb-cdc, s3/gcs/azure-blob (JSONL & raw-text modes), parquet, csv, elasticsearch (scroll), kafka, kinesis, spanner, websocket, redis, and grpc (server-streaming mode).

Sources that intentionally keep the buffered default: grpc unary mode (a single response — no paging primitive) and webhook (buffer-shaped by nature — it collects POSTs over a window). S3/GCS/Azure fall back to buffered for the JSON-array format only (one array object must be parsed whole).

Sinks

Every sink exposes a batch_size knob for write-side re-chunking. For the file/append sinks (jsonl, csv, stdout) it’s a no-op — they write per record.

Object rollover (max_records_per_file / max_bytes_per_file, #618). The object-store sinks — s3, gcs, azure-blob, sftp — accumulate across write_batch calls and roll to a new object when either cap is reached. Before this each upstream page became its own object, so a small batch_size produced a swarm of tiny objects: the small-files problem that dominates read time on S3/Athena/Spark. With no cap set the whole run lands in one object, closed at flush. The byte cap is what bounds buffered memory (rows are a poor proxy for size), and s3/azure-blob additionally stream large objects through multipart so peak memory is O(part size), not O(object size).

Commit accumulation (commit_rows / commit_bytes, #617). The warehouse sinks — snowflake, clickhouse, redshift — accumulate records across write_batch calls and commit once per threshold, plus once at flush. The commit unit used to be the page unit and batch_size could only split a page, never merge two, so a small source page meant one expensive warehouse operation per page — and on ClickHouse, one MergeTree part per page, which fails outright once they accumulate. Only the append path accumulates: delivery: exactly_once and the DLQ path commit per page, because a watermark must land with its own page and a DLQ must name which rows of this page failed.

Auto-create (create_table, #580). Every table-based sink — bigquery, postgres, mysql, sqlite, mssql, duckdb, snowflake, redshift, clickhouse, spanner, delta, iceberg — takes create_table: bool, default true: a first-ever sync cannot assume the destination exists, so a missing table is created from the first written page’s inferred columns. Set false to require a pre-existing target and fail fast with one uniform error naming both ways out.

Every inferred column is created nullable. A column that happened to be present in page 1 is not required forever, and a NOT NULL inferred from one page turns page 2 into a hard failure the first time a record omits the field; narrowing later is the schema: drift policy’s job, which can see more than one page. Three dialect-specific notes:

• clickhouse creates MergeTree ORDER BY tuple() and redshift creates with no DISTKEY/SORTKEY — faucet has no basis to pick a sort or distribution key, and a wrong one is baked into the table. Define the table yourself and set create_table: false when the physical layout matters.
• snowflake creates STRING columns, because its insert path projects every value with ::string and a typed column would reject its own writer’s cast.
• spanner needs a primary key on every table, so it auto-creates only when key: is set; without one it errors naming that requirement rather than inventing a key column that can never be changed.

delta and iceberg already created their tables and now spell the knob create_table like everyone else (their historical create_if_not_missing / create_if_missing stay accepted as aliases). The schemaless destinations — mongodb, elasticsearch — deliberately have no knob: their servers create a collection/index on first write and cannot be told not to, so the field would be inert in one direction, which is exactly the silently-ignored config this project treats as a defect.

⁶ Parquet and Iceberg both handle compression internally at the Parquet column level, so the file-level compression feature doesn’t apply to either. ⁷ Effectively-once = commits data and a watermark token atomically; required for delivery: exactly_once. The BigQuery sink does this via a multi-statement MERGE transaction (distinct from its default bulk-load append path); the Kafka sink uses a transactional producer that writes each page’s records plus a commit-token record into a compacted side-topic in one Kafka transaction; the Snowflake sink runs one multi-statement BEGIN;INSERT;MERGE;COMMIT request; the Redis sink wraps the page plus a _faucet_commit_token:<scope> key in one MULTI/EXEC; the MongoDB sink commits the page plus a watermark document in one multi-document transaction (replica set required); the Cloud Spanner sink buffers the page’s mutations plus a faucet_commit_token row in one read-write transaction. Sinks configured with write_mode: upsert + key also reach effectively-once via keyed dedup, with any source. See Effectively-once delivery. ⁸ Upsert = supports write_mode: upsert / delete (insert-or-update and delete by key) in addition to plain append. The SQL sinks require column-mapping mode (auto_map, or auto_columns for mssql) and a UNIQUE/PRIMARY KEY on key; the schemaless sinks (MongoDB, Elasticsearch) map key to a match filter / _id. Iceberg upsert is not yet supported (a follow-up, blocked on iceberg-rust). write_mode: overwrite (full-refresh: atomically replace the whole destination each run) is additionally supported by PostgreSQL, SQLite, MySQL, MSSQL, MongoDB, BigQuery, and Elasticsearch (via an atomic alias swap — the configured index must be an alias) — not Spanner. See Upsert / mirror tables.

Every sink in this column also supports scoped cleanup (complete_for.on_missing: delete on the source), which deletes destination rows inside the declared scope that a run did not write — the only way an incremental sync can remove records deleted at the source. The two sets are identical today, so there is no separate column; see Removing records deleted at the source.

Arrow columnar (Parquet) fast path

An opt-in, additive Arrow columnar path (RFC 0002 / #375, behind a crate-local arrow feature) lets a run move records end-to-end as Arrow RecordBatches with no serde_json::Value materialization. It engages automatically when both ends of the pipeline are Arrow-native and every configured transform has an Arrow kernel; otherwise the pipeline transparently falls back to the row path.

Governance no longer disqualifies it (#636). masking:, quality:, contract: and schema: now run inside the columnar loop, via the same pass the row path uses — so a parquet → mask + quality → parquet run stays columnar instead of dropping to Value the moment a policy is attached. Quarantine works there too: quarantined rows are written to the dlq: sink under the same per-page and total budgets, and a budget abort still writes the overshoot before stopping, so no quarantined row is dropped. The one configuration that still falls back is on_batch_error: dlq_all, which routes a failed write row-by-row — write_batch_columnar reports no per-row outcomes to route. The governance pass materializes Value for the page it inspects; the source→sink transfer stays columnar.

Arrow-native connectors:

• Parquet source/sink and Delta Lake source/sink — Arrow-native by nature.
• AWS S3 and Google Cloud Storage source — with file_format: parquet.
• AWS S3 and Google Cloud Storage sink — with format: parquet (each object is a self-contained ZSTD-compressed Parquet file).
• Databricks SQL source — with arrow_native: true (fetches EXTERNAL_LINKS + ARROW_STREAM; requires replication: full).
• REST source — with an async_job: (Salesforce Bulk-style) CSV extract, no custom decode: chain, and a header locator (#635). Every column is Utf8: types are never inferred, so a batch’s columns match the Value path’s keys exactly.
• BigQuery source — with read_api: true + read_table (reads the table via the Storage Read API gRPC service as Arrow; full extract only).
• BigQuery sink — a PARQUET load job, for append and overwrite. A bulk_load block stages the Parquet on a GCS bucket first; without one the Parquet is uploaded with the job itself (bucket-free, #635), so a bucket is now only worth configuring for very large batches. Under overwrite the first batch truncates and the rest append, so a mid-run failure leaves the prior table intact.
• Snowflake sink — with a bulk_load block (Parquet uploaded to an external stage then COPY INTO … FILE_FORMAT=(TYPE=PARQUET); append only). The Snowflake source has no Arrow path (its v2 SQL API is jsonv2-only).

So chains like s3(parquet) → parquet, gcs(parquet) → delta, databricks(arrow) → parquet, bigquery(read-api) → parquet, rest(async_job csv) → bigquery, or parquet → snowflake(bulk-load) run Arrow end-to-end. See each connector’s README for the exact config field and feature flag.

Data-integrity notes

A few connectors enforce defaults that prevent silent data loss or corruption. Inspect the exact fields with faucet schema source <name> / faucet schema sink <name>.

• CSV source — strict by default. A row whose field count differs from the header raises an error naming the offending line. Set flexible: true to tolerate ragged rows (the pre-1.x behaviour). (Breaking default change.)
• CSV sink — the column set is frozen from the first batch (the header cannot be rewritten in place). A field that first appears in a later page is dropped; on_unknown_field: warn (default) emits a one-shot warning naming the dropped field(s), while on_unknown_field: error aborts with a typed error.
• Parquet sink — the Arrow schema is re-inferred per output file on rollover, so a file written after the source widens picks up the new schema. A Parquet file’s schema is immutable once opened, so a field appearing only later within a single file is dropped with a per-file one-shot warning.
• MongoDB CDC source — max_staged_records (default unbounded) caps the in-memory change-event buffer (including under batch_size: 0) and aborts with a typed error rather than risking OOM, mirroring postgres-cdc / mysql-cdc.

Schema evolution

The pipeline-level schema: block detects when an incoming page’s top-level shape diverges from the sink’s destination schema and applies one policy (warn / ignore / fail / quarantine / evolve). Which sinks can actually act on it varies:

on_drift: evolve against a detect-only or schemaless sink is rejected at config-load. See Schema drift for the per-sink nuances (e.g. SQLite widening is a no-op; Elasticsearch can only add fields).

Authentication at a glance

Inspect any connector’s exact auth shape with faucet schema source <name> / faucet schema sink <name>.

Batching

Default batch_size is 1000; max is 1,000,000. batch_size: 0 means “no batching” — the source emits the whole result set in one page and the sink writes it in one request (good for small lookup tables or load-job-style sinks). See Performance tuning.

¹¹ Tier = conformance status. T1 ✅ means the connector adds a tests/conformance.rs that invokes the reusable faucet-conformance battery against the real connector and passes it in CI (valid config schema, bounded-memory streaming, honest capabilities, and the further checks as they land) — that battery is the single source of truth for the tier. T2 means the connector is not yet wired into the battery; most still have their own integration tests, so T2 does not mean low quality. See the Faucet Connector Protocol (FCP v0) for the full contract.