Connector catalog

faucet-stream ships 37 sources and 29 sinks. Each is a Cargo feature (source-<name> / sink-<name>) and an independently published crate. Full API docs are on docs.rs.

Run faucet list to see what’s compiled into your binary, and faucet schema source <name> / faucet schema sink <name> for a connector’s exact config fields. Not sure which to pick? See Choosing a connector.

Legend: ✓ supported · ✗ not applicable. Tier: T1 = passes the faucet-conformance battery in CI; T2 = not yet wired into the battery.

Two “tier” signals, distinct on purpose. The Tier column below (T1/T2/T3) tracks whether a connector is wired into the reusable faucet-conformance test battery in CI. Separately, faucet conformance computes a maturity tier — 🟢 Stable / 🟡 Experimental / 🟠 Beta / ⚪ Draft — from each connector’s config schema and advertised capabilities; that tier shows in faucet list and cli/connectors/registry.json. See Connector conformance & tiers.

Sources

¹⁰ Discover = enumerates the datasets behind the connection for faucet discover (tables / collections / indices / prefixes with schemas + row estimates where the catalog provides them). ¹ Streams = yields records in bounded-memory batches rather than buffering the whole result. ² Resumable = persists a bookmark to a state store so re-runs continue where they left off (incremental replication / CDC / Kafka offsets). ³ Effectively-once = the source emits a complete resume position on every page and replaying from a bookmark continues the record stream at exactly that position (immutable-log sources: CDC WAL/binlog/change streams, Kafka partition offsets); required for the atomic-watermark mechanism behind delivery: exactly_once — see Effectively-once delivery. ⁴ gRPC streams natively in server-streaming mode; unary buffers the single response. ⁵ S3/GCS stream in JSONL and raw-text modes; JSON-array mode buffers one object. ⁶ Webhook is buffer-shaped by nature (it collects POSTs over a window). ⁸ MSSQL is resumable only in replication: incremental mode (it persists a tracking-column bookmark); in full mode it is not. ⁹ The Singer bridge is resumable via the tap’s STATE messages, but the granularity of resume (and whether re-emitted rows overlap) depends on the individual tap — pair it with a keyed/upsert sink for clean, effectively-once (idempotent at-least-once) behavior.

Support tiers (the Tier column above). A connector is Tier-1 ✅ when it invokes and passes the faucet-conformance battery in CI against the connector’s real backend — config-schema validity, bounded-memory streaming, and (where applicable) bookmark round-trip, idempotent replay, truthful capabilities, and errors-not-panics (see the Faucet Connector Protocol spec, docs/spec/faucet-connector-spec-v0.md). Each Tier-1 connector wires the battery from its own tests/conformance.rs; that battery is the tiering mechanism — there is no separate scheme.

ᵐ marks a connector whose battery runs in CI against a wiremock HTTP mock, not a live service instance — the rest, graphql, xml, elasticsearch, bigquery, snowflake, and databricks sources and the http sink. The mock faithfully drives the paging, schema, and error-handling behavior the checks assert, but it is not an end-to-end test against the real system (no credentialed cloud/service backend runs in CI). ᵉ marks a connector whose battery runs against an official emulator in Docker — a real implementation, closer to end-to-end than a wiremock but still not the managed service: the Cloud Spanner pair (Spanner emulator, gRPC), the Pub/Sub source and sink (Pub/Sub emulator, gRPC), and the Azure Blob sink (Azurite). Unmarked T1 ✅ connectors run against a real backend with no emulator caveat — a local filesystem (delta, parquet, csv), or a testcontainers-launched real server (postgres, mysql, mongodb, redis, clickhouse, kafka, …).

The connectors still marked Tier-2 are the ones whose full battery cannot run in CI (so they are not conformance-certified — Tier-2 means “not certified,” not “low quality”; they keep their own extensive wiremock/testcontainers tests): the BigQuery and Snowflake sinks and the Elasticsearch sink are cloud-only and tested against wiremock, which cannot validate real idempotent dedup; the GCS source’s bounded-memory check needs a real gRPC backend (the emulator is REST-only); the GCS sink cannot be durably counted against the emulator; the webhook source is buffer-shaped (no bounded-memory page check); and the Iceberg sink is append-only with a terminal flush that does not fit the effectively-once replay check on iceberg-rust 0.9.1. The Singer bridge ⚠️ passes the battery but is additionally experimental (v0, single-stream).

Streaming: native vs. buffered

The Streams¹ column above is not all-or-nothing — every source participates in the bounded-memory streaming loop, but there are two ways it gets there:

• Native streaming (override). The source reads from its underlying primitive incrementally — a database cursor, a WAL/binlog/change stream, an object read line by line, a scroll cursor, a Kafka partition — and emits each StreamPage as it goes. Memory stays at O(batch_size) no matter how large the result set. These sources override Source::stream_pages.
• Buffered fallback (default). The source implements only the one required method, fetch_with_context; the default stream_pages calls it, buffers the whole result, then chunks the buffer into pages. Correct and still streamed to the sink, but peak memory is the full result set because the fetch buffered it first.

A connector author gets the buffered path for free and opts into native streaming only where the primitive supports it — see ADR 0001 and the stream-pages architecture note.

Sources that override stream_pages for native streaming: rest, graphql, xml, postgres, postgres-cdc, mysql, mysql-cdc, mssql, mssql-cdc, sqlite, mongodb, mongodb-cdc, s3/gcs/azure-blob (JSONL & raw-text modes), parquet, csv, elasticsearch (scroll), kafka, kinesis, spanner, websocket, redis, and grpc (server-streaming mode).

Sources that intentionally keep the buffered default: grpc unary mode (a single response — no paging primitive) and webhook (buffer-shaped by nature — it collects POSTs over a window). S3/GCS/Azure fall back to buffered for the JSON-array format only (one array object must be parsed whole).

Sinks

Every sink exposes a batch_size knob for write-side re-chunking. For the file/append sinks (jsonl, csv, stdout) it’s a no-op — they write per record.

⁶ Parquet and Iceberg both handle compression internally at the Parquet column level, so the file-level compression feature doesn’t apply to either. ⁷ Effectively-once = commits data and a watermark token atomically; required for delivery: exactly_once. The BigQuery sink does this via a multi-statement MERGE transaction (distinct from its default streaming insertAll path); the Kafka sink uses a transactional producer that writes each page’s records plus a commit-token record into a compacted side-topic in one Kafka transaction; the Snowflake sink runs one multi-statement BEGIN;INSERT;MERGE;COMMIT request; the Redis sink wraps the page plus a _faucet_commit_token:<scope> key in one MULTI/EXEC; the MongoDB sink commits the page plus a watermark document in one multi-document transaction (replica set required); the Cloud Spanner sink buffers the page’s mutations plus a faucet_commit_token row in one read-write transaction. Sinks configured with write_mode: upsert + key also reach effectively-once via keyed dedup, with any source. See Effectively-once delivery. ⁸ Upsert = supports write_mode: upsert / delete (insert-or-update and delete by key) in addition to plain append. The SQL sinks require column-mapping mode (auto_map, or auto_columns for mssql) and a UNIQUE/PRIMARY KEY on key; the schemaless sinks (MongoDB, Elasticsearch) map key to a match filter / _id. Iceberg upsert is not yet supported (a follow-up, blocked on iceberg-rust). See Upsert / mirror tables.

Arrow columnar (Parquet) fast path

An opt-in, additive Arrow columnar path (RFC 0002 / #375, behind a crate-local arrow feature) lets a run move records end-to-end as Arrow RecordBatches with no serde_json::Value materialization. It engages automatically when both ends of the pipeline are Arrow-native and no Value-shaped transform is configured; otherwise the pipeline transparently falls back to the row path.

Arrow-native connectors:

• Parquet source/sink and Delta Lake source/sink — Arrow-native by nature.
• AWS S3 and Google Cloud Storage source — with file_format: parquet.
• AWS S3 and Google Cloud Storage sink — with format: parquet (each object is a self-contained ZSTD-compressed Parquet file).
• Databricks SQL source — with arrow_native: true (fetches EXTERNAL_LINKS + ARROW_STREAM; requires replication: full).
• BigQuery source — with read_api: true + read_table (reads the table via the Storage Read API gRPC service as Arrow; full extract only).
• BigQuery sink — with a bulk_load block (Parquet on a GCS staging bucket then a PARQUET load job; append only).
• Snowflake sink — with a bulk_load block (Parquet uploaded to an external stage then COPY INTO … FILE_FORMAT=(TYPE=PARQUET); append only). The Snowflake source has no Arrow path (its v2 SQL API is jsonv2-only).

So chains like s3(parquet) → parquet, gcs(parquet) → delta, databricks(arrow) → parquet, bigquery(read-api) → parquet, or parquet → snowflake(bulk-load) run Arrow end-to-end. See each connector’s README for the exact config field and feature flag.

Data-integrity notes

A few connectors enforce defaults that prevent silent data loss or corruption. Inspect the exact fields with faucet schema source <name> / faucet schema sink <name>.

• CSV source — strict by default. A row whose field count differs from the header raises an error naming the offending line. Set flexible: true to tolerate ragged rows (the pre-1.x behaviour). (Breaking default change.)
• CSV sink — the column set is frozen from the first batch (the header cannot be rewritten in place). A field that first appears in a later page is dropped; on_unknown_field: warn (default) emits a one-shot warning naming the dropped field(s), while on_unknown_field: error aborts with a typed error.
• Parquet sink — the Arrow schema is re-inferred per output file on rollover, so a file written after the source widens picks up the new schema. A Parquet file’s schema is immutable once opened, so a field appearing only later within a single file is dropped with a per-file one-shot warning.
• MongoDB CDC source — max_staged_records (default unbounded) caps the in-memory change-event buffer (including under batch_size: 0) and aborts with a typed error rather than risking OOM, mirroring postgres-cdc / mysql-cdc.

Schema evolution

The pipeline-level schema: block detects when an incoming page’s top-level shape diverges from the sink’s destination schema and applies one policy (warn / ignore / fail / quarantine / evolve). Which sinks can actually act on it varies:

on_drift: evolve against a detect-only or schemaless sink is rejected at config-load. See Schema drift for the per-sink nuances (e.g. SQLite widening is a no-op; Elasticsearch can only add fields).

Authentication at a glance

Inspect any connector’s exact auth shape with faucet schema source <name> / faucet schema sink <name>.

Batching

Default batch_size is 1000; max is 1,000,000. batch_size: 0 means “no batching” — the source emits the whole result set in one page and the sink writes it in one request (good for small lookup tables or load-job-style sinks). See Performance tuning.

¹¹ Tier = conformance status. T1 ✅ means the connector adds a tests/conformance.rs that invokes the reusable faucet-conformance battery against the real connector and passes it in CI (valid config schema, bounded-memory streaming, honest capabilities, and the further checks as they land) — that battery is the single source of truth for the tier. T2 means the connector is not yet wired into the battery; most still have their own integration tests, so T2 does not mean low quality. See the Faucet Connector Protocol (FCP v0) for the full contract.