Built for throughput
Native streaming with bounded memory, connection pooling, multi-row inserts, bulk APIs, and parallel I/O. Throughput is a first-class design goal for every connector — not an afterthought.
A config-driven pipeline engine with 78 source & sink connectors,
change-data-capture, effectively-once delivery, and governance built into the
movement path — wired by a single faucet binary, or embedded in your
own service. No Python runtime, nothing external to stand up.
brew install faucet-hq/faucet-stream/faucet-cli curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh cargo install faucet-cli docker run --rm -v "$PWD:/w" ghcr.io/faucet-hq/faucet-stream:full run /w/pipeline.yaml helm install faucet oci://ghcr.io/faucet-hq/charts/faucet-stream
or embed the library — cargo add faucet-stream
Most stacks bolt movement, orchestration, and governance together from separate tools. faucet-stream does all three in a single, fast Rust runtime.
Native streaming with bounded memory, connection pooling, multi-row inserts, bulk APIs, and parallel I/O. Throughput is a first-class design goal for every connector — not an afterthought.
Incremental + resumable replication, change-data-capture, effectively-once delivery, dead-letter queues, automatic retries, adaptive batch sizing, secrets interpolation, cron scheduling, and built-in Prometheus metrics + tracing spans — with zero per-connector code.
Data-quality checks, versioned contracts, PII masking applied before any sink sees a row, schema-drift policy, column-level lineage (OpenLineage), and freshness/volume SLAs — native and zero-config.
Run faucet run pipeline.yaml, call Pipeline::new(&source, &sink).run().await? from Rust, or drive it over HTTP with faucet serve. Same engine three ways — a CLI you drop on any box, a library you compile in, or a control-plane API.
The singer source runs any existing Singer / Meltano tap unchanged, and the singer sink feeds any existing Singer target, so you can move a Meltano pipeline one side at a time and switch to native connectors where throughput matters.
Almost every capability is a Cargo feature — connectors, transforms, state backends, governance, even the HTTP server — so your binary contains exactly what you switch on, nothing else. Ship a lean REST-to-JSONL build, or turn everything on with --features full.
A reproducible 1M-row CSV → JSONL move: faucet-stream finishes in ~1.4s; Meltano takes nearly two minutes to deliver the same rows. That's the best case (it maximally exposes Python's per-row overhead) — sink-bound moves like Postgres → Postgres narrow the gap, and every caveat is in the open.
See the full benchmarks →~80× faster · ~40× less memory · exact row parity · single binary, no daemon
Describe source, in-flight transforms, write mode, and durable state in YAML —
then faucet run. Incremental replication, upsert, secrets, and
bookmarks are config, not glue code. Prefer Rust? The same engine drops into
your service through typed Source / Sink traits — or
drive it over HTTP with faucet serve.
${env:…} & secrets-manager interpolationsql transformRunning it as a service? The control plane ships a full OpenAPI 3 spec — import it into Postman, Insomnia, or Bruno (Import → Link → paste the spec URL).
# orders → warehouse: CDC, masked, upserted
version: 1
name: orders_to_warehouse
pipeline:
source:
type: postgres-cdc
config:
connection_url: ${env:PG_URL}
slot_name: orders_slot
publication_name: orders_pub
transforms:
- type: cdc_unwrap
config: {}
masking:
key: ${env:MASKING_KEY}
rules:
- name: emails
match: { fields: [email] }
action: { type: hash }
sink:
type: bigquery
config:
project_id: my-project
dataset_id: analytics
table_id: orders
write_mode: upsert
key: [id]
delete_marker: { field: __op, values: [d] }
auth:
type: service_account_key
config: { json: "${env:GCP_KEY_JSON}" }
state:
type: file
config: { path: ./.faucet-state } // same engine, embedded in your service
use faucet_stream::Pipeline;
use faucet_stream::source::postgres_cdc::PostgresCdcSource;
use faucet_stream::sink::bigquery::BigQuerySink;
let source = PostgresCdcSource::new(cdc_cfg).await?;
let sink = BigQuerySink::new(bq_cfg).await?;
Pipeline::new(&source, &sink)
.with_state_store(state) // durable bookmarks
.run()
.await?;
// → streams pages, checkpoints after each
// confirmed write. Bounded memory throughout. # run it over the HTTP control plane (faucet serve)
curl -X POST http://localhost:8080/v1/runs \
-H 'authorization: Bearer $FAUCET_TOKEN' \
-H 'content-type: application/json' \
--data-binary @pipeline.json
# → 202 Accepted
# { "id": "run_01H...", "status": "queued" }
# then poll / stream:
curl http://localhost:8080/v1/runs/run_01H...
curl -N .../v1/runs/run_01H.../logs # SSE Every connector shares the same auth, pagination, batching, and state model — so what you learn on one applies to all of them. Arrow-native fast paths run columnar end-to-end where both ends support it.
Everything that usually needs a second platform runs inside the movement path, with zero per-connector code. A masked field never reaches the sink in the clear.
Assert row- and batch-level rules; route failures to a DLQ.
Versioned schemas enforced on the wire, not downstream.
Redact or hash sensitive fields before any sink sees a row.
Detect drift and decide: pass, quarantine, or stop.
OpenLineage events + a movement catalog, per column.
Freshness and volume checks with built-in metrics.
Say which sinks a label may reach, like PII only to EU; refused before any data moves.
See which datasets, contracts and owners a schema change breaks, before it ships.
Learned per-column baselines flag real drift, with no thresholds to write.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli