Built for throughput
Native streaming with bounded memory, connection pooling, multi-row inserts, bulk APIs, and parallel I/O. Throughput is a first-class design goal for every connector — not an afterthought.
A config-driven pipeline engine with 68 source & sink connectors,
change-data-capture, effectively-once delivery, and governance built into the
movement path — wired by a single faucet binary, or embedded in your
own service. No Python runtime, nothing external to stand up.
brew install faucet-hq/faucet-stream/faucet-cli curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh cargo install faucet-cli docker run --rm -v "$PWD:/w" ghcr.io/faucet-hq/faucet-stream:full run /w/pipeline.yaml helm install faucet oci://ghcr.io/faucet-hq/charts/faucet-stream
or embed the library — cargo add faucet-stream
Most stacks bolt movement, orchestration, and governance together from separate tools. faucet-stream does all three in a single, fast Rust runtime.
Native streaming with bounded memory, connection pooling, multi-row inserts, bulk APIs, and parallel I/O. Throughput is a first-class design goal for every connector — not an afterthought.
Incremental + resumable replication, change-data-capture, effectively-once delivery, dead-letter queues, automatic retries, adaptive batch sizing, secrets interpolation, cron scheduling, and built-in Prometheus metrics + tracing spans — with zero per-connector code.
Data-quality checks, versioned contracts, PII masking applied before any sink sees a row, schema-drift policy, column-level lineage (OpenLineage), and freshness/volume SLAs — native and zero-config.
Run faucet run pipeline.yaml, call Pipeline::new(&source, &sink).run().await? from Rust, or drive it over HTTP with faucet serve. Same engine three ways — a CLI you drop on any box, a library you compile in, or a control-plane API.
The singer source runs any existing Singer / Meltano tap unchanged, so you can adopt incrementally — start with the taps you already have and move to native connectors where throughput matters.
Almost every capability is a Cargo feature — connectors, transforms, state backends, governance, even the HTTP server — so your binary contains exactly what you switch on, nothing else. Ship a lean REST-to-JSONL build, or turn everything on with --features full.
A reproducible 1M-row CSV → JSONL move: faucet-stream finishes in ~1.4s; Meltano takes ~2¼ minutes for the identical, row-for-row output. That's the best case (it maximally exposes Python's per-row overhead) — sink-bound moves like Postgres → Postgres narrow the gap, and every caveat is in the open.
See the full benchmarks →~96× faster · ~62× less memory · exact row parity · single binary, no daemon
Describe source, in-flight transforms, write mode, and durable state in YAML —
then faucet run. Incremental replication, upsert, secrets, and
bookmarks are config, not glue code. Prefer Rust? The same engine drops into
your service through typed Source / Sink traits — or
drive it over HTTP with faucet serve.
${env:…} & secrets-manager interpolationsql transformRunning it as a service? The control plane ships a full OpenAPI 3 spec — import it into Postman, Insomnia, or Bruno (Import → Link → paste the spec URL).
# orders → warehouse, incremental + upsert
version: 1
name: orders_to_warehouse
pipeline:
source:
type: postgres
config:
connection_url: ${env:PG_URL}
query: SELECT * FROM orders
replication:
method: incremental
cursor_field: updated_at
transforms:
- sql: SELECT *, amount_cents/100.0 AS amount FROM batch
- redact: { fields: [email] }
sink:
type: bigquery
config:
dataset: analytics
table: orders
write_mode: upsert
primary_key: [id]
state:
type: file
path: .faucet/state // same engine, embedded in your service
use faucet_stream::prelude::*;
let source = PostgresSource::new(pg_cfg)?;
let sink = BigQuerySink::new(bq_cfg)?;
Pipeline::new(&source, &sink)
.with_state_store(state) // durable bookmarks
.run()
.await?;
// → streams pages, checkpoints after each
// confirmed write. Bounded memory throughout. # run it over the HTTP control plane (faucet serve)
curl -X POST http://localhost:8080/v1/runs \
-H 'authorization: Bearer $FAUCET_TOKEN' \
-H 'content-type: application/json' \
--data-binary @pipeline.json
# → 202 Accepted
# { "id": "run_01H...", "status": "queued" }
# then poll / stream:
curl http://localhost:8080/v1/runs/run_01H...
curl -N .../v1/runs/run_01H.../logs # SSE Every connector shares the same auth, pagination, batching, and state model — so what you learn on one applies to all of them. Arrow-native fast paths run columnar end-to-end where both ends support it.
Everything that usually needs a second platform runs inside the movement path, with zero per-connector code. A masked field never reaches the sink in the clear.
Assert row- and batch-level rules; route failures to a DLQ.
Versioned schemas enforced on the wire, not downstream.
Redact or hash sensitive fields before any sink sees a row.
Detect drift and decide: pass, quarantine, or stop.
OpenLineage events + a movement catalog, per column.
Freshness and volume checks with built-in metrics.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli