Rust data-movement platform

Move data at Rust speed.
Govern it in flight.

A config-driven pipeline engine with 78 source & sink connectors, change-data-capture, effectively-once delivery, and governance built into the movement path — wired by a single faucet binary, or embedded in your own service. No Python runtime, nothing external to stand up.

brew install faucet-hq/faucet-stream/faucet-cli
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
cargo install faucet-cli
docker run --rm -v "$PWD:/w" ghcr.io/faucet-hq/faucet-stream:full run /w/pipeline.yaml
helm install faucet oci://ghcr.io/faucet-hq/charts/faucet-stream

or embed the library — cargo add faucet-stream

deploy with Docker or Helm →

pipeline.yaml
Postgres source · CDC source
sql transform · DuckDB transform
BigQuery sink · MERGE sink
0 rows/s
20 MiB RAM bounded & streaming
78 connectors 43 sources · 35 sinks
701k rows / second 1M-row CSV → JSONL
~80× faster than Meltano exact row parity
1 binary to ship no daemon, no runtime
Why faucet-stream

One engine for the whole movement path.

Most stacks bolt movement, orchestration, and governance together from separate tools. faucet-stream does all three in a single, fast Rust runtime.

Built for throughput

Native streaming with bounded memory, connection pooling, multi-row inserts, bulk APIs, and parallel I/O. Throughput is a first-class design goal for every connector — not an afterthought.

A runtime, not just connectors

Incremental + resumable replication, change-data-capture, effectively-once delivery, dead-letter queues, automatic retries, adaptive batch sizing, secrets interpolation, cron scheduling, and built-in Prometheus metrics + tracing spans — with zero per-connector code.

Governance in the movement path

Data-quality checks, versioned contracts, PII masking applied before any sink sees a row, schema-drift policy, column-level lineage (OpenLineage), and freshness/volume SLAs — native and zero-config.

CLI, library, or HTTP API

Run faucet run pipeline.yaml, call Pipeline::new(&source, &sink).run().await? from Rust, or drive it over HTTP with faucet serve. Same engine three ways — a CLI you drop on any box, a library you compile in, or a control-plane API.

Bring your Singer taps and targets

The singer source runs any existing Singer / Meltano tap unchanged, and the singer sink feeds any existing Singer target, so you can move a Meltano pipeline one side at a time and switch to native connectors where throughput matters.

Compile only what you need

Almost every capability is a Cargo feature — connectors, transforms, state backends, governance, even the HTTP server — so your binary contains exactly what you switch on, nothing else. Ship a lean REST-to-JSONL build, or turn everything on with --features full.

Benchmarks

The same million rows — 1.4 seconds, not two minutes.

A reproducible 1M-row CSV → JSONL move: faucet-stream finishes in ~1.4s; Meltano takes nearly two minutes to deliver the same rows. That's the best case (it maximally exposes Python's per-row overhead) — sink-bound moves like Postgres → Postgres narrow the gap, and every caveat is in the open.

See the full benchmarks →
Throughput higher is better
faucet-stream
701,000 rows/s
Meltano (Singer)
8,700 rows/s
Peak memory lower is better
faucet-stream
20 MiB
Meltano (Singer)
819 MiB

~80× faster · ~40× less memory · exact row parity · single binary, no daemon

Declare it once

A pipeline is a file, not a codebase.

Describe source, in-flight transforms, write mode, and durable state in YAML — then faucet run. Incremental replication, upsert, secrets, and bookmarks are config, not glue code. Prefer Rust? The same engine drops into your service through typed Source / Sink traits — or drive it over HTTP with faucet serve.

  • Resumable incremental & CDC replication
  • Upsert / delete write modes with effectively-once delivery
  • ${env:…} & secrets-manager interpolation
  • Page-level embedded-DuckDB sql transform
Full config grammar →

Running it as a service? The control plane ships a full OpenAPI 3 spec — import it into Postman, Insomnia, or Bruno (Import → Link → paste the spec URL).

# orders → warehouse: CDC, masked, upserted
version: 1
name: orders_to_warehouse

pipeline:
  source:
    type: postgres-cdc
    config:
      connection_url: ${env:PG_URL}
      slot_name: orders_slot
      publication_name: orders_pub
  transforms:
    - type: cdc_unwrap
      config: {}
  masking:
    key: ${env:MASKING_KEY}
    rules:
      - name: emails
        match: { fields: [email] }
        action: { type: hash }
  sink:
    type: bigquery
    config:
      project_id: my-project
      dataset_id: analytics
      table_id: orders
      write_mode: upsert
      key: [id]
      delete_marker: { field: __op, values: [d] }
      auth:
        type: service_account_key
        config: { json: "${env:GCP_KEY_JSON}" }
  state:
    type: file
    config: { path: ./.faucet-state }
// same engine, embedded in your service
use faucet_stream::Pipeline;
use faucet_stream::source::postgres_cdc::PostgresCdcSource;
use faucet_stream::sink::bigquery::BigQuerySink;

let source = PostgresCdcSource::new(cdc_cfg).await?;
let sink   = BigQuerySink::new(bq_cfg).await?;

Pipeline::new(&source, &sink)
    .with_state_store(state)   // durable bookmarks
    .run()
    .await?;

// → streams pages, checkpoints after each
//   confirmed write. Bounded memory throughout.
# run it over the HTTP control plane (faucet serve)
curl -X POST http://localhost:8080/v1/runs \
  -H 'authorization: Bearer $FAUCET_TOKEN' \
  -H 'content-type: application/json' \
  --data-binary @pipeline.json

# → 202 Accepted
#   { "id": "run_01H...", "status": "queued" }

# then poll / stream:
curl http://localhost:8080/v1/runs/run_01H...
curl -N .../v1/runs/run_01H.../logs   # SSE
Connectors

43 sources. 35 sinks. One config grammar.

Every connector shares the same auth, pagination, batching, and state model — so what you learn on one applies to all of them. Arrow-native fast paths run columnar end-to-end where both ends support it.

Browse the catalog →

Databases

  • Amazon DynamoDB
  • DuckDB
  • Elasticsearch
  • Microsoft SQL Server
  • MongoDB
  • MySQL
  • Oracle Database
  • PostgreSQL
  • Redis
  • SQLite

Change data capture

  • MongoDB CDC
  • MySQL CDC
  • Oracle CDC
  • PostgreSQL CDC
  • SQL Server CDC

Warehouses

  • Amazon Redshift
  • ClickHouse
  • Cloud Spanner
  • Databricks
  • Google BigQuery
  • Snowflake

Streaming & queues

  • Apache Kafka
  • AWS Kinesis
  • AWS SQS
  • Google Cloud Pub/Sub
  • NATS
  • RabbitMQ

Files & object stores

  • Amazon S3
  • Apache Iceberg
  • Apache Parquet
  • Azure Blob Storage
  • CSV
  • Delta Lake
  • Google Cloud Storage
  • JSON Lines
  • Local files
  • SFTP

APIs & web

  • GraphQL
  • gRPC
  • HTTP
  • REST
  • Webhook
  • WebSocket
  • XML / SOAP

Bridges & output

  • Singer bridge
  • Stdout
Governance in flight

The guardrails live in the pipe — not bolted on downstream.

Everything that usually needs a second platform runs inside the movement path, with zero per-connector code. A masked field never reaches the sink in the clear.

Quality checks

Assert row- and batch-level rules; route failures to a DLQ.

Data contracts

Versioned schemas enforced on the wire, not downstream.

PII masking

Redact or hash sensitive fields before any sink sees a row.

Schema-drift policy

Detect drift and decide: pass, quarantine, or stop.

Column lineage

OpenLineage events + a movement catalog, per column.

SLA monitors

Freshness and volume checks with built-in metrics.

Data-flow policies

Say which sinks a label may reach, like PII only to EU; refused before any data moves.

Change impact

See which datasets, contracts and owners a schema change breaks, before it ships.

Column profiling

Learned per-column baselines flag real drift, with no thresholds to write.

Observability

Metrics and traces, built in.

Every source, sink, transform, and state op emits Prometheus metrics and OpenTelemetry spans automatically — labelled by pipeline, row, and connector. Scrape /metrics, ship spans over OTLP, point Grafana at it.

Prometheus metricsOTLP tracing spans120+ metricszero per-connector code
How it compares

An all-in-one platform — not a pile of services.

A single fast Rust binary and an embeddable library: connectors, orchestration, governance, and delivery in one place.

vs Meltano

Same config-driven ergonomics, 10–80× faster, no Python runtime.

vs Airbyte

No platform to stand up — one binary or a library, governance native not paywalled.

vs Fivetran

Self-hosted and open-source; your data never leaves your infra.

Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli