Notes from the team
Shorter than the papers — feature spotlights and the reasoning behind them. Every claim is grounded in the real implementation.
Effectively-once, honestly: what your sink actually guarantees
Exactly-once is mostly a marketing word. faucet is precise: at-least-once by default, and effectively-once when the sink is idempotent — so re-runs and crash-resumes converge instead of duplicating.
Read the post →Crash-safe by bookmark: when it's safe to move a CDC checkpoint
A CDC checkpoint is a promise that data is delivered. Move it too early and a crash silently drops committed rows. faucet advances it from one place only — a durable bookmark, after the sink flushes.
Read the post →One config, N tables: how the matrix fans out in parallel
Give a source a list of tables and faucet runs them as concurrent pipeline instances — bounded by a default derived from your CPU count — with parent/child streams fanning out beneath each.
Read the post →Flat memory at a million rows
faucet streams page-by-page instead of buffering the result set, so a million-row extract holds roughly the same resident memory as a thousand-row one. Backpressure, not buffering.
Read the post →Adaptive batching: the pipeline tunes its own batch size
A fixed batch size is a guess. faucet runs an AIMD controller that grows batches while throughput improves and backs off on pressure — converging on the size the destination actually wants.
Read the post →The Value tax: when Arrow saves you (and when it doesn't)
Converting Arrow to JSON and back at every connector boundary is pure tax — up to 7–9× the useful work. When both ends speak Arrow, faucet negotiates a zero-cost columnar fast path.
Read the post →Append, upsert, or delete: write modes and what they cost
Appending streams flat out; upserting makes the destination reconcile every row. faucet gives you both, adds a COPY fast lane for append, and refuses incoherent combinations at config time.
Read the post →Bad rows shouldn't stop the stream
One malformed record should quarantine itself, not kill the load. faucet routes rejects to a dead-letter queue with the reason attached — carrying the masked payload, never the raw secret.
Read the post →Govern in the pipe, not after the load
Landing raw data and governing it later means the secret has already landed and the bad row is already queryable. faucet folds masking, quality, contracts, drift and lineage into the movement itself.
Read the post →Injection-safe by construction: parameterized parent/child fan-out
When a parent stream drives child queries, faucet fills `${parent.field}` tokens as positional bind parameters — never string-interpolated. Safe by construction, not by discipline.
Read the post →Secrets that never touch your config
Reference a secret with `${vault:…}` / `${aws-sm:…}` / `${gcp-sm:…}` / `${azure-kv:…}` and faucet resolves it from the manager at run time, using the platform credential chain. Your config holds a pointer, never the value.
Read the post →Observability with zero per-connector code
You shouldn’t have to instrument a pipeline to see it. faucet emits Prometheus metrics and OpenTelemetry traces at the engine level, so every connector — and every future one — is visible for free.
Read the post →Compile only what you need
faucet-stream is ~90 crates, but your binary isn’t. Connectors are opt-in Cargo features, so a REST→JSONL build carries no database drivers and no cloud SDKs — just the code your pipeline runs.
Read the post →No posts match those tags.
Your first pipeline runs in five minutes.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli