Effectively-once, not exactly-once
Most tools promise 'exactly-once.' faucet-stream's type system won't let you write it — because across a crash it isn't achievable, and pretending otherwise is how data quietly goes wrong.
The honest default: at-least-once
faucet-stream moves data in bounded pages, and after each page it does three things in order: write to the sink → flush → persist the resume bookmark. The bookmark is only saved once the sink has confirmed the write. That ordering guarantees no committed change is ever lost — but it leaves one window: if the process dies after the sink commits and before the bookmark is persisted, the next run re-reads that page. So the default guarantee is at-least-once: no loss, but a page can be delivered twice.
Why not "exactly-once"?
Because between two independent systems — a source, a sink, and a crash that can happen at
any instant — there is no way to make "the row is written" and "the position is recorded"
a single atomic fact. Every "exactly-once" claim is really at-least-once plus a
deduplication trick somewhere. faucet-stream names that honestly: its delivery type is
AtLeastOnce or EffectivelyOnce(mechanism) — there is no
ExactlyOnce variant to select, on purpose.
Effectively-once, when the sink can help
Duplicates are removed by pairing the source with a sink that can absorb a replay. Ask for it and faucet selects one of two mechanisms:
- Atomic watermark — an idempotent sink plus a deterministic-replay source and a durable state store. The sink's committed token embeds the resume bookmark; on restart the pipeline reads that token and re-anchors the source, skipping pages already committed.
- Keyed upsert — the sink deduplicates by primary key, so a replayed row overwrites instead of duplicating.
If neither is available, faucet refuses to pretend: requesting exactly-once without an idempotent or keyed sink is a typed error naming the limiting side — not a silent downgrade to "hope."
Why this is the right kind of honest
The worst failures in a data platform are the silent ones. A tool that says "exactly-once" and quietly drops or double-writes a row on a bad day is worse than one that tells you the exact guarantee and lets you close the gap with a keyed sink. faucet-stream picks the second kind: never lose a committed change, and be precise about duplicates.
More on the blog, or read the documentation.