Learn the architecture

Two ways to understand how faucet-stream works. Pick the one that fits you — the switch remembers your choice as you browse.

• 🎓 Beginner’s guide builds the whole system up as a story, one idea at a time.
• 🏛 Architect reference is the condensed, subsystem-by-subsystem view for people who already have the mental model.

🎓 Beginner's guide 🏛 Architect reference

Note

The buttons above switch this page in place on the published documentation site. If you’re reading the raw Markdown on GitHub (which doesn’t run the site’s scripts), both sections simply appear one after the other below.

The one-sentence idea

faucet-stream moves data from one place to another.

Picture a kitchen faucet: water comes from a pipe (the source), flows through the tap, and out into the sink. faucet-stream is the tap — you say where the data comes from and where it goes, and it moves the data reliably, without losing or scrambling it.

Source → faucet pipeline → Sink

Everything else — pages, bookmarks, retries, exactly-once — exists to keep that one sentence true even when things go wrong. We’ll add those ideas one at a time.

Chapter 1 — The two characters: Source and Sink

The whole system is built from two roles:

• A Source knows how to read records from somewhere (a database, an API, a file, a queue).
• A Sink knows how to write them somewhere else.

A connector is just a Source or Sink for one system (faucet-source-postgres, faucet-sink-bigquery, …). They all speak the same two-role language, which is why any source can feed any sink.

Records are just JSON. A database row, an API response, a file line — they all become plain JSON objects flowing through the pipe. At its simplest, a Source is one function (“give me your records”) and a Sink is one function (“here are records, write them”). That’s a working connector; everything else is optional.

Chapter 2 — Moving data once

Connect a Source to a Sink and you have a pipeline: read everything, write everything.

source.fetch → sink.write → done

For a one-time copy, this is all you need. Two real-world problems push us further: you don’t want to re-copy everything every run (Chapter 3), and your data might be too big for memory (Chapter 4).

Chapter 3 — Only the new stuff (incremental)

To avoid re-reading everything each run, the Source leaves itself a note — a bookmark — saying “I got up to here” (a timestamp, a log position, an offset). Next run it resumes from that note instead of the beginning.

Here’s the single most important rule in the whole project, and it’s just common sense:

Note

The bookmark is saved only after the data is safely written.

If we saved “got to row 1000” first and then crashed before writing those rows, they’d be lost forever. So the order is always write → make sure it’s really saved → then save the bookmark. Crash in between, and the worst case is redoing a little work (safe) — never skipping data (catastrophic). Keep this rule in your pocket; every advanced feature respects it.

Chapter 4 — Bigger than memory (streaming)

Reading a billion rows into memory won’t work. So instead of “all the data,” the Source produces a stream of pages — chunks of, say, 1,000 records at a time — and the pipeline handles one page at a time:

page 1 → write → page 2 → write → page 3 (+ bookmark) → write → flush → save bookmark

Only one page is ever in memory, so a thousand rows or a billion, memory stays flat. The bookmark rides along on the pages, and it’s still saved after the page is safely written — Chapter 3’s rule, now per-page.

Chapter 5 — The production toolbox (reach for these when you need them)

You now understand the spine: a source streams pages, the pipeline writes each page and checkpoints safely, so you can resume after a crash. Everything below is optional — a toolbox you pull from the day you hit the problem a tool solves. Find your situation, then follow the tool to its how-to. The family almost every real pipeline reaches for — shaping the data — comes first.

Shaping the data

The situation you’re in	The tool you reach for
The data isn’t in the shape the destination wants	Transforms
You need joins, aggregates, or real query power	SQL transform

Guarding the data

The situation you’re in	The tool you reach for
Some incoming rows are garbage (nulls, out-of-range)	Quality checks
Downstream must never get a surprise shape	Contracts
The data has PII you must never leak	Masking
The incoming shape drifts from the destination’s	Schema drift

Moving it reliably

The situation you’re in	The tool you reach for
A few bad rows keep killing the whole run	Dead-letter queue
The network or endpoint is flaky	Retries & resilience
You must never write a row twice, even after a crash	Exactly-once
You need a destination table kept mirrored (upserts, deletes)	Upsert / write modes

Getting data in and out at scale

The situation you’re in	The tool you reach for
One source is too big for a single worker	Sharding
Bootstrap a table, then follow its changes with no gap	Replication
Replay a bounded slice of history	Backfill
Auto-generate configs from a live catalog	Discovery
Read or write compressed files	Compression

Running & operating it

The situation you’re in	The tool you reach for
Run on a cron schedule	Scheduling
Run as a long-lived HTTP service	Serve
Spread runs across many machines	Cluster
Start runs on events (a file lands, a webhook, a queue fills)	Triggers
Turn one config into many pipelines (a DAG)	Matrix & composition
Pull credentials from a secrets manager	Secrets

Seeing what happened

The situation you’re in	The tool you reach for
Get metrics and traces	Observability
See where data came from and went	Lineage
Alert when data goes stale or volume looks wrong	SLA monitoring
Browse every dataset your pipelines have touched	Data Movement Catalog
Get paged (Slack / PagerDuty) when something breaks	Notifications

When several of the data-guarding tools are on, each page runs them in a fixed, safe order — mask first (so PII can’t leak), then validate (so bad data never lands), then write, then save the bookmark last:

page → mask → quality → contract → drift → write → flush → save bookmark

The golden rule never bends, no matter how many tools you add.

The one rule that ties it all together

Note

A bookmark is saved only after the sink has durably written and flushed the page. Write → flush → checkpoint. Always.

Every failure mode, retry, and exactly-once guarantee is a consequence of that one ordering.

Where to go next

• Run a real pipeline: Your first pipeline.
• The concepts, precisely: Core concepts.
• The full story with diagrams and code: the beginner guide on GitHub.
• Flip this page to 🏛 Architect reference for the condensed deep view.

Architecture at a glance

faucet-core is a lean library: it knows how to move one source to one sink and checkpoint safely. All orchestration (matrix DAGs, scheduling, the HTTP control plane, clustering) is CLI-layer code built on top. The full reference lives in the repository under docs/architecture/; this is the condensed view.

How a run is assembled

config → compose → interpolate → secrets → parse → expand → executor → Pipeline → run_stream

expand is where a config becomes runnable and where the load-time gates run (exactly-once, write-mode × sink, quarantine-requires-DLQ) — an impossible topology fails faucet validate before any record moves. Deep dive: execution model.

The pipeline loop

run_stream consumes one StreamPage { records, bookmark } at a time and, per page, runs the fixed-order passes then one of three write paths:

page → mask → quality → contract → drift → write path → flush → checkpoint

• Default (at-least-once): write_batch → flush → persist bookmark.
• Exactly-once (atomic watermark): write_batch_idempotent(scope, token) → flush → persist (bookmark, seq); a replayed token-stamped write is a no-op.
• DLQ: write_batch_partial routes per-row failures aside → flush → persist.

Deep dive: pipeline engine, stream pages.

The load-bearing invariant

Note

A page’s bookmark is persisted only after the sink has durably written and flushed that page. Write → flush → checkpoint, in all three paths.

The state store is therefore never ahead of the sink, so recovery can only ever replay attempted work — never skip it. Deep dive: design invariants, recovery.

Delivery guarantees

Guarantee	Requires	On the crash window
At-least-once (default)	nothing	replays the page — may duplicate
Effectively-once / atomic-watermark	idempotent sink + deterministic-replay source + durable state + no DLQ	skips or re-anchors — no duplication
Effectively-once / keyed-upsert	upsert-capable sink + write_mode: upsert|delete + key	re-upsert is a no-op — no duplication

Retry safety

A non-idempotent write_batch is retried only when the sink advertises idempotence — otherwise a lost response could silently duplicate every row. Deep dive: retries, resilience.

The subsystems

Area	Reference
Connector SDK (Source/Sink traits)	connector-sdk
State & bookmarks	state-management
Batching & adaptive control	batching
Schema / quality / contracts / masking	schema
Observability	observability
Security model	security
Performance & extensibility	performance · extensibility

Decision history lives in the ADRs; proposals in the RFCs.

Flip this page to 🎓 Beginner’s guide if you’d like the same story from zero.