One config, N tables: matrix pipelines with bounded concurrency
A source runs one query — one stream. To move a hundred tables you don't write a hundred configs: you write one and fan it out into a matrix, running in parallel under a deliberate ceiling.
One source is one stream
A faucet pipeline is one source and one sink. A Postgres source takes a single
query and streams it as bounded pages — sequential, predictable, bounded
memory. That's the unit. So "move these forty tables" isn't a single source with forty
queries; it's forty units.
The matrix: fan one template into many runs
You express those forty units as a matrix — one row per table, each deep-merging its own patch (its query/table) over the shared connection config. Source discovery can even generate the matrix for you: one row per discovered table. One template, N executions, no copy-paste.
The same mechanism powers parent/child fan-out: a parent stream's records parameterize a child query per record (with safe bind parameters — see this post).
They run in parallel — under a ceiling
Matrix rows don't run one after another. The executor runs them concurrently under a
semaphore whose width is execution.max_concurrent. The default is
min(available CPUs, 8) — and the cap of 8 is deliberate:
Each row is a full pipeline with its own connection pools and clients, and matrix rows usually target the same database. An unbounded fan-out across a 64-core box wouldn't go faster — it would blow through that database's connection and rate limits. The ceiling protects the system you're reading from.
Workloads that genuinely benefit from more parallelism set execution.max_concurrent
explicitly to opt out of the cap. And the number of hardware threads is read with
available_parallelism(), which honors container CPU limits — so inside a pod
pinned to 2 CPUs you get 2, not the host's 64.
Takeaway
One config becomes many parallel runs, each isolated and checkpointed on its own, with a concurrency default chosen to be fast and a good citizen of the database behind it. Scale the knob when you know the backend can take it.
More on the blog, or read the documentation.