One config, N tables: matrix pipelines with bounded concurrency

One source is one stream

A faucet pipeline is one source and one sink. A Postgres source takes a single query and streams it as bounded pages — sequential, predictable, bounded memory. That's the unit. So "move these forty tables" isn't a single source with forty queries; it's forty units.

The matrix: fan one template into many runs

You express those forty units as a matrix — one row per table, each deep-merging its own patch (its query/table) over the shared connection config. Source discovery can even generate the matrix for you: one row per discovered table. One template, N executions, no copy-paste.

The same mechanism powers parent/child fan-out: a parent stream's records parameterize a child query per record (with safe bind parameters — see this post).

They run in parallel — under a ceiling

Matrix rows don't run one after another. The executor runs them concurrently under a semaphore whose width is execution.max_concurrent. The default is min(available CPUs, 8) — and the cap of 8 is deliberate:

Workloads that genuinely benefit from more parallelism set execution.max_concurrent explicitly to opt out of the cap. And the number of hardware threads is read with available_parallelism(), which honors container CPU limits — so inside a pod pinned to 2 CPUs you get 2, not the host's 64.

Takeaway

One config becomes many parallel runs, each isolated and checkpointed on its own, with a concurrency default chosen to be fast and a good citizen of the database behind it. Scale the knob when you know the backend can take it.