Flat memory at a million rows
A data mover's memory should be a function of its batch size, not its dataset size. faucet-stream streams in bounded pages, so a 1M-row move and a 1B-row move use about the same RAM.
The failure mode: buffering the dataset
Plenty of pipelines quietly assume the data fits in memory: read it all, transform it, write it. That works in the demo and falls over on the real table — OOM at 3 a.m., or a box sized for the worst-case dataset instead of the workload. Memory that scales with rows is a liability.
The design: bounded pages, end to end
faucet-stream never materializes the dataset. The source yields a bounded page of
records; that page flows through transforms and governance and into the sink; then it's
released and the next page is read. Peak memory is the cost of one page in flight —
set by batch_size — not the total row count. A 1M-row move and a 1B-row move
have essentially the same memory profile; the second just runs longer.
What that looks like on the bench
On the reproducible Postgres → JSONL move of 1,000,000 rows, faucet held a peak RSS of about 14 MiB — while a Singer-based pipeline doing the same move used around 743 MiB, roughly 50× more, with exact row-count parity. The gap isn't a tuning trick; it's the difference between streaming pages and marshalling every row through an in-memory pipe.
Flat memory is also what makes the concurrency story safe: because each pipeline is cheap in RAM, you can run several in parallel (a matrix) without the memory adding up to trouble.
Takeaway
Size your box for the throughput you want, not for the biggest table you might ever move. With bounded-page streaming, the dataset can grow without the memory bill following it.
More on the blog, or read the documentation.