Adaptive batching: let the sink set the pace
The right write batch size isn't a constant — it depends on the sink, the network, and the moment. faucet-stream tunes it at run time with an AIMD controller that reads the sink's own latency and errors.
Why a fixed batch size is wrong
Batch too small and you pay per-request overhead and round-trips; batch too large and you hit timeouts, memory pressure, or the sink's payload limits. The sweet spot moves with the destination, the network latency, and the load on the target at that instant. A number you hard-code in config is right in exactly one situation and wrong the rest of the time.
An AIMD controller on the write path
faucet-stream runs an AIMD controller — additive-increase, multiplicative-decrease, the same shape that governs TCP congestion control — that adjusts the effective write batch size per pipeline from two observed signals: sink latency and error rate. When writes are fast and clean, it nudges the batch up a step to chase throughput. When latency climbs or the sink starts rejecting, it backs off sharply to recover. The pipeline converges on the batch size the sink can actually sustain right now.
It's self-correcting by design: additive increase probes gently for more throughput, multiplicative decrease bails out fast when the sink is unhappy — so a transient slowdown doesn't turn into a pile-up.
You can watch it work
The controller isn't a black box: it emits faucet_pipeline_adaptive_batch_adjustments_total
so you can see it reacting in your metrics, alongside the sink-latency signals it's tuning
from. And it composes with everything else — the adjusted batch is still one bounded page,
so memory stays flat and
checkpointing is unchanged.
Takeaway
Instead of asking you to guess a batch size — and re-guess it every time the destination or network changes — faucet-stream measures and adapts. The fast path finds itself.
More on the blog, or read the documentation.