Observability with zero per-connector code
You shouldn't have to instrument a pipeline to see it. faucet-stream emits Prometheus metrics and OpenTelemetry traces for every source, sink, transform, and state op — and no connector writes a line of that code.
Instrumentation shouldn't be a connector's job
If every connector had to remember to emit metrics, you'd get uneven coverage: the popular ones instrumented, the long tail dark, and metric names that don't line up. The right place for observability is the engine — the one loop every pipeline runs through.
Emitted at the engine, for free
faucet-stream measures the pipeline itself: record counts and throughput, per-page and
per-batch timings, sink latency, retries and errors, DLQ writes, adaptive-batch adjustments,
and topology-node stats (join matches/misses, merge counts). They're standard
faucet_* Prometheus series — for example
faucet_pipeline_adaptive_batch_adjustments_total — so a new connector inherits
full observability the day it's written, with no per-connector instrumentation.
Because it's engine-level, coverage is uniform: the same metrics for a REST source and a Postgres CDC source, the same names across every sink. Nothing goes dark just because a connector author didn't add telemetry.
Metrics and traces, opt-in by feature
Prometheus metrics come with the observability feature; OpenTelemetry tracing
(spans across source → transform → sink → state, with pipeline and matrix-row labels) comes
with otel, exporting over OTLP to your collector. Build without them and you pay
nothing; build with them and your pipelines are visible in the tools you already run — no
code changes to the connectors.
Takeaway
Observability is a property of the runtime, not a chore per integration. Flip on a feature flag and every pipeline — and every future connector — shows up in Prometheus and your tracing backend automatically.
More on the blog, or read the documentation.