Observability

Every source, sink, transform, and state-store operation is automatically wrapped to emit tracing spans and metrics counters/histograms. Connector authors write no observability code — they only override connector_name() for a friendly label.

Enabling the Prometheus endpoint

The CLI’s observability feature (on by default in the full build) installs a Prometheus exporter. Turn it on in the pipeline config; the process logs Prometheus /metrics listening on … at startup:

observability:
  prometheus:
    listen: "127.0.0.1:9464"

The endpoint has no authentication. Bind it to loopback and scrape through a local agent, or bind 0.0.0.0 only inside a pod whose network policy admits just Prometheus. A port already in use is a startup error. faucet serve serves its own /metrics on the API port, also unauthenticated.

A counter appears only after its first increment (overlaps, DLQ records), so write alerts that tolerate a missing series.

Common labels

pipeline, row (matrix row id; empty for non-matrix runs), and connector (from connector_name()). run_id is a span attribute only — it’s high cardinality and never a Prometheus label.

Key metrics

• Source: faucet_source_records_total, faucet_source_errors_total{kind}, faucet_source_page_duration_seconds, faucet_source_in_flight.
• Incremental key coverage (#747): faucet_source_replication_key_missing_total{pipeline,row,connector} — records an incremental source received without its replication_key (kept, dropped or failed per on_missing_key). A non-zero rate usually means a misspelled or wrongly nested key.
• Source lag (#733): faucet_source_lag_bytes, faucet_source_lag_events, faucet_source_lag_seconds{pipeline,row,connector} — how far a CDC or streaming source is behind its head (unread WAL / binlog, unconsumed messages, age of the oldest unread change), polled at page boundaries and at the end of a run. Sources without a head (query sources) emit nothing. faucet_source_lag_stale{pipeline,row,connector} is 1 while polling is paused after a slow lag query (the lag gauges then hold their last value). See Source lag.
• Sink: faucet_sink_records_total, faucet_sink_writes_total, faucet_sink_errors_total, faucet_sink_write_duration_seconds, faucet_sink_flush_duration_seconds, faucet_sink_in_flight.
• Batch outcomes (#737): faucet_batch_outcomes_total{pipeline,row,sink,outcome} — one increment per sink write, outcome ∈ committed | dlq_partial | dlq_all | failed; their sum is the writes attempted. See batch atomicity.
• Row accounting (#879): faucet_rows_unaccounted_total{pipeline,row,connector} — rows a sink was offered but neither wrote, merged by key, declined by design nor reported failed per row. Any increase means a run failed with RowsUnaccounted before its bookmark advanced; faucet run prints each row’s account (offered / accepted / coalesced / skipped / failed to DLQ / quarantined / dropped / filtered by transforms) and --output json carries it as rows.
• Upstream round trips (#638): faucet_source_roundtrips_total{op} and faucet_sink_roundtrips_total{op} — how many calls a connector actually made to its backend, with companion *_roundtrip_duration_seconds{op} histograms. This is the number that maps onto an API quota, S3 GET cost, or database load; faucet_source_pages_total only proxies the data fetches for paged HTTP sources and misses job submits, poll loops, and every non-HTTP connector. op is a closed, connector-defined set — the REST source emits submit / poll / fetch / page / discover. Retries count: a retried call is a real round trip. A connector that has not been instrumented emits nothing.
• Transform: faucet_transform_records_in_total, faucet_transform_records_out_total (use the out/in ratio for filter drop rate or explode fan-out), faucet_transform_errors_total{kind}, faucet_transform_duration_seconds.
• State: faucet_state_{get,put,delete}_total (get carries outcome=hit|miss), faucet_state_errors_total{op,kind}, plus duration histograms; faucet_state_key_migrations_total{outcome=moved|raced} counts bookmarks moved from a key an earlier release rendered (see State keys).
• Pipeline: faucet_pipeline_runs_total{status=ok|err,kind}, faucet_pipeline_run_duration_seconds, faucet_pipeline_in_flight, faucet_pipeline_seconds_since_last_bookmark, faucet_pipeline_last_bookmark_unix_seconds, and faucet_pipeline_invocation_duration_seconds{pipeline,row,source,sink,status} — the wall-clock of one matrix-row invocation, measured after it gets its concurrency slot (queue wait excluded). The two bookmark gauges are set only when a bookmark holding a timestamp is written: faucet_pipeline_last_bookmark_unix_seconds is that timestamp and faucet_pipeline_seconds_since_last_bookmark is how old it was at the write. The second gauge stops changing when writes stop, so measure staleness as time() - faucet_pipeline_last_bookmark_unix_seconds. kind on a failed run is the error kind.
• Local outputs (catalog feature): the retention GC for local sink output files — faucet_local_outputs_recorded_total{kind}, faucet_local_outputs_sweeps_total{scope}, faucet_local_outputs_deleted_total{scope}, faucet_local_outputs_bytes_deleted_total{scope}, and faucet_local_outputs_skipped_total{scope,reason}. Deleted/bytes are emitted even at zero — a sweep that found nothing is the healthy steady state, and its absence is how you notice the sweeper stopped. A rising skipped{reason="delete_failed"} (or "other_host", files recorded under another host identity) means the footprint is not being bounded and is worth alerting on; reason="pre_existing" is benign (files faucet did not create are never deleted). Distinct from faucet_cleanup_*, which counts destination rows removed by scoped cleanup.
• Build: faucet_build_info{version} is set to 1 — group_left it onto other metrics to annotate dashboards with the running version. version is the binary’s own version (what faucet --version prints); a library application sets its own with faucet_core::set_build_version, else the label is faucet-core’s version.

Finding the problem from metrics

Compare series by row; each question has a home:

• Which row is slow, and which side? Rank rows by faucet_pipeline_invocation_duration_seconds{status="ok"}, then compare that row’s faucet_source_page_duration_seconds with faucet_sink_write_duration_seconds / faucet_sink_flush_duration_seconds: whichever dominates is the bottleneck (tuning).
• Is it waiting on a quota? faucet_source_throttle_wait_seconds — time slept on rate limits is not a speed problem (source-side throttling).
• Is it behind? The faucet_source_lag_* gauges and time() - faucet_pipeline_last_bookmark_unix_seconds (source lag).
• Why did it fail? kind on faucet_pipeline_runs_total{status="err"}, faucet_{source,sink,state}_errors_total, and the faucet_resilience_* retry / circuit-breaker series (metrics).
• Where did rows go? faucet_source_records_total vs faucet_sink_records_total, the transform out/in ratio, faucet_source_replication_key_missing_total and faucet_batch_outcomes_total.
• Is the scheduler keeping up? The faucet_schedule_* series (health metrics).

A one-shot faucet run exits when it finishes, so a Prometheus scrape can miss it entirely; use faucet run --output json and the logs for one-shot runs, and metrics for faucet schedule and faucet serve.

Dashboards for these metrics

The six Grafana dashboards under observability/grafana/ chart every metric on this page; Dashboards & alerts lists what each one shows and the alert rules that go with them. Row accounting, verify results, policy violations and the other governance signals are on faucet-governance. Estimated cost, bytes moved and backend round trips are on faucet-cost. Log shipping, lineage and notification delivery are on faucet-reliability. A CI check fails when the code emits a metric that no dashboard shows and that observability/metrics-not-charted.txt does not list with a reason. So a new metric needs a panel, or a line in that file saying why it has none.

Reliability properties

• Drop-guard timers sample durations even when a task is cancelled.
• Panic isolation — a panicking connector surfaces as a Panic error kind rather than crashing the process, on the row path and on the columnar and native fast paths alike: a panic while writing an Arrow batch, loading native bytes, or opening or polling a fast-path stream fails that run only.
• Idempotent install — installing the recorder/subscriber twice warns rather than panics.

Cardinality rules

Never use high-cardinality values (record ids, URLs, query strings) as metric labels. parent_record_key in a DAG is a span attribute only. Connector authors must return a non-empty &'static str from connector_name().

Structured (JSON) logs

--log-format json (or FAUCET_LOG_FORMAT=json) renders every log record as one JSON object per line on stderr, so a k8s / ECS / Nomad log pipeline can ingest it without grok or regex:

$ faucet run --log-format json pipeline.yaml
{"timestamp":"2026-09-19T10:02:11.481Z","level":"INFO","target":"faucet_cli::executor","pipeline":"orders","row":"contact","records_written":4821,"message":"row completed"}

The span fields faucet already records — pipeline, row, run_id, connector, error kind — arrive as fields rather than being rendered into the message, so they are filterable at the collector.

Two things follow from “every line is one object”:

• Under json the end-of-run human status block, per-row timing table, and peak-RSS line are not printed; the same numbers leave as structured events (pipeline completed, row completed, process peak rss).
• faucet mcp keeps logs on stderr under either format, because stdout carries the JSON-RPC stream and two JSON streams on one pipe would corrupt it.

Secret redaction is unaffected: it operates on the serialized bytes, so a resolved ${vault:…} value appearing in a field is still scrubbed.

text remains the default.

Tracing

Spans carry run_id, pipeline, row, and per-operation timing. Point a tracing subscriber at your logging/trace backend; control verbosity with --log-level or FAUCET_LOG. The level is taken from the first that is set: --log-level, FAUCET_LOG, RUST_LOG, then the config’s observability.tracing.level. Any of them accepts a filter directive (faucet_core=debug,info) as well as a plain level.

Note

Full design: docs/superpowers/specs/2026-05-23-observability-otel-prometheus-design.md.

OTLP / OpenTelemetry export

The otel feature pushes traces and metrics to any OTLP-compatible collector (Jaeger, Grafana Tempo, Honeycomb, Datadog, the OpenTelemetry Collector, etc.) alongside — not instead of — the Prometheus endpoint. Build the CLI with cargo install faucet-cli --features otel; the feature is included in the full aggregate. Enable it in your pipeline config with an otel: sub-block under the existing observability: key:

observability:
  prometheus:
    listen: "0.0.0.0:9090"
  otel:
    endpoint: "https://api.honeycomb.io"
    protocol: grpc                        # grpc (default) | http
    headers:
      x-honeycomb-team: "${env:HONEYCOMB_KEY}"
    sample_ratio: 0.1                     # head-based; 1.0 = keep all traces
    export: [traces, metrics]             # which signals to push
    service_name: faucet                  # OTel resource service.name
    timeout_secs: 10
    metric_interval_secs: 60

The observability.prometheus: and observability.otel: blocks coexist independently — both can be active in the same run and metrics are fanned out to both exporters.

Protocol notes:

• grpc uses tonic (the default). The faucet CLI always runs inside a tokio runtime, so gRPC works without any extra setup. headers are sent as gRPC metadata (names are lower-cased); an invalid header name or value disables export with a warning.
• http uses HTTP/Protobuf. When endpoint does not already end in a per-signal path (/v1/traces, /v1/metrics), faucet appends it automatically — point endpoint at the base URL of the collector (e.g. http://localhost:4318) and the right path is added per signal.

Traces are exported whenever export lists traces, with or without an observability.tracing level, by run, schedule, replicate/mirror and the other commands that load a config. faucet serve takes no config, so it reads the same block from its own file: faucet serve --otel-config otel.yaml (or FAUCET_SERVE_OTEL_CONFIG), a YAML/JSON document shaped like observability.otel (${env:…} / ${file:…} resolved, header values kept out of logs). Traces cover every run the server executes; with export: [metrics] the metrics /metrics serves are also pushed over OTLP.

otel.yaml

endpoint: http://otel-collector:4318
protocol: http
export: [traces, metrics]
service_name: faucet-serve
headers: { authorization: "Bearer ${env:OTLP_TOKEN}" }

Reliability: export is best-effort. An unreachable or slow collector never fails or delays a pipeline run. Export failures increment faucet_otel_export_failures_total{signal} so you can alert on a broken pipeline to your observability backend.

See examples/infra/otel-collector.yaml for a minimal local collector config you can run with otelcol --config examples/infra/otel-collector.yaml.

OTLP metrics

Metric	Labels	Description
faucet_otel_export_failures_total	signal (traces/metrics/logs/export)	OTLP export attempts that failed. Failures are non-fatal; the pipeline continues.

Shipping logs

Add logs to export and faucet ships every run’s log lines to the OTLP collector — and never loses one because the collector, the network or the faucet process went down. Every line is written locally first; a background shipper delivers it and advances a per-run delivery watermark only after the collector acknowledges the batch. Local copies are kept until delivered (bounded), then for a short window.

observability:
  otel:
    endpoint: http://collector:4317
    export: [traces, metrics, logs]     # `logs` turns shipping on
  logs:                                 # optional — the defaults are shown
    spool_dir: ~/.local/state/faucet/logs   # faucet run / schedule only
    retention_secs: 86400               # keep delivered lines 24 h
    buffer_max_age_secs: 604800         # drop undelivered lines after 7 days
    buffer_max_bytes: 1073741824        # …or once the buffer passes 1 GiB
    max_lines_per_run: 100000
    flush_timeout_secs: 10              # `faucet run` waits this long at exit
    notify_after_secs: 300              # `log_export_failed` after 5 min failing
    link_template: "https://grafana.example/explore?...{run_id}..."

Where the buffer lives

Runtime	Buffer	Shipper
faucet serve	The run-history log store (--history sqlite:… / postgres://…). With --history memory it is an in-memory, non-durable buffer — the server warns at startup.	A background task. In --cluster, each run’s delivery is leased (faucet_serve_log_ship), so one instance ships it; a peer resumes it once the lease lapses. When several instances log one run (a sharded run), each records how far it has persisted (faucet_serve_log_writers) and delivery is acknowledged only up to the slowest live one, so a line one instance stores late is still shipped; an instance that stops reporting for two minutes no longer holds the run back.
faucet run	A spool directory: <run_id>.jsonl (append-only, fsync’d every 1024 lines and at run end), <run_id>.cursor.json (the watermark, written temp + rename), <run_id>.meta.json, and an advisory <run_id>.lock.	Ships in the background while the run goes, then for at most flush_timeout_secs at exit. Anything left is shipped by the next faucet process using the spool, or by faucet logs ship.
faucet schedule	The same spool; every tick is a run.	A background task for the process lifetime, draining between ticks.

faucet serve reads its settings from flags (each also an env var): --log-retention-secs (default 86400 with shipping on, 604800 without), --log-buffer-max-age-secs, --log-buffer-max-bytes, --log-link-template, --log-export-notify-after-secs, with export: [logs] in --otel-config.

Delivery guarantees

• At least once per batch. A batch whose acknowledgement is lost (or whose watermark write fails) is sent again; every record carries faucet.seq and the batch scope carries faucet.batch.first_seq / faucet.batch.last_seq, so a backend can drop the duplicate. An acknowledged line is never re-sent.
• The pipeline is never slowed. Capture hands each line to a bounded queue; a full queue drops the line (counted as queue_full) instead of blocking, and a local-write failure is reported on stderr and counted — never a run failure.
• Redacted at capture. Every resolved secret is scrubbed before the line is written locally, so neither the buffer nor the export ever holds one. Lines longer than 16 KiB are cut and marked.
• Two processes, one spool. A run’s .lock makes sure only one process ships it; a run whose writer died is picked up by the next shipper.
• Bounds drop the oldest, visibly. Past buffer_max_age_secs / buffer_max_bytes (or the per-run line cap) the oldest undelivered lines are dropped, counted in faucet_logs_dropped_lines_total{reason}, and the run is reported partially_dropped.
• A malformed line never blocks a run. When the collector refuses a batch as malformed (HTTP 400 / 413 / 422, gRPC InvalidArgument), the batch is split until the offending line is found; that line is dropped (reason rejected) and the rest delivered. A line is dropped only once the collector has accepted another line of the batch (when the first two lines are refused, the batch’s last line is tried on its own first), so a collector that refuses every payload drops nothing; one run’s refused lines do not stop the other runs from shipping. Every other failure — the collector down, throttling, a credential problem — is retried and nothing is dropped.

Was a run’s log shipped?

GET /v1/runs/{id} carries log_export:

{ "status": "failed", "delivered_seq": 7338097199674630147, "pending_lines": 4,
  "dropped_lines": 0, "last_error": "OTLP HTTP export returned 503 Service Unavailable",
  "last_attempt_at": "2026-10-09T06:34:13Z",
  "link": "https://grafana.example/explore?q=01a11f5e-…" }

status is one of not_configured, pending, exported, failed, partially_dropped. last_error carries the status only — never the collector’s URL or response body, which go to the server log. The console’s run page shows it as a Log export panel, with View logs ↗ built from the link template; once a run’s local copy has aged out, the log endpoint returns that link (and the SSE stream opens with a link event) instead of nothing. faucet run prints logs: exported or logs: 1,204 lines pending — run "faucet logs ship" and adds log_export to --output json.

faucet logs ship [CONFIG] [--spool DIR] [--endpoint URL] [--protocol grpc|http] drains a spool once (a cron job or sidecar); its exit code is the number of runs it left undelivered, not counting runs a live process is still writing.

A log_export_failed notification goes through the run’s notifications: when its export has been failing for notify_after_secs, and when lines were dropped.

Log-shipping metrics

Metric	Labels	Description
faucet_logs_buffered_lines	—	Lines buffered and not yet acknowledged.
faucet_logs_buffered_bytes	—	Bytes of those lines.
faucet_logs_oldest_undelivered_seconds	—	Age of the oldest undelivered line.
faucet_logs_shipped_lines_total	—	Lines the collector acknowledged.
faucet_logs_dropped_lines_total	reason (max_age/max_bytes/max_lines/queue_full/rejected)	Lines dropped before delivery.
faucet_logs_export_errors_total	—	Failed export attempts (also faucet_otel_export_failures_total{signal="logs"}).
faucet_logs_local_write_failures_total	—	Lines that could not be written to the local buffer.

observability/prometheus/alerts.yml has FaucetLogsUndelivered (oldest undelivered line > 15 min) and FaucetLogsDropped. Receiving-end recipes — Alloy → Loki, the OpenTelemetry Collector → S3 / GCS / Azure, a Docker Compose example — are in deploy/otel/. --log-format json stays the option for shippers that read stdout.