Column profiling: learned baselines & drift detection

Quality checks catch what someone anticipated and wrote down (null_rate <= 5%), and the SLA monitor learns only row volume. The failures that actually reach dashboards are the unanticipated ones: an upstream change makes a column 40 % null, a currency field starts arriving in cents, an enum gains a value. The top-level profiling: block closes that gap with no thresholds to write: every run profiles the records it wrote, the profile is compared with a rolling baseline of earlier runs, and a statistically significant change on any column is reported per column and per metric.

version: 1
name: orders
pipeline:
  source: { type: postgres, config: { connection_url: "${env:PG_URL}", query: "SELECT * FROM orders" } }
  sink: { type: file, config: { path: ./orders.jsonl } }
  state: { type: file, config: { path: ./state } }
profiling:
  min_history: 5        # runs of baseline before detection starts (default 5)
  window: 20            # rolling baseline size (default 20)
  on_drift: warn        # warn | notify | fail

A runnable example lives at cli/examples/csv_to_jsonl_with_profiling.yaml.

What a profile holds

Per top-level column of the records the run wrote (after transforms and masking, without the _faucet_* metadata columns):

Nested objects and arrays count as one column whose value is their canonical JSON. Memory is bounded per column regardless of row count, so a 10 M-row run profiles in the same memory as a 10-row one.

A column whose estimated distinct count exceeds categorical_max_distinct (default 100) is high-cardinality: it keeps null / type / length / numeric statistics but publishes no values (an id or free-text column should never land in a catalog as a list of examples). Because the pass runs after masking, a masked column’s profile holds only its masked values.

Drift detection

Detection starts once the baseline holds min_history runs; earlier runs just learn. Every finding names a column and one of these metrics:

A column absent from the baseline (new this run) is learned, not flagged — schema drift owns added and removed columns.

on_drift

Whatever the policy, the run’s profile joins the baseline, so a repeated drift fades once the new shape is the norm (after min_history runs). For a planned change, re-baseline explicitly:

faucet profiling reset pipeline.yaml                    # forget every column's history
faucet profiling reset pipeline.yaml --column amount    # just one column
faucet profiling reset pipeline.yaml --row invoices     # one matrix row

Inspecting profiles

faucet profiling show pipeline.yaml            # latest profile + drift per root row
faucet profiling show pipeline.yaml --full     # every column's full statistics
faucet profiling show pipeline.yaml --json

row row-0 — 6 run(s) in the baseline (state orders::row-0)
  latest run 0199a3… at 2026-09-25T02:00:04Z: 12,480 row(s), 4 column(s)
  drift: 2 finding(s)
    ! amount.null_rate: null_rate 0.4000 vs baseline mean 0.0012 — |z| 41.30 exceeds 3 (…)
    ! region.new_value: value "latam" is 12.5% of the run; never among the top values in 6 baseline runs
  columns:
  amount                       null  40.0%  distinct     6 812  [number:7488]  min 10.5 max 99.5 mean 54.9 p50 55.5 p95 95.5
  region                       null   0.0%  distinct         4  [string:12480]  top "eu" 41.0%, "us" 33.2%, "latam" 12.5%, "apac" 13.3%

faucet doctor reports the baseline depth (warming up: 2 of 5 run(s)) and whether the last run raised drift. With a catalog: block (or under faucet serve), every run’s profile is also recorded on the sink dataset: faucet catalog show <id> prints a profile: section, the dataset detail endpoint carries profile.latest + profile.history, and the web console’s dataset page renders per-column null-rate sparklines with drift markers.

Where the baseline lives

The rolling history is stored in the pipeline’s state: store under {name}::{row}::__profiling__ (topology mode: {name}::{sink node}), next to the bookmarks — so it needs a state: block (enforced at config load) and a memory store only baselines within one faucet schedule / serve process. The catalog copy is for browsing; it is never the detector’s input.

Profiling runs for real root invocations only: --dry-run, --limit, shard executions, and cancelled runs neither profile nor touch the baseline. A per-row profiling: on a matrix row replaces the top-level block for that row, and a deployment overlay may set it.

Fields

faucet schema profiling prints the JSON Schema.

Metrics

The shipped Prometheus rules include a FaucetProfileDrift alert on the counter.

Relationship to the other governance blocks

• Quality checks assert what you declared, per page, and can quarantine or abort before the write. Profiling learns what is normal and reports after the write.
• Schema drift watches the shape (columns and types against the destination schema). Profiling watches the content of the columns that exist.
• SLA monitoring baselines one number per run (volume) with the same z-score / IQR test profiling applies to every column statistic.