Data-flow policies

A data-flow policy answers “may this column go there?” before any data moves. It labels columns (classifications) and constrains where a label may flow (rules): a customer email must reach only a sink that declares residency: eu, a salary may leave the region only hashed, a finance column must never land in a plain file in production. The verdict is decided statically by faucet validate / policy / plan / doctor, refuses faucet run and the faucet serve submit path, and is backed at run time by value detectors on the real records.

Note

Masking (cookbook) says how to hide a value inside one pipeline. A policy says whether a movement is allowed, and is applied centrally to configs it does not own (--policy, serve --policy).

A policy

cli/examples/policy/eu-pii.yaml

version: 1
description: PII stays in the EU and never lands in plain files in production.
classifications:
  - label: pii
    fields: [email, phone, ssn, date_of_birth]     # full dot-path or leaf key
    field_pattern: "(?i)(^|[._])(first|last|full)_?name$"   # regex over the dot-path
    value_detector: email                          # email|credit_card|ssn|phone|phone_loose|ipv4
  - label: pii
    value_detector: ssn
  - label: finance
    field_pattern: "^(amount|balance|salary)"
rules:
  - name: pii-eu
    when: { label: pii }
    require: { residency: [eu] }        # the sink must declare one of these
    mask: [hash, tokenize, redact]      # or the column reaches it masked
  - name: finance-no-prod-files
    when:
      label: finance
      sink_kind: [jsonl, csv, stdout]
      sink: { environment: [prod] }     # only sinks that declare this
    deny: true
    on_runtime: quarantine              # fail (default) | quarantine

• A classification labels a column by name (fields matches the full dot-path or its leaf key — ssn covers user.ssn; field_pattern is a regex over the dot-path) and/or by value (value_detector, the masking detectors). Name matches are decided before any data moves; value detectors are enforced by the runtime backstop.
• A rule applies to every column carrying when.label, optionally only for some sink_kinds and only for sinks whose when.sink attributes have a listed value (a sink that declares no environment never matches a rule scoped to environment: prod). It is satisfied when every require attribute is present with a listed value, or the column reaches the sink masked with a listed mask action. deny: true is never satisfied.
• on_runtime is what the backstop does when a value detector fires on a record heading for a non-compliant sink: fail the page, or quarantine the row to the DLQ (a dlq: block is required).

faucet schema policy prints the JSON Schema.

Sink attributes

Rules reason about free-form string pairs on the sink:

pipeline:
  sinks:
    eu_warehouse:
      type: postgres
      attributes: { residency: eu, environment: prod }
      config: { … }
matrix:
  - id: customers
    sink: { ref: eu_warehouse, attributes: { environment: staging } }   # adds / overrides

A sink template’s attributes apply to every row that resolves to it; a matrix row’s sink.attributes adds or overrides keys.

Applying a policy

faucet policy   pipeline.yaml --policy eu-pii.yaml   # the per-row report; exit = violations
faucet validate pipeline.yaml --policy eu-pii.yaml   # invalid on any violation
faucet plan     pipeline.yaml --policy eu-pii.yaml   # reports, never fails (a preview)
faucet doctor   pipeline.yaml --policy eu-pii.yaml   # a `policy` probe per root row
faucet run      pipeline.yaml --policy eu-pii.yaml   # refused before any connector is built
faucet serve --policy eu-pii.yaml                    # every submission

The policy can also live in the config as a top-level policy: block (a deployment overlay may carry one), and --policy merges on top: classifications and rules concatenate; two rules with one name are a load error.

The report

$ faucet policy pipeline.yaml --policy eu-pii.yaml
policy: 2 rule(s), 3 classification(s), value detectors enforced at run time — 1 violation(s)
  row customers → sink files (jsonl; environment=prod): 3 column(s) from the contract; labelled: email[pii] amount[finance]
    ! rule `pii-eu`: column `email` (pii) → sink `files` (jsonl) missing attribute `residency` (one of eu)

Per row the report says what the static pass knows:

• Columns come from the row’s data contract (contracts), or from a schema the caller supplies — plan --sample runs the sample and labels its columns; the serve POST /v1/plan does the same. A row with neither has no static knowledge: the report says so and the runtime backstop is the enforcement.
• Labels follow the transform chain through the same column-lineage ops OpenLineage emission uses, so rename_field: { email: contact } yields a labelled contact (via: lineage). A contract describes the row’s output (it is enforced after the transforms), so it lists contact; each contract field is traced back through the chain to the input it came from and keeps that input’s labels. An opaque transform (flatten, explode, keys_case, sql, wasm, custom) carries every labelled input conservatively: a rename cannot hide a label, and the report flags the row as opaque.
• A column the row’s masking policy provably rewrites at that sink (name-matched rules, honouring applies_to) counts as masked, so mask: [hash] is satisfied by a hash masking rule on that field. The match is the masking pass’s own: fields names the full dot-path, and a rule on a container (fields: [phones]) masks every leaf beneath it.

Topology graphs (pipeline.nodes) are evaluated per sink node from the pipeline-level contract; any transform node in the graph makes the labels conservative, and a contract field inherits the labels of every name any transform node renames into it.

The runtime backstop

Whatever the static pass could not see — a column no contract declared, a value that only looks like PII — the policy sink catches. It is the outermost sink decorator (after masking, so a masked value no longer trips its detector): every record’s scalar leaves are classified by name and by value detector and the same rules are evaluated against the sink’s attributes. A label on a container (fields: [phones]) covers every leaf beneath it (phones.0, phones.1), and a leaf the masking pass rewrote by name counts as masked with that action, exactly as in the static pass. The sink sees records after the transforms, so a column a row’s transforms renamed (ssn → tax_id) carries the labels of its original name there too.

• on_runtime: fail — the page is refused before it is written; faucet run fails with Policy pii-euviolated on columnmail“ and faucet serve audits policy.denied by principal runtime.
• on_runtime: quarantine — the offending rows go to the DLQ with the rule in the envelope; the rest of the page is written. The DLQ receives the whole record, so it is evaluated as a destination too (sink id dlq, with the attributes: of dlq.sink): when the rules would not let the record reach the DLQ either, the run fails instead of moving it there. Every labelled column of the record is checked at the DLQ, not only the one that broke the main sink’s rule. faucet validate / faucet policy report those violations against sink dlq.
• Every row that reaches the DLQ is checked there, whichever pass sent it: a quality, contract or schema-drift quarantine, a per-row sink failure, or a policy quarantine. The DLQ is a governed destination of its own, so a row whose labels may not reach the DLQ’s attributes fails the run instead of landing there. The static pass evaluates sink dlq whenever a dlq: block is configured, with the masking that reaches it — a masking rule whose applies_to names dlq (or the DLQ’s connector kind) masks the envelope payloads and satisfies a mask: rule there.
• A quarantine rule makes the run take the row path, so the same input quarantines the same rows whichever fast path (columnar, native) the connectors would otherwise take. It cannot be combined with delivery: exactly_once on the atomic-watermark path (that commits whole pages and forbids a DLQ): faucet validate refuses the pair by name and suggests on_runtime: fail or keyed-upsert effectively-once. Only a quarantine rule that can reach the row’s sink counts — one whose when.sink_kind / when.sink attributes exclude that sink does not block an exactly-once row.

pipeline:
  dlq:
    sink:
      type: jsonl
      config: { path: /eu-bucket/quarantine.jsonl }
      attributes: { residency: eu }

faucet_policy_violations_total{pipeline,row,rule,phase,action} counts both phases (phase = static | runtime, action = refuse | fail | quarantine).

Under faucet serve

A server checks every submission (POST /v1/runs, a template trigger, a backfill, a trigger fire) against its policy merged with the config’s own block: a violation is a 422 whose details carry the report, and a policy.denied audit entry naming the rule, column and sink. Registering a violating pipeline template warns (warnings[] on the summary) rather than refusing — a source template may still be composed with a compliant sink — and triggering it hits the submit gate. POST /v1/plan reports the verdict under policy.

Where the policy lives

faucet serve --policy-store server|file (default server) decides where the server policy is kept.

• server (the default): the policy in force is the newest approved version in the server’s store, the run-history backend (so a SQLite or Postgres --history keeps it across restarts and shares it between replicas). A --policy FILE only seeds it: when the store holds no version, the file is imported as version 1, recorded as imported from the file by system (audit policy.imported). After that the stored version wins on every start. A file that later differs from the version in force is not applied: the server logs a warning at start, GET /v1/policy reports file_drift: {path, differs: true}, the console shows a banner with Import the file as a new version (a policy_update change request, approved like any edit), and faucet_serve_policy_file_drift is 1 (alert FaucetServePolicyFileDrift).

  The store is the only authority. Under --policy-store server the file is a seed and nothing more: editing it never changes what is enforced, on this replica or any other. The drift warning, banner and gauge only tell you the two differ; the policy changes when someone imports the file (or proposes any other version) and an admin approves it. To make a file the policy, run with --policy-store file.

• file: the file is the policy, re-read on every start. The console and API show it read-only; every change endpoint answers 409. It needs --policy.

With the in-memory history (no --history) nothing approved survives a restart, so every policy edit — server or tenant, propose or rollback — is refused with 409 (“set a persistent —history to edit the policy here”). A --policy file still loads and is enforced, and the console shows it read-only with that hint.

What a restart does:

--policy-store	--history	--policy FILE	On start
server	persistent	none	the newest stored version is in force (none stored: nothing enforced)
server	persistent	given, store empty	the file is imported as version 1
server	persistent	given, store has a version	the stored version stays in force; a differing file is reported (file_drift), not applied
server	in memory	given	the file is enforced, read-only; edits answer 409
server	in memory	none	nothing is enforced; edits answer 409
file	any	given	the file is re-read and enforced, read-only; edits answer 409
file	any	none	the server refuses to start

A server-managed policy is never edited in place. An admin proposes a new version (POST /v1/policy, or Policy → Edit policy in the console); the request is a policy_update change that always needs approval, by default from an admin other than the requester (the approvals: block of --auth-config can name other approvers for policy_update). On approval the draft is checked again and stored as the next version only if the version it was based on is still the one in force — a second change approved after the first is invalidated instead of overwriting it. Every replica reads the version in force at admission, so an approved change reaches all of them for the next submission.

# Preview a draft: its problems (with lines), or its diff and impact
curl -s -X POST localhost:8899/v1/policy/preview -H "Authorization: Bearer $ADMIN" \
  -H 'Content-Type: application/json' -d "$(jq -Rs '{policy: .}' eu-pii.yaml)"

# Propose it (base_version: the version you edited; null for the first)
curl -s -X POST localhost:8899/v1/policy -H "Authorization: Bearer $ADMIN" \
  -H 'Content-Type: application/json' \
  -d "$(jq -Rs '{policy: ., reason: "Hash-only for email", base_version: 7}' eu-pii.yaml)"

# Another admin approves it
curl -s -X POST localhost:8899/v1/changes/$ID/approve -H "Authorization: Bearer $OTHER_ADMIN"

The preview checks the draft with the same validation as a submission and measures its impact: every registered pipeline template (its stable version, params bound to placeholders) and the stored configs of the last 7 days of runs are checked against the policy in force and the draft, and the ones the draft would newly refuse or newly allow are listed. For this the server keeps, on each run, a projection of its config: connector kinds and attributes, the DLQ’s kind and attributes, transforms, contracts, masking rules (never their keys) and the config’s own policy: block — no connector settings and no value from a secret reference. It is kept only while the policy lives in the store, never shown by the API or faucet history --json; a topology config is counted as checked at run time only. A template whose columns are unknown before a run (no contract:) is counted as checked at run time only — the runtime backstop still applies to it. The approver sees the same diff and impact on the change request.

Versions are append-only (GET /v1/policy/versions): each names who requested it, who approved it and why. A rollback (POST /v1/policy/rollback {version, reason}) is a change request too; approved, it stores a new version with the earlier document (restores: N).

Runs keep the version they were admitted under. The stored versions a run was checked against are recorded on its record (policy_pin), and a claim, shard or resume on another replica enforces those versions rather than a newer one. A run admitted while a layer had no stored policy stays unchecked on that layer when it is reloaded, rather than meeting a version approved after it. A run admitted under a --policy file (--policy-store file) records the SHA-256 of the file’s canonical policy. The server stores that content once, at startup, under its address, and every replica loads a pinned run’s policy from there, whatever file it was started with; only a run whose pinned content is not stored is refused. Until the content is stored (the history was unreachable at startup) /readyz shows policy_seed: "pending" and new work answers 503. A record from 1.17.0 has no pin: the newest policy applies to it.

Upgrading a cluster. A replica from before 1.18 does not enforce pinned versions. While one is a live member, runs under a policy and policy changes answer 503 (“finish the rolling upgrade”) on upgraded replicas; runs with no policy in force are admitted as before. Finish the rolling upgrade promptly, and do not roll back below this release while pinned runs are pending.

When the history is unreachable (the server is degraded), the server admits no new work at all, so no run is checked against a policy it could not read. Policy edits and approvals answer 503 until the store recovers. A --policy file that should seed an empty store waits for it the same way: /readyz shows policy_seed: "pending" and admissions answer 503 until it has.

Refusals and the dry run

GET /v1/policy/refusals lists what the policy stopped — submissions the static gate refused and runs the backstop failed or quarantined — with the rule, column, label, sink and the layer (server or tenant) whose rule it was. POST /v1/policy/check (operator) dry-runs a config, or a registered pipeline template, against the policy in force without running anything; with sample records for one row, their values label the columns too.

Tenant policies

Each tenant can add its own classifications and rules on top of the server policy. A tenant policy is add-only: it is merged with the server’s (server ⊕ tenant ⊕ pipeline), so it can tighten what the server allows but never remove, rename or loosen a server rule — a draft that reuses a server rule’s name, or tries to disable one, is refused. Its rules may govern labels the server classifies. It applies to that tenant’s runs only. A server draft is checked against every tenant policy too: one that would leave a tenant’s merged policy invalid (a tenant rule governing a label the draft no longer classifies) is refused, naming the tenant. The check runs again when an approved change executes, so a server change and a tenant change approved together cannot leave a broken pair: the second is invalidated. If the two land at the very same moment, the tenant’s runs are refused (never run without policy) until either change is rolled back; faucet_serve_policy_merge_failures_total{tenant} counts it and FaucetServePolicyMergeBroken alerts.

# acme's policy: personal data never reaches a partner-tier sink
version: 1
rules:
  - name: acme-no-partner
    when: { label: pii }          # a server classification
    require: { tier: [internal] }

The tenant’s own admin principals (--auth-config entries with tenant: and role: admin) and server admins propose changes (POST /v1/tenants/{tenant}/policy, preview and rollback alongside); the tenant_policy_update request is approved by another admin of that tenant or a server admin, never the requester. A tenant’s viewers and operators read it; another tenant’s is a 404. A tenant principal sees the server rules that apply to it as outlines (name, label, what they require) — not the rest of the server policy — and its own refusals. A server policy change that would reuse a tenant rule’s name is refused, so the layers can always be merged. Deleting the tenant deletes its policy versions.

In the console

Govern → Policy shows the policy in force (classifications, rules, and how many columns the catalog has seen per label), its refusals, a dry run and the version history, with Edit policy for admins on a server-managed policy. The editor validates as you type, then previews the diff and impact and asks for a reason before it creates the change request; pending changes show as a banner until approved. A --policy file that differs from the version in force shows as a banner with Import the file as a new version; on the in-memory history the page is read-only and says to set a persistent --history. With the tenant switcher set to a tenant, the page shows the server rules beside that tenant’s rules and history; a tenant principal sees its own tenant’s view. See the web console.

Worked example

cli/examples/csv_to_jsonl_with_policy.yaml is compliant: its sink declares residency: eu and the salary column is hashed. Remove the attribute and faucet validate exits 1 naming pii-eu; remove the masking rule and finance-hashed-or-nothing fires.

Reference

• policy and attributes in the config reference
• faucet policy, the --policy flag on run / validate / plan / doctor, serve --policy
• POST /v1/plan and the 422 submit refusal
• The policy endpoints (/v1/policy*, /v1/tenants/{tenant}/policy*) and change requests
• RFC 0010