Dead-letter queues

A dead-letter queue (DLQ) keeps a pipeline running when a handful of records fail to write, instead of aborting the whole run. Failing rows are wrapped in a fixed-shape envelope and routed to a separate DLQ sink before the page’s bookmark advances.

When it helps

Sinks whose underlying API reports per-row results — BigQuery insertAll, Elasticsearch _bulk — can tell exactly which records failed. The DLQ captures just those, while the good rows commit normally.

Configure a DLQ

Add a dlq: block naming a sink to receive the bad rows and the policy for sinks that can’t report per-row outcomes:

pipeline:
  source: { type: rest, config: { /* … */ } }
  sink:   { type: bigquery, config: { batch_size: 0, /* … */ } }   # one insertAll per write
  dlq:
    on_batch_error: dlq_all      # or `propagate`
    sink:
      type: file
      config:
        path: ./dead-letters.jsonl

A DLQ always appends. Once a record is dead-lettered the bookmark moves on, so the DLQ file holds its only copy. The file, jsonl, csv, S3, GCS, Azure Blob and SFTP sinks are therefore always opened in append mode when used as a DLQ — every run, row and fan-out invocation adds to the same file, and rows of one run that share a DLQ write through a single writer. A DLQ setting that would replace the file (append: false, if_exists: replace) or a single .parquet file (which cannot be appended to) is refused when the config loads, as is a DLQ path that is also a data sink’s path. Clear handled dead letters with faucet dlq discard.

The envelope

Each dead-lettered record is wrapped in a fixed-shape envelope — the original record plus the metadata needed to inspect, fix, and replay it:

{
  "error": { "kind": "ContractViolation", "message": "status: value not in enum" },
  "reason": "contract",
  "payload": { "order_id": "A-17", "status": "backordered" },
  "ts_ms": 1751760000000,
  "sink": "jsonl",
  "pipeline": "orders_csv_with_contract",
  "row": "",
  "record_index": 3
}

• payload — the record as it entered the write path: after the transform chain and after masking, since those passes run before the sink. This is what a replay re-feeds, and it is why a replay does not re-run them (see Replaying).
• reason — which stage quarantined the row: quality, contract, schema_drift, or partial / dlq_all for a sink-side row failure. This is the value the --reason filter matches.
• error.kind / error.message — the typed failure and its message.
• record_index — the row’s position within its original page.

on_batch_error policy

For a sink that can only succeed or fail a whole batch (no per-row detail):

• propagate — a batch failure aborts the run (the default, fail-fast behavior).
• dlq_all — route every row in the failed batch to the DLQ and keep going.

Sinks that do report per-row results (BigQuery, Elasticsearch, and the HTTP sink in Individual mode) override the partial-write path so only the genuinely failed rows are dead-lettered — the already-delivered rows are not duplicated into the DLQ.

Batch atomicity and dlq_all

dlq_all sends every row of a failed write to the DLQ. That is only safe when the failed write landed nothing: a DLQ replay writes every routed row again, so rows that had already committed would land twice — silently.

So every sink declares what a failed write leaves behind:

Batch atomicity	A failed write…	dlq_all
atomic	landed nothing (one statement, a transaction around every chunk, one object upload, one table commit)	allowed
per_row	reports which rows failed; a whole-write error means nothing from that write landed	allowed
best_effort	may have landed some rows and cannot say which	refused

Several sinks are all-or-nothing only in some configurations — typically batch_size: 0, which makes one page one statement or one request. The capability matrix lists each sink and the config that makes it atomic. A third-party sink that does not declare anything counts as best_effort.

faucet validate and faucet run (before reading anything) refuse dlq_all on a best_effort sink:

row 'orders': dlq: on_batch_error 'dlq_all' is unsafe with sink 'jsonl' (batch
atomicity 'best_effort'): a failed batch may already have written some rows, and
replaying the DLQ would write them again. Use on_batch_error 'propagate',
configure the sink with write_mode 'upsert' and a key so a replay overwrites
instead of duplicating, or set allow_duplicates_on_dlq_all: true to accept the
duplicates

The ways out, safest first:

1. Make the write atomic — e.g. batch_size: 0 on BigQuery, SQLite, Snowflake, ClickHouse or Redshift COPY.

2. Write by key — write_mode: upsert (or delete) with a key: a replayed row overwrites itself, so duplicates cannot happen.

3. Use propagate — fail the run and let the next one retry from the bookmark.

4. Accept the duplicates — set allow_duplicates_on_dlq_all: true on the dlq: block when the destination tolerates them (downstream dedup, an append-only audit log):

   dlq:
     sink: { type: file, config: { path: ./dead-letters.jsonl } }
     on_batch_error: dlq_all
     allow_duplicates_on_dlq_all: true

Seeing what happened to each batch

Every sink write is counted by outcome — committed, dlq_partial (some rows went to the DLQ per row), dlq_all (the whole write went to the DLQ) or failed (the error propagated):

• the faucet_batch_outcomes_total{pipeline,row,sink,outcome} counter;
• faucet run --output json (rows[].batches);
• faucet status, which marks a row degraded when its last run sent writes to the DLQ (last run: of 40 sink write(s), 2 went to the DLQ whole (dlq_all));
• the run-history record (invocations[].batches) and the batches column of the web console’s run detail.

Failure budgets

A DLQ keeps a run going through occasional bad rows, but a flood of failures usually means something is broken upstream. Two optional budgets turn the DLQ into a circuit breaker:

  dlq:
    sink: { type: file, config: { path: ./dead-letters.jsonl } }
    max_failures_per_page: 50    # abort if a single page dead-letters > 50 rows
    max_failures_total: 500      # abort once the run has dead-lettered > 500 rows

When a budget trips, the run aborts — but only after the page that crossed the threshold is fully committed: its surviving rows are written to the main sink, its failed rows are routed to the DLQ, and (if the page carried one) the bookmark advances. So the committed survivors are not re-delivered when you fix the upstream problem and re-run, and the failed rows are preserved in the DLQ for replay rather than dropped. The run still stops, so you get alerted.

The DLQ is a governed destination

A row on its way to the DLQ is held to the same guarantees as one on its way to the main sink (RFC 0013, #870):

• Policy — every envelope’s payload is classified and checked against the data-flow policy with the DLQ’s own attributes:, whichever pass quarantined it (quality, contract, schema drift, a policy rule, or a per-row sink failure). A row whose labels may not reach the DLQ fails the run instead of landing there.
• Masking — a masking rule whose applies_to names dlq (or the DLQ sink’s connector kind) and not the main sink masks the envelope payloads. Rules that apply to the main sink already ran on the page.
• Budget — DLQ envelopes count against the run’s budget: together with the main sink’s records; a ceiling the DLQ alone crosses stops the run.
• Metering — DLQ records and bytes appear in the usage record as dlq_records_written / dlq_bytes_written.
• Resilience — a DLQ write that fails transiently is retried under resilience: when the DLQ sink’s write is replay-safe (keyed); an append-only DLQ is not retried, because a retry after a lost response would write the envelopes twice.

pipeline:
  masking:
    rules:
      - { match: { fields: [notes] }, action: { type: redact }, applies_to: [dlq] }
  dlq:
    sink:
      type: jsonl
      config: { path: /eu-bucket/quarantine.jsonl }
      attributes: { residency: eu }

Inspecting the DLQ

faucet dlq inspect reads a DLQ location back and groups it by reason and error kind, with a sample — so you can see why rows failed before deciding what to do:

$ faucet dlq inspect ./dlq/contract_breaches.jsonl
DLQ inspect: ./dlq/contract_breaches.jsonl
  files read: 1   envelopes: 42   malformed: 0   non-envelope: 0
  by reason:
    contract       42
  by error kind:
    ContractViolation    42
  sample (5 of 42):
    [contract/ContractViolation] status: value not in enum
      {"order_id":"A-17","status":"backordered"}

The location may be a single .jsonl file, a directory of JSON Lines files (.jsonl / .ndjson, plain or .gz / .zst-compressed, which are decoded), or a glob. A directory or glob skips discard archives, the replay-failed files a replay writes, and files a writer has not finished (*.faucet-tmp*, *.faucet-purge); name such a file directly to read it. Blank, malformed, and non-envelope lines are counted (malformed / non-envelope) but never abort the read. Add --reason contract to restrict the breakdown, --limit N to size the sample, or --json for a machine-readable summary.

Replaying

Once you’ve fixed the root cause — a transform, a contract, the destination schema — faucet dlq replay re-feeds the quarantined payloads through a pipeline config (quality → contract → sink):

$ faucet dlq replay orders.yaml --from ./dlq/contract_breaches.jsonl --dry-run
DLQ replay (dry-run): 42 candidate record(s) from ./dlq/contract_breaches.jsonl
would be re-fed; 42 would reach the sink. Failures would go to
./dlq/contract_breaches.replay-failed.jsonl.

$ faucet dlq replay orders.yaml --from ./dlq/contract_breaches.jsonl
DLQ replay: 42 candidate record(s) re-fed; 42 written to the sink. …

A replay skips the transform chain and the masking pass, because the payload already went through both before it was captured, and neither is idempotent: re-running the hash transform or masking’s hash/tokenize action would produce H(H(x)) — breaking the joinability masking exists to guarantee — a set stage would re-stamp ${now.*} with the replay time, and cast / json_parse / split would re-run against already-converted values. The quality and contract passes are re-applied: they are pure checks over the record, so re-checking is idempotent, and a replay is exactly when you want them enforced.

Under faucet serve a replay masks a payload again unless it proves it was masked under today’s rules. When the server has a vault key (--vault-key), every envelope a pipeline with a masking: block dead-letters carries a masking object: {"rules": "<fingerprint>", "sig": "<signature>"}. The fingerprint covers the masking rules that apply to the pipeline’s destination and the masking key; the signature, keyed from the vault key, covers the tenant, pipeline, row, fingerprint and payload. A replay through POST /v1/dlq/replay (including one for a tenant) writes a payload as it is only when its signature verifies under the current or a previous vault key and its fingerprint matches the rules in force. Any other payload — no masking object, a payload or row that changed, rules or a masking key that changed since, or a server without a vault key — goes through the masking rules again, which is safe for redact and partial (applying them twice gives the same value). A payload a hash or tokenize rule would rewrite is left out instead, since it may hold a value that is hashed already: the replay counts it in skipped_unproven and leaves it in the DLQ. Replay such rows with a local faucet dlq replay from the row that wrote them, which trusts the envelope’s pipeline and row, or run the server with --vault-key so new dead letters carry a masking record. A DLQ written by an older release carries no masking object, so its hash and tokenize rows are left out of such a replay. A local replay with --all-rows leaves out the same rows for envelopes another row wrote.

A replay runs under the config’s budget: and resilience:, like any run. Under faucet serve, POST /v1/dlq/replay returns the replay’s id (or takes one you choose), and POST /v1/runs/{id}/cancel stops it at its next page boundary while it runs. A replay tenant (tenants feature) loads the config under that tenant’s scope and is admitted against its limits, whose per-run ceilings bound the replay. A replay that writes is recorded as a run under its id (label dlq_replay: "true", replay_of = the DLQ location) for as long as it runs, so it shows in the run list and counts toward its tenant’s max_concurrent_runs: with the limit at 1, a run submitted for the tenant while its replay runs is refused with 429. A --dry-run replay writes nothing and is not recorded. An id is claimed atomically: a second replay with an id in flight, or one naming a run already in the history, is refused with 409 and leaves the first untouched.

The DLQ locations a request names — from and failed_dlq here, location for inspect and discard — are server-local paths, so the server checks them against its reference policy like a connector path in a submitted config: a system tree or a server credential file is refused, a path outside --allow-path-root is refused (403), and a tenant’s request is confined to --tenant-path-root/<tenant>.

Rows that fail again on replay are quarantined to a fresh DLQ — a replay-failed.jsonl sibling of the source by default (override with --failed-dlq) — never back to the source, so a replay can’t loop. --dry-run reports what would be replayed without writing; --reason replays only matching envelopes; --row picks a specific root when the config has several.

A replay only adds records, and only the target row’s own. Every envelope names the pipeline and row that dead-lettered it, and a shared DLQ (a pipeline-level dlq: is inherited by every matrix row) holds other rows’ records too, so a replay keeps the envelopes the selected row of this config’s name: wrote and reports the rest as skipped_other_rows — replay them with --row <theirs>, or pass --all-rows to send every envelope through the selected row. Partitions of one row share a destination and replay together. Nothing in a replay replaces the destination: a sink’s write_mode: overwrite becomes an append (an overwrite would swap in a table holding only the replayed rows), a file sink that replaces its file each run appends to it, a sink that writes one whole local .parquet file is refused, and a row’s scoped cleanup (complete_for with on_missing: delete) is not run — the replayed rows are never the complete set it deletes against.

Make replay idempotent

A replay is a fresh run — if some of a page originally landed before the failure, replaying can duplicate it on an append-only sink. Use write_mode: upsert on the target so a replayed row overwrites rather than duplicates.

Discarding

Once envelopes are handled (replayed, or known-bad), faucet dlq discard clears them from the backlog:

$ faucet dlq discard ./dlq/contract_breaches.jsonl --reason contract --before 7d
DLQ discard: archived 42 envelope(s) across 1 file(s) → ./dlq/contract_breaches.jsonl.archived

By default the DLQ file itself is not rewritten — a running pipeline may be appending to it, and a rewrite would lose what it appends meanwhile. Instead each discarded envelope is recorded in a <file>.discarded sidecar, and inspect, replay, discard and faucet status skip the lines it lists. By default discarded envelopes are also copied to a <file>.archived sibling; --delete only records them. The file keeps its size, so write the DLQ to a dated path (./dlq/${now.date}.jsonl) and remove old days to reclaim space. Archives and sidecars are not .jsonl files, and a directory or glob location skips them (including the <stem>.archived.jsonl archives older versions wrote), so a later inspect or replay of the directory never brings discarded envelopes back. Name an archive file explicitly to read it. --reason and --before (an RFC3339 timestamp or a relative age like 7d / 24h / 30m) select what to discard — everything else, including non-envelope lines, is left untouched.

A discarded envelope’s payload stays in the file. When it must not be kept — PII that reached the DLQ after a masking mistake, say — add --purge: every discarded envelope is removed from the file itself (an atomic rewrite), with no archive. Run it while nothing writes the DLQ; a file that changes while it is being read is refused. Complete lines appended between that check and the swap are carried over to the end of the rewritten file (re-sealed into its integrity chain for an encrypted DLQ) as a best effort, but a line still being written at that moment, or anything a writer that keeps the file open writes after the swap, goes to the replaced copy and is lost — so stop the pipeline first. An encrypted DLQ (below) is re-sealed as it is rewritten — a fresh header and an integrity trailer over the records kept, under the current key — so it still opens and the sink can keep appending to it; that needs the key (--encryption-key), and a file that fails its integrity check is refused.

A file DLQ that rolls over writes numbered parts. Its configured path reads every part — failed-{part}.jsonl, or failed.jsonl with max_records_per_file / max_bytes_per_file (written as failed-00001.jsonl, …) — in inspect, replay, discard and faucet status.

Encryption at rest

DLQ envelopes carry failed records verbatim — on a shared or compliance-scoped host that can be a plaintext-at-rest gap. When the DLQ sink is a file sink writing JSON Lines (or the deprecated jsonl sink), seal every envelope line with AES-256-GCM (requires a build with the encryption feature — included in --features full):

dlq:
  sink:
    type: file
    config:
      path: ./dlq/failed.jsonl
      encryption:
        key: ${vault:secret/faucet#dlq-key}
        # previous_keys: ["${env:OLD_KEY}"]   # rotation: read-only candidates

Each record line is encrypted individually and written base64-encoded, so the file stays line-oriented and append-safe. Keep the DLQ uncompressed: with compression (or a .gz / .zst path) the file sink seals the whole file instead of each line, which faucet dlq and faucet status cannot read line by line — so a file DLQ with both compression and encryption is refused when the config loads. (The jsonl sink refuses the combination outright.)

The faucet dlq verbs handle sealed files transparently:

• inspect / discard — pass --encryption-key <KEY> (repeat the flag to also try rotated keys). Without a key, sealed lines are counted and reported as encrypted — never mistaken for malformed lines, never mangled.

• replay — picks the key up automatically from the config’s own dlq: block’s encryption (a JSON Lines file sink or a jsonl sink); --encryption-key overrides.

• With the key, inspect, replay and discard first check the file’s whole-file integrity (header, trailers, line order): a file with lines removed, reordered or added is refused rather than read as a subset. Lines after the last trailer — a flush a writer has not finished — are not covered by any trailer, so they are skipped with a warning (and read once the flush completes). The check streams the file, so it needs no more memory than the longest line.

• discard keeps and archives lines verbatim (still sealed) — filtering decrypts only in memory; nothing is ever re-written in plaintext.

• A sealed file’s .discarded sidecar counts only entries sealed with the key. Plain entries — lines discarded before the file was sealed, or written by someone without the key — are not honoured, so those lines are read again; inspect, replay and discard --purge report how many there are (unsealed_discards in --json). To discard them again, record them sealed with the key:

  $ faucet dlq discard ./dlq/out.jsonl --reseal-plain --encryption-key-file ./dlq.key
  DLQ discard: re-recorded 3 discard entr(ies) sealed with the key.

• Rows that fail again during a replay land in the fresh failure DLQ sealed with the same keys.

• Supply keys without putting them on the command line with FAUCET_DLQ_ENCRYPTION_KEY or --encryption-key-file <path>.

The same encryption block also seals file state-store bookmarks — see State & resumability.

Note

The full design is in docs/superpowers/specs/2026-05-24-dlq-design.md and the faucet_core::dlq module on docs.rs.