Reliability testing

faucet’s claim is not that rows arrive — it is that the data can be trusted: nothing is lost, nothing is duplicated, nothing is silently corrupted, a full refresh is atomic, an incremental run resumes exactly where it stopped. Those are promises about behaviour under failure, so they are verified by a dedicated test program rather than by the connector suites.

This page describes what that program covers, how to run each tier locally, and how to add to it.

Why these tests exist separately

A guarantee like “the bookmark is persisted only after the sink confirms the data it covers” is not a property of any single function — it is a property of the order in which the engine calls two collaborators. A unit test over pure logic cannot observe an order. A connector test cannot either: it sees its own side of the boundary.

So the reliability suites run the real Pipeline::run against doubles that fail at one named boundary and record everything that happened, in order. The assertions are then written against the sequence — which is what the guarantee actually says — rather than against a final row count, which is what a weaker test checks and which stays green through a real regression.

The tiers

Tier	What it covers	Docker?	Runs
Engine guarantees	Bookmark ordering, exactly-once (both mechanisms), overwrite atomicity, cleanup safety, cancel-still-flushes, panic isolation on every transfer path	No	Every PR, required
State compatibility	Bookmarks / exactly-once envelopes / versioned state envelopes / commit tokens written by a release still load; an older bookmark shape migrates; a newer one is refused; every state key a release wrote still renders, or is migrated from	No	Every PR, required
Connector conformance	Per-connector battery (capabilities truthful, bounded memory, idempotent replay, …); every declared capability has its check (scripts/conformance-coverage.py)	Per connector	Every PR
Drift policy matrix	All five on_drift policies against a live evolving destination	No	Every PR, required
Containerized integration	Real databases, Kafka, object stores; CDC replication	Yes	Every PR, reported
Fidelity round-trip	The shared corpus lands type-exact through every connector — a source↔sink pair, a one-sided run against the backend, or the connector’s value converter; every connector runs it (scripts/fidelity-coverage.py) and none converts a decimal through a float (scripts/decimal-f64.py). See value fidelity	Per connector	Every PR, reported
Container fault injection	A real connection cut mid-cursor	Yes	Every PR, reported
Deployment chaos	faucet serve from the Helm chart on a 3-node kind cluster: pod and node kills, a full outage, a history-database outage, rolling restarts, a burst, a poison run, single-pod upgrades. See chaos testing	Yes (kind)	Nightly and on demand

The first two tiers are deliberately Docker-free and run in seconds. They are the required gate because a regression in them corrupts data silently — the run still reports success — so they must never be skippable or flaky.

Running them locally

# The required gate: engine guarantees + state-format compatibility.
cargo test -p faucet-conformance --all-features

# The engine's own unit suites, which cover the DLQ routing/budget matrix
# and the masking-before-DLQ ordering.
cargo test -p faucet-core --all-features

# One guarantee at a time.
cargo test -p faucet-conformance --test reliability_bookmark_ordering
cargo test -p faucet-conformance --test reliability_exactly_once
cargo test -p faucet-conformance --test reliability_lifecycle
cargo test -p faucet-conformance --test reliability_cleanup_safety
cargo test -p faucet-conformance --test reliability_fault_injection
cargo test -p faucet-conformance --test compat_state_format
cargo test -p faucet-conformance --test reliability_state_migration
cargo test -p faucet-conformance --test reliability_panic_isolation
cargo test -p faucet-conformance --test reliability_governance
cargo test -p faucet-conformance --test forwarding

# Each resumable source's frozen bookmark shapes (#736).
cargo test -p faucet-source-postgres-cdc --test state_compat

# The drift-policy matrix (Docker-free, required tier).
cargo test -p faucet-sink-sqlite --test drift_policy_matrix

# Fidelity pairs. jsonl + sqlite need no Docker; the rest boot a container.
cargo test -p faucet-sink-jsonl   --test fidelity
cargo test -p faucet-sink-sqlite  --test fidelity
cargo test -p faucet-sink-postgres --test fidelity
cargo test -p faucet-sink-mysql   --test fidelity
cargo test -p faucet-sink-mongodb --test fidelity

# Container fault injection: stops the database mid-cursor.
cargo test -p faucet-source-postgres --test fault_injection

Container-backed suites need a Docker socket. On macOS with colima:

colima start
export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"
cargo test --workspace --all-features

Note

The SQL Server image is x86-only, so the MSSQL suites cannot run on an ARM Mac — CI covers them.

What each guarantee suite asserts

Bookmark ordering (reliability_bookmark_ordering)

Every persisted bookmark is backed by writes the sink already confirmed. A failed write, a failed flush, and a failed state-store put each leave the durable bookmark behind the failure, so a resumed run re-reads rather than skips. Resume starts at the page after the last durable bookmark — no gap, and at most one page of duplicates.

Exactly-once (reliability_exactly_once)

Both mechanisms, separately:

• Atomic watermark — writes route through the idempotent path with monotonic, distinct commit tokens; a crash in the window between the sink’s commit and the state-store put replays nothing; re-running a completed pipeline writes nothing; a failed write leaves no token behind to skip on.
• Keyed upsert — convergence with no watermark at all.

The suite also runs a control arm: the identical injected failure under the default at-least-once mode, asserting that it does duplicate. Without that, the exactly-once assertions would not be distinguishing the two modes and could pass for the wrong reason.

Overwrite atomicity and cancellation (reliability_lifecycle)

begin_overwrite precedes the first write and commit_overwrite follows the last. A failed write, a failed flush, and a cancel each abort without ever swapping — a mid-run truncate-then-fail is the most destructive thing a data tool can do. A failing commit surfaces rather than reporting success.

Separately, a cancelled run flushes: that is the difference between a cooperative cancel and a dropped future, and it is what makes a buffered Parquet footer or an S3 multipart commit instead of orphaning.

A source that buffers records it cannot re-read (a live WebSocket feed) takes the run’s token through Source::set_cancel and returns true; the pipeline then keeps polling it after the cancel, so the tail it yields is written and flushed before the run ends.

Cleanup safety (reliability_cleanup_safety)

Scoped cleanup is the only feature that deletes destination rows, so its tests are about when it must not fire: a failed run, a failed flush, a cancelled run (which returns Ok — the trap), and an overflowed key set all delete nothing. A complete run that legitimately read zero rows still cleans the scope, because that is a complete answer.

State-format compatibility (compat_state_format)

Frozen files under crates/conformance/tests/fixtures/state/ hold state written by a released version. The tests read them and assert the current code still understands them.

A round-trip test cannot catch this class of bug, because it serializes and deserializes with the same code — both halves move together and the test stays green through a breaking change. If one of these fails, the fixture is not what is wrong: either the reader regressed, or the change needs an explicit migration.

Since #736 every bookmark is stored in a versioned envelope ({faucet_state, owner, schema, data}); the envelope fixtures (state-envelope-v1.json, eo-state-envelope-v1.json, scalar-state-envelope-v1.json) freeze its shape, and every pre-envelope fixture must still resolve as a legacy schema-0 bookmark. Each resumable source (postgres-cdc, mysql-cdc, mssql-cdc, mongodb-cdc, kafka, kinesis) keeps its own released bookmark shapes under crates/source/<kind>/tests/fixtures/state/, read by its state_compat.rs.

State keys are frozen the same way (#882). A bookmark is found by its key, and a key rendered differently from the one a release wrote under looks exactly like a first run — a full re-sync, or a skipped incremental range — with a green run. key-renderings-v<scheme>.json lists, per key scheme (faucet_core::KEY_SCHEME_VERSION), each key’s components and the key that scheme rendered: the current scheme’s keys must render byte-for-byte, and every key an earlier scheme rendered must be one of StateKey::legacy_renderings() — the keys the executor’s run-start migration moves a bookmark from. Changing a rendering without bumping the scheme and keeping the old rendering reachable fails these tests. cli/tests/state_key_construction.rs complements them: it fails when a state key in the CLI or core is assembled with format! instead of StateKey.

State migration (reliability_state_migration)

Two guarantees about state crossing a boundary, both asserted through a real Pipeline::run:

• Across backends (#735) — faucet state export → import onto another backend resumes exactly where the last run stopped, at-least-once and exactly-once.
• Across releases (#736) — a bare bookmark in an older shape is migrated by the source and resumed with no gap and no re-read, and the next bookmark is written in the envelope at the current schema; inside an exactly-once wrapper only the bookmark is migrated and the sequence carries on; state from a newer release, a newer envelope, or another source is refused with StateIncompatible before anything is read or written; legacy writes (a mixed-version cluster) store the bare bookmark and refuse a source past schema 0.

Drift policies (sink/sqlite/tests/drift_policy_matrix, interop-tests/tests/sqlite_drift_policy_matrix)

All five on_drift arms driven through the real pipeline against a destination that genuinely evolves. Each has a different silent wrong answer — warn dropping the page, evolve writing without the DDL, quarantine writing the drifting rows anyway, fail writing and then raising — so each is asserted separately, plus the control that a non-drifting page is untouched by every policy, and that quarantine without a DLQ is refused before any data moves.

Fidelity pairs (sink/*/tests/fidelity)

Five destinations, one shared corpus. What they establish together is more than each does alone:

Destination	Result
jsonl	Exactly lossless, no tolerance. The control: no schema, so no excuse.
sqlite	JSON mode lossless; auto-map loses booleans (1/0) and -0.0’s sign
postgres	Typed columns fully faithful — including -0.0 in DOUBLE PRECISION; JSONB normalises -0.0 away via numeric
mysql	Typed columns faithful; backslashes survive (bind params, not literals)
mongodb	BSON fully faithful, -0.0 included; large integers stored as Int64, not Double

The Postgres result is the instructive one: “Postgres loses negative zero” is false in general and true for the JSONB document path. A per-pair test is what makes that distinction visible instead of folklore.

Container fault injection (source/postgres/tests/fault_injection)

Stops the database container while a cursor is open — a harder cut than a proxy toxic, since the socket dies with no FIN, and it needs no extra image. Asserts the run fails rather than reporting success on partial data, terminates inside a hard ceiling, and never hands back a bookmark it cannot back.

The read is made slow deterministically with pg_sleep per row rather than by tuning a delay: the first version of this test raced, the table streamed in before the cut, and it “failed” on a perfectly healthy run.

Dead-letter boundary (reliability_dlq_boundary)

The DLQ is the one error path where the run keeps going and still reports success, which is what makes a mistake here so expensive: a row is dropped, or a bookmark advances past a row that was never made durable anywhere, and nothing in the exit code says so.

Asserts, against one interleaved log shared by the main sink, the DLQ sink and the state store: a rejected row is durable in the DLQ before the bookmark covering its page is persisted (reverse the two and a crash in that window loses the row from both destinations); the surviving rows of a partial page are still written and the stream still advances; on_batch_error: propagate aborts and never bookmarks the failed page; a DLQ sink that itself fails fails the run instead of swallowing the row; the per-page failure budget aborts rather than silently capping; and whether row outcomes are consulted at all is decided by the presence of a DLQ, not by the sink.

It also pins the batch-atomicity guard (#737): on_batch_error: dlq_all against a sink declared best-effort is refused before a single row is read, written, dead-lettered or bookmarked — routing such a batch would send rows that had already landed to the DLQ too, and a replay would duplicate them — while the same sink runs when it writes by key or the caller opts in to the duplicates. ScriptedSink is per-row by default (an outer failure lands nothing), and .best_effort() declares the other case.

Boundary::RowsInWrite { batch, rows } is the row-level injector this needed — Boundary::Write is whole-batch, and deliberately stays an outer error even on the write_batch_partial path, because that is exactly what on_batch_error keys off.

Governance and row accounting (reliability_governance)

The guarantees RFC 0013 moved into core, run through Pipeline::with_governance and a destination built by Governance::sink_builder — the way every CLI entry point builds them:

• The DLQ is a governed destination (#870). A row the quality pass quarantines — and one a sink rejects per row — is checked against the policy with the DLQ’s own attributes: a US DLQ never receives an EU-only label (the run fails, the DLQ stays empty), an EU DLQ does. DLQ envelopes appear in the usage snapshot and count against the run budget; a budget the DLQ alone crosses stops the run with nothing written to it; a transient DLQ write failure is retried under resilience: when the DLQ’s write is replay-safe, and is not retried when it is not.
• Row accounting (#879). A sink that reports one row fewer than it was offered, with no per-row outcome, fails the run with RowsUnaccounted on both the plain and the per-row (DLQ) path; a keyed sink that merges rows, and one that declines a row by design and reports it as skipped, balance; quarantined and DLQ’d rows are counted.
• Policy quarantine is per row on every path (#873). A columnar-capable source and sink give the same written rows and the same quarantined set as the row path, and the columnar write is never called.

The canonical sink-layer order (#875) is asserted by a unit test in crates/core/src/governance.rs.

Panic isolation (reliability_panic_isolation)

A connector that panics must fail its run with a typed error, never unwind through the pipeline. Under on_error: continue an unwinding panic kills the task, not just the row, and the other matrix rows go down with it. The observability decorators turn a panic into an error of kind Panic (counted in faucet_source_errors_total / faucet_sink_errors_total) on every path.

Asserts, by driving Pipeline::run down each fast path with a connector that panics at one point, that the run returns an error naming the panic: a panic in the sink’s columnar write (write_batch_columnar) or native load (load_native), and a panic in the source’s columnar or native stream, both while opening it and while polling an item. The row path has the same guard and its own unit tests in faucet-core. The CLI test cli/tests/fast_path_panic_isolation.rs checks the matrix case: a row that panics on the columnar path fails, and the next row still runs.

Wrapper forwarding (forwarding)

Every wrapper the engine puts between the pipeline and a connector (the observability decorators, transforms, metadata columns, budgets, policies, profiling, cleanup, lineage sampling, and the CLI’s state-key, bind-context, limit, capture and mirror wrappers) must forward every defaulted hook. A missed hook falls back to its trait default and quietly turns the feature behind it off. This suite checks the canonical hook lists against the Source and Sink trait definitions, so a hook added to a trait without being added to the harness fails CI. It also checks that forward_source! / forward_sink! forward every listed hook, and that the harness names any hook a wrapper drops. Each wrapper has its own harness test next to it.

File-format fidelity (format_fidelity)

faucet_core::file_format is the one “connector” every file source and sink shares — what the s3/gcs/azure-blob/sftp sinks write is what those sources read back — so the pair-level fidelity question is asked of each format once, here, rather than eight times.

Each format’s lossiness is asserted as an explicit Tolerance rather than assumed, so a change that makes a format quietly lossier turns a green tolerance into a red mismatch. The JSON formats must be exact; CSV and XML are text-only; xlsx keeps scalar types within the double it stores them in. The named edges — XML trimming surrounding whitespace, XML having no representation for an empty array, xlsx returning an integer past 2^53 as text rather than rounding it — are pinned individually, and tabulated in the file-formats cookbook.

Adding to the program

A new guarantee

1. Add a Boundary variant to faucet_conformance::scripted if the failure point is new, and log a matching Event.
2. Write the test against Pipeline::run and assert on the event sequence, not a final count.
3. Prove the assertion can fail. Where the check is reusable, put it in scripted and add a #[should_panic] test that feeds it a deliberately wrong event log — the existing durability assertion does exactly this. A check that cannot fail is worthless.

A new source↔sink fidelity pair

Two pairs exist as references:

• crates/sink/jsonl/tests/fidelity.rs — the strictest, and the control for the whole category. A JSONL file has no schema, no column types and no affinity rules, so nothing can be excused as a destination limitation: the assertion carries no tolerance at all. If a value survives here but not in a typed destination, the loss belongs to that destination; if it fails here, the loss is faucet’s.
• crates/sink/sqlite/tests/fidelity.rs — a typed destination. Its JSON-document mode is exactly lossless; its auto-mapped mode is not, and the test names each reason (SQLite has no boolean type, so true/false become 1/0; negative zero loses its sign through a TEXT-affinity column) and pins the observed values so a future change that drops those columns entirely cannot hide behind the tolerance.

A pair that reads the destination back through the matching source connector (for example crates/interop-tests/tests/oracle_fidelity.rs) lives in crates/interop-tests: a connector crate never depends on another connector, not even for a test.

Use the shared corpus so the pair cannot quietly pick easier data:

use faucet_conformance::fidelity::{self, Tolerance};

let landed = /* read the destination back */;
fidelity::assert_round_trip(&fidelity::corpus(), &landed, Tolerance::exact());

Start at Tolerance::exact(). Widen only with a comment naming the destination limitation that forces it — Tolerance exists so “this column cannot survive here” is an explicit statement, not a loose comparison hiding a bug.

A new state shape

Commit a fixture named <shape>-v<version>.json, byte-exact as the writer emitted it, and read it in compat_state_format.rs. Never edit an existing fixture to make a test pass.

A source that changes its bookmark shape bumps Source::state_schema, teaches Source::migrate_state the step (a pure function), commits the old shape as a fixture under its own tests/fixtures/state/, and asserts in its state_compat.rs that the fixture migrates (see mongodb-cdc, schema 0 → 1). The CLI mirrors the version offline in registry::source_state_schema / migrate_source_state.

Coverage expectations

Changed lines land at ≥95% patch coverage. Most low coverage on a failure path is a design smell rather than an inherent limit: if a branch is only reachable through I/O, extract the decision into a pure function and test that — the engine suites above exist precisely because the decisions were extracted from the I/O paths that used to hide them.