Reliability testing

faucet’s claim is not that rows arrive — it is that the data can be trusted: nothing is lost, nothing is duplicated, nothing is silently corrupted, a full refresh is atomic, an incremental run resumes exactly where it stopped. Those are promises about behaviour under failure, so they are verified by a dedicated test program rather than by the connector suites.

This page describes what that program covers, how to run each tier locally, and how to add to it.

Why these tests exist separately

A guarantee like “the bookmark is persisted only after the sink confirms the data it covers” is not a property of any single function — it is a property of the order in which the engine calls two collaborators. A unit test over pure logic cannot observe an order. A connector test cannot either: it sees its own side of the boundary.

So the reliability suites run the real Pipeline::run against doubles that fail at one named boundary and record everything that happened, in order. The assertions are then written against the sequence — which is what the guarantee actually says — rather than against a final row count, which is what a weaker test checks and which stays green through a real regression.

The tiers

The first two tiers are deliberately Docker-free and run in seconds. They are the required gate because a regression in them corrupts data silently — the run still reports success — so they must never be skippable or flaky.

Running them locally

# The required gate: engine guarantees + state-format compatibility.
cargo test -p faucet-conformance --all-features

# The engine's own unit suites, which cover the DLQ routing/budget matrix
# and the masking-before-DLQ ordering.
cargo test -p faucet-core --all-features

# One guarantee at a time.
cargo test -p faucet-conformance --test reliability_bookmark_ordering
cargo test -p faucet-conformance --test reliability_exactly_once
cargo test -p faucet-conformance --test reliability_lifecycle
cargo test -p faucet-conformance --test reliability_cleanup_safety
cargo test -p faucet-conformance --test reliability_fault_injection
cargo test -p faucet-conformance --test compat_state_format

# The drift-policy matrix (Docker-free, required tier).
cargo test -p faucet-sink-sqlite --test drift_policy_matrix

# Fidelity pairs. jsonl + sqlite need no Docker; the rest boot a container.
cargo test -p faucet-sink-jsonl   --test fidelity
cargo test -p faucet-sink-sqlite  --test fidelity
cargo test -p faucet-sink-postgres --test fidelity
cargo test -p faucet-sink-mysql   --test fidelity
cargo test -p faucet-sink-mongodb --test fidelity

# Container fault injection: stops the database mid-cursor.
cargo test -p faucet-source-postgres --test fault_injection

Container-backed suites need a Docker socket. On macOS with colima:

colima start
export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"
cargo test --workspace --all-features

The SQL Server image is x86-only, so the MSSQL suites cannot run on an ARM Mac — CI covers them.

What each guarantee suite asserts

Bookmark ordering (reliability_bookmark_ordering)

Every persisted bookmark is backed by writes the sink already confirmed. A failed write, a failed flush, and a failed state-store put each leave the durable bookmark behind the failure, so a resumed run re-reads rather than skips. Resume starts at the page after the last durable bookmark — no gap, and at most one page of duplicates.

Exactly-once (reliability_exactly_once)

Both mechanisms, separately:

• Atomic watermark — writes route through the idempotent path with monotonic, distinct commit tokens; a crash in the window between the sink’s commit and the state-store put replays nothing; re-running a completed pipeline writes nothing; a failed write leaves no token behind to skip on.
• Keyed upsert — convergence with no watermark at all.

The suite also runs a control arm: the identical injected failure under the default at-least-once mode, asserting that it does duplicate. Without that, the exactly-once assertions would not be distinguishing the two modes and could pass for the wrong reason.

Overwrite atomicity and cancellation (reliability_lifecycle)

begin_overwrite precedes the first write and commit_overwrite follows the last. A failed write, a failed flush, and a cancel each abort without ever swapping — a mid-run truncate-then-fail is the most destructive thing a data tool can do. A failing commit surfaces rather than reporting success.

Separately, a cancelled run flushes: that is the difference between a cooperative cancel and a dropped future, and it is what makes a buffered Parquet footer or an S3 multipart commit instead of orphaning.

Cleanup safety (reliability_cleanup_safety)

Scoped cleanup is the only feature that deletes destination rows, so its tests are about when it must not fire: a failed run, a failed flush, a cancelled run (which returns Ok — the trap), and an overflowed key set all delete nothing. A complete run that legitimately read zero rows still cleans the scope, because that is a complete answer.

State-format compatibility (compat_state_format)

Frozen files under crates/conformance/tests/fixtures/state/ hold state written by a released version. The tests read them and assert the current code still understands them.

A round-trip test cannot catch this class of bug, because it serializes and deserializes with the same code — both halves move together and the test stays green through a breaking change. If one of these fails, the fixture is not what is wrong: either the reader regressed, or the change needs an explicit migration.

Drift policies (sink/sqlite/tests/drift_policy_matrix)

All five on_drift arms driven through the real pipeline against a destination that genuinely evolves. Each has a different silent wrong answer — warn dropping the page, evolve writing without the DDL, quarantine writing the drifting rows anyway, fail writing and then raising — so each is asserted separately, plus the control that a non-drifting page is untouched by every policy, and that quarantine without a DLQ is refused before any data moves.

Fidelity pairs (sink/*/tests/fidelity)

Five destinations, one shared corpus. What they establish together is more than each does alone:

The Postgres result is the instructive one: “Postgres loses negative zero” is false in general and true for the JSONB document path. A per-pair test is what makes that distinction visible instead of folklore.

Container fault injection (source/postgres/tests/fault_injection)

Stops the database container while a cursor is open — a harder cut than a proxy toxic, since the socket dies with no FIN, and it needs no extra image. Asserts the run fails rather than reporting success on partial data, terminates inside a hard ceiling, and never hands back a bookmark it cannot back.

The read is made slow deterministically with pg_sleep per row rather than by tuning a delay: the first version of this test raced, the table streamed in before the cut, and it “failed” on a perfectly healthy run.

Dead-letter boundary (reliability_dlq_boundary)

The DLQ is the one error path where the run keeps going and still reports success, which is what makes a mistake here so expensive: a row is dropped, or a bookmark advances past a row that was never made durable anywhere, and nothing in the exit code says so.

Asserts, against one interleaved log shared by the main sink, the DLQ sink and the state store: a rejected row is durable in the DLQ before the bookmark covering its page is persisted (reverse the two and a crash in that window loses the row from both destinations); the surviving rows of a partial page are still written and the stream still advances; on_batch_error: propagate aborts and never bookmarks the failed page; a DLQ sink that itself fails fails the run instead of swallowing the row; the per-page failure budget aborts rather than silently capping; and whether row outcomes are consulted at all is decided by the presence of a DLQ, not by the sink.

Boundary::RowsInWrite { batch, rows } is the row-level injector this needed — Boundary::Write is whole-batch, and deliberately stays an outer error even on the write_batch_partial path, because that is exactly what on_batch_error keys off.

File-format fidelity (format_fidelity)

faucet_core::file_format is the one “connector” every file source and sink shares — what the s3/gcs/azure-blob/sftp sinks write is what those sources read back — so the pair-level fidelity question is asked of each format once, here, rather than eight times.

Each format’s lossiness is asserted as an explicit Tolerance rather than assumed, so a change that makes a format quietly lossier turns a green tolerance into a red mismatch. The JSON formats must be exact; CSV and XML are text-only; xlsx keeps scalar types within the double it stores them in. The named edges — XML trimming surrounding whitespace, XML having no representation for an empty array, xlsx returning an integer past 2^53 as text rather than rounding it — are pinned individually, and tabulated in the file-formats cookbook.

Adding to the program

A new guarantee

1. Add a Boundary variant to faucet_conformance::scripted if the failure point is new, and log a matching Event.
2. Write the test against Pipeline::run and assert on the event sequence, not a final count.
3. Prove the assertion can fail. Where the check is reusable, put it in scripted and add a #[should_panic] test that feeds it a deliberately wrong event log — the existing durability assertion does exactly this. A check that cannot fail is worthless.

A new source↔sink fidelity pair

Two pairs exist as references:

• crates/sink/jsonl/tests/fidelity.rs — the strictest, and the control for the whole category. A JSONL file has no schema, no column types and no affinity rules, so nothing can be excused as a destination limitation: the assertion carries no tolerance at all. If a value survives here but not in a typed destination, the loss belongs to that destination; if it fails here, the loss is faucet’s.
• crates/sink/sqlite/tests/fidelity.rs — a typed destination. Its JSON-document mode is exactly lossless; its auto-mapped mode is not, and the test names each reason (SQLite has no boolean type, so true/false become 1/0; negative zero loses its sign through a TEXT-affinity column) and pins the observed values so a future change that drops those columns entirely cannot hide behind the tolerance.

Use the shared corpus so the pair cannot quietly pick easier data:

#![allow(unused)]
fn main() {
use faucet_conformance::fidelity::{self, Tolerance};

let landed = /* read the destination back */;
fidelity::assert_round_trip(&fidelity::corpus(), &landed, Tolerance::exact());
}

Start at Tolerance::exact(). Widen only with a comment naming the destination limitation that forces it — Tolerance exists so “this column cannot survive here” is an explicit statement, not a loose comparison hiding a bug.

A new state shape

Commit a fixture named <shape>-v<version>.json, byte-exact as the writer emitted it, and read it in compat_state_format.rs. Never edit an existing fixture to make a test pass.

Coverage expectations

Changed lines land at ≥95% patch coverage. Most low coverage on a failure path is a design smell rather than an inherent limit: if a branch is only reachable through I/O, extract the decision into a pure function and test that — the engine suites above exist precisely because the decisions were extracted from the I/O paths that used to hide them.