Chaos testing faucet serve

The reliability suites prove the engine’s ordering guarantees in-process. This page covers the layer above: a real deployment of faucet serve from the Helm chart, put through the failures a production cluster sees: pods and nodes dying, the history database going away, upgrades, and load. scripts/chaos/run.sh runs every scenario on a throwaway kind cluster and writes a pass/fail report.

Running it

scripts/chaos/run.sh                                   # every scenario, ~30 minutes
scripts/chaos/run.sh --only kill-owner,history-outage  # a subset
scripts/chaos/run.sh --image ghcr.io/faucet-hq/faucet-stream:<version>-core

It needs Docker, kind, kubectl, helm, jq and curl, about 8 GB of free disk and 6 GB of memory for the Docker VM. It writes its own kubeconfig, so your current kubectl context is never changed, and deletes the cluster when it finishes (--keep-cluster leaves it running for inspection). The exit status is the number of failed scenarios.

Option	Default	Meaning
--image	the chart’s appVersion with -core	the image under test
--chart	deploy/helm/faucet-stream	a chart directory or an oci:// reference
--replicas	2	clustered replicas
--only	every scenario	comma-separated scenario names
--out	./chaos-report	where chaos-report.md, chaos-report.json and one log per scenario go
--keep-cluster	off	leave the kind cluster running

On an arm64 machine an image published for amd64 only (releases up to 1.14.0) runs under the Docker VM’s emulation: slower, but every timing the scenarios assert is derived from the lease TTL, not wall-clock speed.

The setup

A three-node cluster: Postgres on the control-plane node (so stopping a worker never takes the databases down), and the faucet pods spread one per worker. Every scenario’s data lands in Postgres tables through an upsert sink, so row counts prove whether a re-run duplicated anything. A “slow” pipeline reads 600 rows at 0.15 s each (about 90 s), long enough to fail something mid-run.

The clustered deployment is --replicas pods with cluster.enabled, Postgres run history, token auth, a 30 s lease TTL and a trigger that fires every minute. The single-pod deployment is one replica with SQLite history on a volume and a one-shot Job.

What each scenario proves

Scenario	Failure	Passes when
membership	pods start together on a fresh history database	every pod reports ready, history_ok, the full member count, and none started degraded
kill-owner	the pod running a job is force-deleted	another pod reclaims the run (attempt ≥ 1) and completes it with exactly 600 rows
cross-cancel	a run is cancelled through a pod that does not own it	it ends cancelled, no rows arrive after that status, and records_written matches the table
idempotency	one idempotency key is sent to both pods	both get the same run_id; the key with a different body is 409
node-stop	the node under the owning pod is stopped	the survivor completes the run with 600 rows, long before Kubernetes evicts the dead pod
rolling-restart	every pod is rolled mid-run	the run completes with 600 rows
all-pods	every pod is killed with runs queued	the interrupted run and every queued run complete
history-outage	the history database refuses connections	every pod’s /readyz turns history_ok: false within the lease window, the owner cancels its own run before its lease expires (no concurrent execution), and after recovery the run completes with 600 rows
burst	40 runs submitted at once across pods	40 are accepted and each completes exactly once
poison	the pod running the same job is killed again and again	the run ends failed as a poison run instead of looping
schedule	everything above, while a trigger fires every minute	each minute produced one run; a tick missed during the outage fired once as catch-up
single-kill	the only pod dies mid-run	the run ends failed with the expired-lease reason
single-upgrade	the single pod with a volume is restarted	the Deployment uses Recreate, and two servers never run at once on the SQLite file
job-upgrade	an upgrade changes the one-shot Job	helm upgrade replaces the Job instead of failing on its immutable template

In CI

The Chaos (faucet serve) workflow builds a Postgres-only image from the commit, runs every scenario nightly, and can be started by hand with a scenario subset. The report is in the run summary and the chaos-report artifact. It is not a required check: a run takes about 30 minutes.

Adding a scenario

A scenario is a s_<name> shell function in scripts/chaos/run.sh that returns non-zero on failure and prints its evidence as its last line; register it with run_scenario <name> s_<name> in the clustered or single-pod block, and add a row to the table above. Assert outcomes a user would notice (row counts, run status, /readyz), and derive waits from LEASE_TTL rather than fixed sleeps.