Running faucet as a service (faucet serve)

faucet serve turns faucet from a one-shot CLI into a long-running HTTP control plane: an orchestrator (Airflow, Temporal, Dagster, Argo) submits pipeline configs over HTTP, polls status, cancels runs, and streams logs — while faucet amortizes startup (TLS handshakes, connection pools, schema introspection) across many runs in one process. It is the second supported runtime mode alongside one-shot faucet run and the cron faucet schedule.

The full endpoint/schema reference is in HTTP API reference; this page is the guided tour. serve requires the serve Cargo feature (cargo install faucet-cli --features serve, or --features full).

Quickstart

# Start the server (loopback by default). Auth is mandatory — see below.
FAUCET_SERVE_AUTH_TOKEN=s3cret faucet serve --listen 127.0.0.1:8080

# Submit a run.
curl -XPOST http://127.0.0.1:8080/v1/runs \
  -H "Authorization: Bearer s3cret" -H 'content-type: application/json' \
  -d '{"config":"version: 1\npipeline:\n  source: {type: file, config: { path: in.csv }}\n  sink: {type: file, config: { path: out.jsonl }}\n","name":"adhoc"}'
# → {"run_id":"0192…","status":"queued","submitted_at":"…"}

# Poll it to completion.
curl -H "Authorization: Bearer s3cret" http://127.0.0.1:8080/v1/runs/0192…

# Tail its logs (SSE).
curl -N -H "Authorization: Bearer s3cret" http://127.0.0.1:8080/v1/runs/0192…/logs

⚠️ Security model — read before exposing

serve executes arbitrary client-supplied pipeline configs with the server’s identity. That is a real privilege surface:

Full interpolation: submitted configs resolve ${env:…}, ${file:…}, ${secret:…}, and ${vault:…}/${aws-sm:…}/… against the server’s environment, filesystem, and credentials — exactly like faucet run. An authenticated caller can read any secret the server can reach.
SSRF / egress: a submitted REST/HTTP source can be pointed at 169.254.169.254 or internal services and will be fetched with the server’s network identity.

Mitigations are deployment-level and mandatory:

Never run with --no-auth on a non-loopback bind. The no-auth gate is explicit: without --auth-token/FAUCET_SERVE_AUTH_TOKEN and without --no-auth, startup fails. --no-auth with a non-loopback --listen / FAUCET_SERVE_LISTEN (0.0.0.0, ::, a LAN address) also fails unless --allow-unauthenticated-network (FAUCET_SERVE_ALLOW_UNAUTHENTICATED_NETWORK) accepts the exposure, and then the server logs a warning at startup. Inside a container that binds 0.0.0.0, publish the port on loopback only (-p 127.0.0.1:8080:8080) when you pass it.
A browser can reach a loopback server too. A --no-auth loopback server therefore answers only localhost, 127.0.0.1, [::1] and its listen address as Host, every server refuses a state-changing request whose Origin is another site’s, and /mcp and webhook triggers take only the content types they parse. Behind a proxy, list the public host name with --allowed-host (which also turns the Host check on for an authenticating server) and, when the proxy rewrites Host, the console’s origin with --allowed-origin. See Authentication.
Use long random tokens (openssl rand -hex 32): the server warns about one under 16 characters, and refuses an address for a minute after 20 invalid tokens.
Run single-tenant, behind authentication, behind egress controls / network policy. The default loopback bind (127.0.0.1) is deliberate — exposing externally is an explicit choice.
Terminate TLS at a proxy/ingress (serve speaks plain HTTP).
Prefer FAUCET_SERVE_AUTH_TOKEN over --auth-token (the latter leaks through ps//proc).
Never run a serve process at FAUCET_LOG=debug when submitted configs hold resolved secrets — only faucet’s own log output is redacted, not third-party connector debug logging.

RBAC & audit log

A single --auth-token is one implicit admin principal — fine for a personal deployment, but a team needs scoped access and attribution. --auth-config <file> enables role-based access control: a YAML/JSON list of principals, each a { name, token, role } where role is viewer (read-only), operator (submit/cancel/delete runs, doctor, triggers), or admin (everything, including the audit log).

# auth.yaml — tokens resolve ${env:…} / ${file:…} / ${secret:…} and
# secret-manager references (${vault:…}, …) when the server starts
principals:
  - { name: alice, token: "${env:ALICE_TOKEN}", role: admin }
  - { name: ci,    token: "${env:CI_TOKEN}",    role: operator }
  - { name: dash,  token: "${env:DASH_TOKEN}",  role: viewer }

faucet serve --auth-config auth.yaml --history postgres://…/faucet

A viewer’s POST /v1/runs returns 403; its GET /v1/runs returns 200. --auth-config is mutually exclusive with --auth-token / --no-auth. A reference that does not resolve refuses the start, and a token still holding ${ is refused — never accepted literally. No config the server loads can read those variables or files, nor FAUCET_VAULT_KEY and the server’s own token variables — not as a reference, and not as data either: a connector, state, DLQ or preview path that reaches one of the server’s own files is refused. Those files are the .env it loaded and every .env / .env.* in its working directory, the --auth-config, --policy, --triggers, --templates-sync, --connect-providers, --otel-config and --default-config files and everything they read, the SQLite run-history database with its -wal / -shm files, and the --default-config’s file state directory. Keep other secrets out of the working and temp directories, or start the server with --allow-path-root set to a data directory of its own.

A config submitted over HTTP or MCP may not use the singer source or sink — they run a program on the host — unless the server is started with --allow-subprocess-connectors. Registered templates may.

Nor may it authenticate with the server’s own identity — its environment, an instance or workload role (IRSA, EKS Pod Identity, GKE workload identity, Azure managed identity), application-default credentials or local credential files — unless the server is started with --allow-host-credentials (FAUCET_SERVE_ALLOW_HOST_CREDENTIALS=true, Helm serve.allowHostCredentials). A submitted config names its own keys instead; a registered template may use the server’s identity, but a value a caller supplies for one of its params may not land in such a connector’s config without the flag. A tenant never gets it. Keep it off on a server whose role reaches anything a caller should not, and set it the same on every replica.

Every other reference a submitted config makes goes through the same allow-list: local paths stay under the working and temp directories, ${env:} and secret-manager references are refused, and cloud-metadata hosts are unreachable, until the operator widens it:

faucet serve --auth-config auth.yaml \
  --allow-path-root /data/pipelines \
  --allow-url-host '*.example.com' \
  --allow-env 'PIPELINE_*' --allow-secret-scheme vault

See Reference policy for the full table.

Every mutating action (run.submit / run.cancel / run.delete) and every denied attempt is written to a tamper-evident audit log — principal, role, action, run id, config fingerprint, source IP, timestamp, result. Admins read it:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  'http://127.0.0.1:8080/v1/audit?action=run.submit&limit=50'

Audit records persist in the run-history backend (faucet_serve_audit on the SQL backends; an in-memory ring for the default backend, lost on restart) and expire with --retain-terminal-runs-secs. For a durable trail, use a --history postgres://…/sqlite:… backend.

Bounded concurrency & backpressure

--max-concurrent-runs (default min(16, cpu_count())) bounds how many runs execute at once; --max-queued-runs (default 8×) bounds the queue. A submit past the queue cap returns 429 with Retry-After. Note that total concurrent pipeline work ≈ max-concurrent-runs × each config's execution.max_concurrent.

Idempotency

Supply idempotency_key to make retries safe (the common Idempotency-Key pattern):

First submit with a key → runs normally.
Re-submit the same key + same request within --idempotency-retention-secs (default 24h) → returns the original run_id (replayed, no new run).
Same key + a different request → 409 Conflict.
After the retention window, the key is re-usable for a fresh run.
Deleting a run also frees its idempotency key immediately — a later submit with that key starts a fresh run rather than 404-ing on the deleted record.

The “request” identity covers the merged config and the run-affecting request fields — clock, timeout_secs, and labels. In particular, a retry that reuses the key but changes the backfill clock is a 409, not a replay of the original window (so you can’t silently get the original clock’s results).

The claim is atomic, so concurrent retries can’t both start a run.

Overlapping runs of one row

Each run leases its rows in the config’s durable state: store, so two runs of the same row never start from the same bookmark. The server has no override for a live lease: a second run of a row that is already running — from the API, a trigger, a template, a backfill, or a faucet run / faucet schedule on the same store — fails with the holder’s run id, pid and lease expiry. Cancel the holder (its lease is released as it stops) or wait for it to finish; a holder that crashed on another host keeps the lease until it expires, about a minute after it stopped renewing.

Cancellation

POST /v1/runs/{id}/cancel cooperatively cancels an in-flight run (202); on an already-terminal run it’s a 200 no-op. The same cooperative path handles a run that hits its timeout_secs and the server-shutdown drain.

Cancellation is flush-completing: the pipeline stops at its next page boundary and flushes the sink, so a buffered sink (e.g. Parquet, whose footer is only written on flush) commits the rows written so far rather than orphaning the whole file (#146 H16). The run is then marked cancelled, with the records and invocations it wrote before stopping — there is no cross-process resume, so re-submit to continue. A run still stuck mid-write after a bounded flush grace is hard-dropped (its buffered output may be lost), so a hung run can’t wedge shutdown.

`--default-config` (workspace defaults)

Pass --default-config <file> to merge shared settings under every submitted run (submitted values win; objects merge, scalars/arrays replace). Pin state:, execution:, and the auth: catalog once instead of repeating them per request. See cli/examples/serve_minimal.yaml.

Hot-reloading the default-config

After editing the --default-config file, reload it in place — no restart, no interruption to in-flight runs (they already captured their config):

curl -fsS -X POST -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8080/v1/reload
# → {"reloaded": true, "path": "…"}

POST /v1/reload is admin-only (RBAC Reload permission). It re-reads and re-validates the file and atomically swaps the merge base; subsequent submissions merge onto the new base. An invalid new config returns 422 and the previous base is kept. When the server was started without --default-config, it is a no-op ({"reloaded": false}).

Run history & persistence

By default run records live in memory and are lost on restart. For durable history across restarts, point --history at a database (requires the matching build feature):

# Postgres (feature: serve-history-postgres)
faucet serve --history 'postgres://user:pw@db/faucet'
# SQLite (feature: serve-history-sqlite)
faucet serve --history 'sqlite:/var/lib/faucet/runs.db'

Both create their schema on first connect. If the backend is unreachable at startup, or becomes unreachable at runtime (a refused or dropped connection, a pool timeout, an I/O error, a server shutting down or out of connections — each retried a few times first), serve degrades so it stays up: it logs once, sets the faucet_serve_history_degraded gauge, and /readyz returns 503. Any other database error (a constraint, a decode failure, lock contention) is returned to the request that hit it and never degrades the server.

A pool timeout while every pooled connection is in use is treated as load, not an outage, as long as the backend answered some call in the last 30 seconds: the request gets the error and the server stays out of degraded mode. Once the backend has answered nothing for 30 seconds — every connection hung on a database that stopped responding — the next pool timeout degrades the server too.

In cluster mode (--cluster) the server never falls back to memory: peers must share one store, so requests that need the database fail instead. The lease heartbeat (every third of --lease-ttl-secs) is the health check there: while it cannot reach the database, /readyz returns 503 with history_ok: false and faucet_serve_history_unreachable is 1. Both clear on the first heartbeat that gets through.

A Postgres primary that fails over to a read-only standby counts as unreachable too: a write refused with SQLSTATE 25006 (read_only_sql_transaction) or 57P04 (database_dropped) degrades the server, every pooled connection opened before that answer is closed instead of reused, and pooled connections live at most five minutes, so the pool reaches the new primary soon after a failover.

While degraded, the server admits nothing. Every new run, template trigger, backfill, DLQ replay, verify repair, rollback, webhook trigger and schedule tick answers 503 (“history unavailable”), on a single server as in a cluster, so no work starts that the server could not record. The check runs again just before a run is recorded, and that write is never kept in memory only: a backend that fails at that moment refuses the run before anything has run. Templates, schedules, policy versions, tenants, connections, change requests and approvals, and submissions with an idempotency key fail with 503 too. Reads are served from memory meanwhile.

Runs already in flight keep going. In a cluster, a replica that cannot renew its leases cancels its local runs and hands them back before a peer may claim them. When a run finishes, its terminal record — built from the run’s own record, so its name, labels and tenant are kept — is written owner-fenced; one the backend does not take is held and retried from every lease tick until it lands. The completion callback, the run metrics and the tenant counters follow only a landed write. A record the backend keeps refusing for good falls back, after five refusals, to its status alone. On a single server a refused status is dropped (logged, counted in faucet_serve_pending_finalize_dropped_total) and orphan recovery marks the run failed. In a cluster it is never dropped: the replica keeps the run’s lease, so no peer runs it again, and retries the status with a backoff of up to ten minutes (counted in faucet_serve_pending_finalize_refused_total). faucet_serve_pending_finalize counts what waits (alert FaucetServePendingFinalize). A server that stops while records are held loses them; orphan recovery then marks those runs failed. Re-running such a run is safe: bookmarks advance only after the sink confirms a write.

Every lease tick probes a degraded backend with a no-op write — so a primary that answers reads but refuses writes stays degraded — or, when it was unreachable at startup, tries to connect again; the server leaves degraded mode once the write succeeds. /readyz reports degraded: {since} while it lasts. A --policy file that seeds the policy store (or, with --policy-store file, is stored for the runs pinned to it) waits for the history the same way: until it is stored, /readyz shows policy_seed: "pending" and new work answers 503. Terminal records are retained for --retain-terminal-runs-secs (default 7 days).

Multi-instance orphan recovery (run-ownership leases)

A persistent backend can be shared by several faucet serve instances (an HA pair, a rolling/blue-green deploy). Each instance gets a fresh id at startup and owns the runs it executes; while a run is in flight its owner heartbeats a lease on the run record (at ~⅓ of --lease-ttl-secs, default 30s). A run is only recovered — marked failed with owning serve instance's lease expired — once its lease has expired, i.e. its owner stopped heartbeating (crashed or was shut down). Recovery runs both at startup and periodically, so a surviving instance reclaims a dead peer’s orphans without waiting for a restart.

This means a starting or running instance never fails another live instance’s in-flight runs — the gap that an unscoped “fail every non-terminal run at startup” sweep would open on a shared database. Tune --lease-ttl-secs above your worst-case GC/IO stall so a healthy-but-slow instance is never falsely reclaimed (a longer TTL is safer but slows how quickly a crashed instance’s runs are cleaned up). The in-memory backend is single-process and unshared, so leases don’t apply to it. There is still no cross-process resume: a recovered run is marked failed, not continued — re-submit to retry.

Graceful shutdown

On SIGTERM/SIGINT the server starts draining: /readyz turns 503 ("draining": true) so a load balancer stops routing to it, and every new submission — runs, template triggers, webhook fires — is refused with 503, code draining, and Retry-After: 5. In-flight runs get --shutdown-grace-secs (default 60) to finish; the remainder are cancelled cooperatively. On a single instance they are marked failed (server_shutdown); in cluster mode they are handed back as pending for a peer to claim, which resumes them from their bookmark (at-least-once, like any failover). A cancelled, timed-out or shutdown-interrupted run keeps the record counts and invocations it reached, so its record shows the rows that did land.

Health & observability

/healthz — liveness (always 200 while serving).
/readyz — 503 when history is degraded, the queue is full, or the server is draining.
/metrics — Prometheus, including faucet_serve_* series. /metrics is unauthenticated; restrict it at the network layer if its labels are sensitive.