Install
cargo install faucet-cli --features source-singer faucet init --source singer
# Cargo.toml — use it as a library cargo add faucet-source-singer
Starter config
source:
type: singer
config:
executable: "" # REQUIRED — The tap executable to run (looked up on `PATH`, or an absolute path), e.g.
# args: [] # Extra positional/flag arguments appended after faucet's own `--config` / `--catalog` / `--state` flags.
# tap_config: null # The tap's configuration object.
# catalog: null # Optional Singer catalog object, passed as `--catalog` when present.
stream: "" # REQUIRED — The single stream to emit records for.
# state_key: null # State-store key for the resume bookmark.
# flush_on_state: true # Flush the current page whenever the tap emits a STATE message (so the pipeline can checkpoint at the tap's own commit points).
# idle_timeout_secs: 3600 # Abort the run if no line arrives from the tap within this many seconds (default 3600).
# max_line_bytes: 67108864 # Longest tap output line accepted, in bytes (default 64 MiB).
# on_malformed: null # What to do when the tap emits a line that is not valid Singer JSON.
# inherit_env: null # The `inherit_env` setting of a Singer tap or target.
Capabilities
| Replay | Non-deterministic replay |
|---|---|
| Exactly-once replay | No |
| Incremental | Yes |
| Dataset discovery | No |
| Reports lag | No |
| Compression | No |
Config fields
| Field | Type | Default | Description |
|---|---|---|---|
executablerequired | string | — | The tap executable to run (looked up on PATH, or an absolute path), e.g. tap-github or /opt/taps/tap-csv. |
streamrequired | string | — | The single stream to emit records for. RECORD messages for any other stream are dropped (v0 is single-stream). |
args | array | [] | Extra positional/flag arguments appended after faucet's own --config / --catalog / --state flags. |
tap_config | any | — | The tap's configuration object. faucet's loader has already resolved any ${secret:…} references before this reaches the source; it is materialized to a private temp file and passed as --config. |
catalog | any | — | Optional Singer catalog object, passed as --catalog when present. Selects streams/fields; required by most SDK-based taps. |
state_key | string | — | State-store key for the resume bookmark. Defaults to singer:{executable}:{stream} when unset. |
flush_on_state | boolean | true | Flush the current page whenever the tap emits a STATE message (so the pipeline can checkpoint at the tap's own commit points). Default true. |
idle_timeout_secs | integer | 3600 | Abort the run if no line arrives from the tap within this many seconds (default 3600). null waits indefinitely — a stalled tap then hangs the run. |
max_line_bytes | integer | 67108864 | Longest tap output line accepted, in bytes (default 64 MiB). A longer line — a huge RECORD, or binary output with no newline — fails the run instead of exhausting memory. |
on_malformed | object | skip | How to handle a line that is not valid Singer JSON. Default MalformedPolicy::Skip. |
inherit_env | boolean | array | true | Which of faucet's environment variables the tap receives: true (default) the whole environment, false only PATH, HOME, LANG, LC_ALL and TMPDIR, or a list of variable names passed on top of that baseline. Set it to keep faucet's own credentials away from the tap. |
Get started
Your first pipeline runs in five minutes.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli