Install
cargo install faucet-cli --features source-grpc faucet init --source grpc
# Cargo.toml — use it as a library cargo add faucet-source-grpc
Starter config
source:
type: grpc
config:
endpoint: "" # REQUIRED — gRPC endpoint URL (e.g.
service_name: "" # REQUIRED — Fully qualified service name (e.g.
method_name: "" # REQUIRED — Method name (e.g.
request: null # REQUIRED — Request message as JSON.
descriptor_set_path: "" # REQUIRED — Path to the compiled `FileDescriptorSet` file.
auth:
type: none # one of: none, bearer, metadata
# --- Alternative variants — replace the block above with one of these ---
# type: bearer
# config:
# token: "" # REQUIRED
#
# type: metadata
# config:
# entries: [] # REQUIRED
# tls: false # Whether to use TLS (detected from `https://` in endpoint by default).
# ca_cert: null # PEM file with an extra CA certificate to trust — for a server whose certificate is signed by a private CA.
# domain_name: null # Host name to verify the server certificate against, when it differs from the endpoint's host (connecting by IP or throug…
# records_path: "" # JSONPath to extract records from the response.
# batch_size: 1000 # Records per emitted `StreamPage`.
# rpc_kind: null # Kind of gRPC RPC to invoke.
# max_messages: null # For `RpcKind::ServerStreaming`: maximum number of messages to consume before terminating the stream.
# terminate_on_error: false # For `RpcKind::ServerStreaming`: if `true`, transient stream errors (server-side disconnects, transport errors, etc.) terminate the run with `FaucetError::Source`.
# reconnect_initial_backoff: 1 # For `RpcKind::ServerStreaming` reconnect: initial backoff delay before the first retry.
# reconnect_max_backoff: 30 # For `RpcKind::ServerStreaming` reconnect: maximum backoff cap.
# reconnect_max_attempts: 10 # For `RpcKind::ServerStreaming` reconnect: maximum consecutive reconnect attempts before surfacing the error.
# reconnect_replay_from_start: false # For `RpcKind::ServerStreaming` reconnect: whether the server replays the response stream from the beginning when the identical request is re-issued after a disconnect.
# max_decoding_message_size: null # Maximum size, in bytes, of a single inbound (decoded) gRPC message.
# max_encoding_message_size: null # Maximum size, in bytes, of a single outbound (encoded) gRPC request message.
# connect_timeout: 10 # Seconds to wait for the TCP/TLS connection to the endpoint.
# timeout: 30 # Seconds a unary call may take, and how long a server-streaming call may take to start (response headers).
# idle_timeout: 300 # For `RpcKind::ServerStreaming`: seconds to wait for the next message before treating the stream as stalled and reconnecting.
Capabilities
| Replay | Non-deterministic replay |
|---|---|
| Exactly-once replay | No |
| Incremental | No |
| Dataset discovery | No |
| Reports lag | No |
| Compression | No |
Config fields
| Field | Type | Default | Description |
|---|---|---|---|
endpointrequired | string | — | gRPC endpoint URL (e.g. "http://localhost:50051"). |
service_namerequired | string | — | Fully qualified service name (e.g. "mypackage.MyService"). |
method_namerequired | string | — | Method name (e.g. "ListUsers"). |
requestrequired | any | — | Request message as JSON. Fields are mapped to protobuf fields using the FileDescriptorSet. |
descriptor_set_pathrequired | string | — | Path to the compiled FileDescriptorSet file. |
authrequired | object | — | Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog. |
tls | boolean | — | Whether to use TLS (detected from https:// in endpoint by default). The server certificate is verified against the operating system's trust store and the bundled Mozilla roots, plus ca_cert. |
ca_cert | string | — | PEM file with an extra CA certificate to trust — for a server whose certificate is signed by a private CA. |
domain_name | string | — | Host name to verify the server certificate against, when it differs from the endpoint's host (connecting by IP or through a tunnel). |
records_path | string | — | JSONPath to extract records from the response. If not set, the entire response is returned as a single record. |
batch_size | integer | 1000 | Records per emitted StreamPage. Defaults to DEFAULT_BATCH_SIZE. |
rpc_kind | object | unary | Kind of RPC to invoke. Defaults to RpcKind::Unary. |
max_messages | integer | — | For RpcKind::ServerStreaming: maximum number of messages to consume before terminating the stream. None means consume until the server closes the stream (or the run is cancelled). |
terminate_on_error | boolean | false | For RpcKind::ServerStreaming: if true, transient stream errors (server-side disconnects, transport errors, etc.) terminate the run with FaucetError::Source. When false (the default), the source reconnects with exponential backoff up to reconnect_max_attempts. |
reconnect_initial_backoff | integer | 1 | For RpcKind::ServerStreaming reconnect: initial backoff delay before the first retry. Doubles after each failure up to reconnect_max_backoff. Defaults to 1s. |
reconnect_max_backoff | integer | 30 | For RpcKind::ServerStreaming reconnect: maximum backoff cap. Defaults to 30s. |
reconnect_max_attempts | integer | 10 | For RpcKind::ServerStreaming reconnect: maximum consecutive reconnect attempts before surfacing the error. Defaults to 10; the count resets whenever a stream delivers a message. null means unlimited retries. |
reconnect_replay_from_start | boolean | false | For RpcKind::ServerStreaming reconnect: whether the server replays the response stream from the beginning when the identical request is re-issued after a disconnect. Defaults to false. |
max_decoding_message_size | integer | — | Maximum size, in bytes, of a single inbound (decoded) gRPC message. None (the default) keeps tonic's built-in 4 MiB limit. Raise this for sources that legitimately return large messages; a too-low limit surfaces as a decode error and aborts the call. Applies to both unary and server-streaming RPCs. |
max_encoding_message_size | integer | — | Maximum size, in bytes, of a single outbound (encoded) gRPC request message. None (the default) keeps tonic's built-in limit. Rarely needs tuning for a data source, since requests are typically small. |
connect_timeout | integer | 10 | Seconds to wait for the TCP/TLS connection to the endpoint. Defaults to 10; null waits indefinitely. |
timeout | integer | 30 | Seconds a unary call may take, and how long a server-streaming call may take to start (response headers). Defaults to 30; null waits indefinitely. A timed-out server-streaming start is retried like any other transient error. |
idle_timeout | integer | 300 | For RpcKind::ServerStreaming: seconds to wait for the next message before treating the stream as stalled and reconnecting. Defaults to 300; null waits indefinitely (a half-open connection then hangs the run). |
Get started
Your first pipeline runs in five minutes.
Install the CLI, scaffold a config, and move real data — nothing external to stand up.
curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh brew install faucet-hq/faucet-stream/faucet-cli