Connectors / APIs & web

gRPC

source

gRPC source with dynamic protobuf

Install

cargo install faucet-cli --features source-grpc
faucet init --source grpc
# Cargo.toml — use it as a library
cargo add faucet-source-grpc

Starter config

source:
  type: grpc
  config:
    endpoint: ""    # REQUIRED — gRPC endpoint URL (e.g.
    service_name: ""    # REQUIRED — Fully qualified service name (e.g.
    method_name: ""    # REQUIRED — Method name (e.g.
    request: null    # REQUIRED — Request message as JSON.
    descriptor_set_path: ""    # REQUIRED — Path to the compiled `FileDescriptorSet` file.
    auth:
      type: none    # one of: none, bearer, metadata
      # --- Alternative variants — replace the block above with one of these ---
      # type: bearer
      # config:
        # token: ""    # REQUIRED
      #
      # type: metadata
      # config:
        # entries: []    # REQUIRED
    # tls: false    # Whether to use TLS (detected from `https://` in endpoint by default).
    # ca_cert: null    # PEM file with an extra CA certificate to trust — for a server whose certificate is signed by a private CA.
    # domain_name: null    # Host name to verify the server certificate against, when it differs from the endpoint's host (connecting by IP or throug…
    # records_path: ""    # JSONPath to extract records from the response.
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # rpc_kind: null    # Kind of gRPC RPC to invoke.
    # max_messages: null    # For `RpcKind::ServerStreaming`: maximum number of messages to consume before terminating the stream.
    # terminate_on_error: false    # For `RpcKind::ServerStreaming`: if `true`, transient stream errors (server-side disconnects, transport errors, etc.) terminate the run with `FaucetError::Source`.
    # reconnect_initial_backoff: 1    # For `RpcKind::ServerStreaming` reconnect: initial backoff delay before the first retry.
    # reconnect_max_backoff: 30    # For `RpcKind::ServerStreaming` reconnect: maximum backoff cap.
    # reconnect_max_attempts: 10    # For `RpcKind::ServerStreaming` reconnect: maximum consecutive reconnect attempts before surfacing the error.
    # reconnect_replay_from_start: false    # For `RpcKind::ServerStreaming` reconnect: whether the server replays the response stream from the beginning when the identical request is re-issued after a disconnect.
    # max_decoding_message_size: null    # Maximum size, in bytes, of a single inbound (decoded) gRPC message.
    # max_encoding_message_size: null    # Maximum size, in bytes, of a single outbound (encoded) gRPC request message.
    # connect_timeout: 10    # Seconds to wait for the TCP/TLS connection to the endpoint.
    # timeout: 30    # Seconds a unary call may take, and how long a server-streaming call may take to start (response headers).
    # idle_timeout: 300    # For `RpcKind::ServerStreaming`: seconds to wait for the next message before treating the stream as stalled and reconnecting.

Capabilities

Replay Non-deterministic replay
Exactly-once replay No
Incremental No
Dataset discovery No
Reports lag No
Compression No

Config fields

FieldTypeDefaultDescription
endpoint
required
string — gRPC endpoint URL (e.g. "http://localhost:50051").
service_name
required
string — Fully qualified service name (e.g. "mypackage.MyService").
method_name
required
string — Method name (e.g. "ListUsers").
request
required
any — Request message as JSON. Fields are mapped to protobuf fields using the FileDescriptorSet.
descriptor_set_path
required
string — Path to the compiled FileDescriptorSet file.
auth
required
object — Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI's top-level auth: catalog.
tls boolean — Whether to use TLS (detected from https:// in endpoint by default). The server certificate is verified against the operating system's trust store and the bundled Mozilla roots, plus ca_cert.
ca_cert string — PEM file with an extra CA certificate to trust — for a server whose certificate is signed by a private CA.
domain_name string — Host name to verify the server certificate against, when it differs from the endpoint's host (connecting by IP or through a tunnel).
records_path string — JSONPath to extract records from the response. If not set, the entire response is returned as a single record.
batch_size integer 1000 Records per emitted StreamPage. Defaults to DEFAULT_BATCH_SIZE.
rpc_kind object unary Kind of RPC to invoke. Defaults to RpcKind::Unary.
max_messages integer — For RpcKind::ServerStreaming: maximum number of messages to consume before terminating the stream. None means consume until the server closes the stream (or the run is cancelled).
terminate_on_error boolean false For RpcKind::ServerStreaming: if true, transient stream errors (server-side disconnects, transport errors, etc.) terminate the run with FaucetError::Source. When false (the default), the source reconnects with exponential backoff up to reconnect_max_attempts.
reconnect_initial_backoff integer 1 For RpcKind::ServerStreaming reconnect: initial backoff delay before the first retry. Doubles after each failure up to reconnect_max_backoff. Defaults to 1s.
reconnect_max_backoff integer 30 For RpcKind::ServerStreaming reconnect: maximum backoff cap. Defaults to 30s.
reconnect_max_attempts integer 10 For RpcKind::ServerStreaming reconnect: maximum consecutive reconnect attempts before surfacing the error. Defaults to 10; the count resets whenever a stream delivers a message. null means unlimited retries.
reconnect_replay_from_start boolean false For RpcKind::ServerStreaming reconnect: whether the server replays the response stream from the beginning when the identical request is re-issued after a disconnect. Defaults to false.
max_decoding_message_size integer — Maximum size, in bytes, of a single inbound (decoded) gRPC message. None (the default) keeps tonic's built-in 4 MiB limit. Raise this for sources that legitimately return large messages; a too-low limit surfaces as a decode error and aborts the call. Applies to both unary and server-streaming RPCs.
max_encoding_message_size integer — Maximum size, in bytes, of a single outbound (encoded) gRPC request message. None (the default) keeps tonic's built-in limit. Rarely needs tuning for a data source, since requests are typically small.
connect_timeout integer 10 Seconds to wait for the TCP/TLS connection to the endpoint. Defaults to 10; null waits indefinitely.
timeout integer 30 Seconds a unary call may take, and how long a server-streaming call may take to start (response headers). Defaults to 30; null waits indefinitely. A timed-out server-streaming start is retried like any other transient error.
idle_timeout integer 300 For RpcKind::ServerStreaming: seconds to wait for the next message before treating the stream as stalled and reconnecting. Defaults to 300; null waits indefinitely (a half-open connection then hangs the run).
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli