faucetstream
Get started

Connectors / Streaming & queues

Google Cloud Pub/Sub

sink

Google Cloud Pub/Sub producer with ordering keys and partial-failure retry

Install

cargo install faucet-cli --features sink-pubsub
faucet init --sink pubsub
# Cargo.toml — use it as a library
cargo add faucet-sink-pubsub

Starter config

sink:
  type: pubsub
  config:
    topic: ""    # REQUIRED — Topic id (short name, not the fully-qualified path).
    # project_id: ""    # GCP project id that owns the topic / subscription.
    # endpoint: ""    # Override the Pub/Sub API endpoint (host:port or URL).
    # emulator_host: ""    # Point the client at a Pub/Sub emulator (`host:port`).
    # credentials: { type: application_default }    # one of: application_default, service_account_json_file, service_account_json_inline, anonymous
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_json_file
      # config:
        # path: ""    # REQUIRED
      #
      # type: service_account_json_inline
      # config:
        # json: ""    # REQUIRED
      #
      # type: anonymous
    # value_format: null    # How each record is encoded into the Pub/Sub message `data` blob.
    # ordering_key: { type: none }    # one of: none, field, jsonpath
      # --- Alternative variants — replace the block above with one of these ---
      # type: field
      # name: ""    # REQUIRED
      #
      # type: jsonpath
      # path: ""    # REQUIRED
    # attributes_field: ""    # Optional record field holding a JSON object of message attributes.
    # batch_size: 100    # Records per publish batch (1–1000).
    # concurrency: 4    # Bounded concurrent in-flight publishes.

Capabilities

Delivery At-least-once
Write modes append
Exactly-once writes No
Schema evolution No
Scoped cleanup No
Staged bulk load No
Run rollback No
Compression No

Config fields

FieldTypeDefaultDescription
topic
required		 string — Topic id (short name, not the fully-qualified path). The client forms projects/<project>/topics/<id> from the connection's project.
project_id string — GCP project id that owns the topic / subscription. Required for real Pub/Sub; the emulator infers it from PUBSUB_PROJECT_ID when unset.
endpoint string — Override the Pub/Sub API endpoint (host:port or URL). Rarely needed — prefer emulator_host for the emulator.
emulator_host string — Point the client at a Pub/Sub emulator (host:port). When set, auth is skipped and the endpoint is taken from this value — mirrors the PUBSUB_EMULATOR_HOST environment variable the SDK honours.
credentials object type: application_default How to authenticate. Defaults to Application Default Credentials.
value_format object json Record payload encoding. Default: json.
ordering_key object type: none Ordering-key derivation. Default: none.
attributes_field string — Optional record field holding a JSON object of message attributes. Values are stringified; the field is stripped from the payload before encoding. Absent field → no attributes (not an error).
batch_size integer 100 Records per publish batch (1–1000). Default 100.
concurrency integer 4 Bounded concurrent in-flight publishes. Default 4.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli
Read the guide → Star on GitHub